A multi-modal data clustering method based on kernelized optimal transport tensor graph

Abstract

This invention provides a multimodal data clustering method based on kernelized optimal transport tensor graphs, belonging to the field of multimodal data processing and pattern recognition technology. The method first maps each modality's data to a regenerating kernel Hilbert space using a kernel function to construct a kernel matrix, learning a kernelized low-dimensional representation to preserve nonlinear structure and reduce noise. Then, it introduces self-transfer optimal transport, using Wasserstein distance to measure sample similarity and obtain a discriminative similarity map. Next, it stacks the auxiliary graphs of each modality into a third-order tensor and applies low-rank constraints to capture intramodal and intermodal structural consistency. Through alternating iterative optimization, it jointly solves for the kernelized representation, optimal transport graph, and tensor graph, finally constructing a consensus graph and using spectral clustering to output the clustering results. This invention effectively integrates the nonlinear relationships and high-order interaction information of multimodal data, significantly improving the clustering accuracy and robustness in complex scenarios.

Description

Technical Field

[0001] This invention relates to the field of multimodal data processing and pattern recognition technology, specifically to a multimodal data clustering method based on kernelized optimal transport tensor graphs. Background Technology

[0002] Multimodal data clustering is an important research direction in pattern recognition and machine learning. Its core objective is to effectively fuse multimodal information (such as texture and shape features of images, or DNA methylation and RNA expression data in bioinformatics) from different sources or represented by different feature extractors under unsupervised conditions to achieve automatic and high-precision sample segmentation. This technology has wide application value in many practical scenarios such as image annotation, video analysis, bioinformatics, and social network mining. To achieve this goal, the academic community has proposed a variety of strategies, among which graph learning-based methods have attracted much attention due to their sound theoretical foundation and interpretability. For example, researchers have proposed multimodal clustering methods that construct a unified graph through weighted graph learning to capture intramodal and intermodal relationships, or graph clustering methods that simultaneously mine multimodal consistency and diversity within a unified framework. In recent years, some works have gone a step further, using manifold topology to enhance graph learning, or using tensor learning to capture cross-modal consistent low-rank structures to improve graph quality.

[0003] However, existing graph-driven multimodal clustering methods still face several key bottlenecks when dealing with complex real-world data. First, traditional methods typically construct similarity graphs directly in the original feature space based on Euclidean distance. However, Euclidean distance has limited discriminative power and struggles to effectively capture the complex nonlinear relationships and high-order interaction structures prevalent in real-world data, often resulting in suboptimal graph learning outcomes. Second, the original high-dimensional feature space is often contaminated with noise and redundant information; directly using it for similarity calculation introduces unreliable metrics, weakening the discriminative power of the constructed graph. Furthermore, when modeling cross-modal relationships, existing methods often perform simple linear fusion of graphs from different modalities, lacking effective mechanisms for in-depth mining of high-order consistency and complementary structures between modalities, thus limiting the potential for synergistic enhancement of multimodal information.

[0004] In view of the above, this application is hereby submitted. Summary of the Invention

[0005] This invention provides a multimodal data clustering method based on kernelized optimal transport tensor graphs, which can at least partially improve the above-mentioned problems.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A multimodal data clustering method based on kernelized optimal transport tensor graphs includes: Acquire the multimodal sample data to be processed, preprocess the multimodal sample data to be processed, and construct the kernel matrix of each mode; The kernel matrix of each modality is input into a pre-trained multimodal clustering model based on kernelized optimal transport tensor graph to generate clustering results; The training steps for the multimodal clustering model based on the kernelized optimal transport tensor graph are as follows: Obtain the training multimodal dataset, and based on the preset kernel function, map the original feature matrices of each modality in the training multimodal dataset to the regenerated kernel Hilbert space to construct the corresponding kernel matrix; Based on the kernel matrix of each modality, the corresponding kernelized low-dimensional representation is learned, and the similarity graph of each modality is learned based on the kernelized low-dimensional representation. Construct a cross-modal tensor graph model, stack the auxiliary graphs of each modality into a third-order tensor along the modal dimension, and apply a low-rank constraint to the third-order tensor based on the similarity graph; An alternating iterative optimization strategy is adopted to solve the sub-problems corresponding to the kernelized low-dimensional representation, the similarity graph, and the auxiliary graph respectively, thereby achieving joint optimization of the three. Based on the optimized auxiliary graphs of each modality, a unified consensus graph is constructed through weighted averaging and symmetry processing. The consensus graph is input into the spectral clustering algorithm. The low-dimensional spectral embedding is obtained by calculating the eigenvalue decomposition of the Laplacian matrix. Then, the K-means clustering algorithm is used to divide the data to obtain the final clustering result. The final clustering results are evaluated according to the preset evaluation indicators. When the evaluation results are determined to reach the preset performance threshold, the training is completed, and a trained multimodal clustering model based on the kernelized optimal transport tensor graph is obtained.

[0008] In summary, in practical applications, data often originates from different channels or is represented by different feature extractors, forming multimodal data. How to effectively integrate the consistency and complementarity information among multimodal data to achieve high-precision clustering under unsupervised conditions is a core challenge in this field. Existing graph-driven multimodal clustering methods typically rely on Euclidean distance to construct similarity graphs, which struggles to capture the complex nonlinear structures and high-order interactions of real-world data, leading to suboptimal graph learning and limited clustering performance. To overcome these limitations, this invention proposes a multimodal data clustering method based on kernelized optimal transport tensor graphs. This method includes acquiring multimodal sample data and preprocessing it to construct kernel matrices for each modality; inputting the preprocessed data into a pre-trained multimodal clustering model based on kernelized optimal transport tensor graphs to obtain clustering results. This addresses at least one of the technical problems of existing technologies, such as over-reliance on Euclidean distance metrics, difficulty in capturing nonlinear data structures, and insufficient modeling of high-order consistency across modalities.

[0009] The training steps of the model are as follows: First, acquire multimodal data and map the original features of each modality to the regenerating kernel Hilbert space based on a preset kernel function (such as a polynomial kernel or a Gaussian kernel), constructing the corresponding kernel matrix to implicitly capture the nonlinear correlations within the data. Second, learn the kernelized low-dimensional representation of each modality, which achieves dimensionality reduction and noise reduction while preserving the nonlinear structure of the kernel space. Third, combine self-transfer optimal transport to learn a similarity graph, using Wasserstein distance instead of traditional Euclidean distance to measure the relationship between samples, overcoming the shortcomings of traditional methods in characterizing complex data distributions. Fourth, construct a cross-modal tensor graph model, stacking the auxiliary graphs of each modality along the modal dimension into a third-order tensor, and applying tensor low-rank constraints (such as tensor kernel norm or tensor Tubal rank) to capture the shared structural consistency within and between modalities, achieving cross-modal high-order information interaction. Fifth, employ an alternating iterative optimization strategy to solve the sub-problems corresponding to the kernelized representation, optimal transport graph, and tensor graph, achieving joint optimization of the three until convergence. Based on the optimized multimodal auxiliary graph, a unified consensus graph is constructed through weighted averaging and symmetric processing to ensure graph symmetry. The consensus graph is input into a spectral clustering algorithm, and low-dimensional spectral embeddings are obtained by calculating the eigenvalue decomposition of the Laplacian matrix. Then, K-means and other clustering algorithms are used to obtain the final clustering results. Clustering performance is evaluated on a multimodal benchmark dataset by calculating three metrics: clustering accuracy (ACC), normalized mutual information (NMI), and purity (PUR), comprehensively judging whether the model has reached a preset performance threshold until a pre-trained clustering model is obtained. This invention achieves a unified optimization framework for nonlinear relationship mining, robust distance measurement, and cross-modal high-order consistency modeling of multimodal data through the seamless integration of kernel representation learning, optimal transport graph learning, and tensor graph learning. This significantly improves clustering accuracy and interpretability in multimodal data scenarios such as image recognition and bioinformatics.

[0010] Compared with existing technologies, this method innovatively integrates kernel representation learning, optimal transport graph learning, and tensor graph learning into a unified optimization framework. It offers the following advantages: First, by mapping the original data to a regenerating kernel Hilbert space through kernel functions, it implicitly captures the nonlinear relationships within the data, overcoming the limitation of traditional methods in characterizing complex data structures in the original feature space. Second, by introducing optimal transport theory and replacing traditional Euclidean distance with Wasserstein distance to measure sample similarity, it more accurately reflects the geometric structural relationships of data in high-dimensional space, improving the discriminative power of the graph. Third, by constructing a cross-modal tensor graph model and imposing low-rank constraints, it promotes the sharing of consistent low-dimensional latent factors among modal auxiliary graphs in the Fourier domain, achieving effective capture and deep fusion of high-order structural consistency within and between modalities. Furthermore, the alternating iterative optimization strategy enables mutual promotion and synergistic enhancement among components: kernel representation guides optimal transport learning, optimal transport graph assists in kernel representation updates, and tensor graph refines the cross-modal consistent structure, forming a closed-loop optimization. Experimental results show that the present invention significantly improves clustering accuracy and robustness on multimodal benchmark datasets, outperforming existing mainstream multimodal clustering methods. Attached Figure Description

[0011] Figure 1 This is a flowchart illustrating the multimodal data clustering method based on kernelized optimal transport tensor graphs provided in this embodiment of the invention.

[0012] Figure 2 This is a schematic diagram of the training process of a multimodal clustering model based on a kernelized optimal transport tensor graph provided in an embodiment of the present invention.

[0013] Figure 3 This is a diagram of the core algorithm framework for the collaborative optimization of kernelized representation learning, optimal transport graph learning, and cross-modal tensor graph learning provided in the embodiments of the present invention.

[0014] Figure 4 This is the main architecture diagram of the multimodal clustering model (WKGL) based on kernelized optimal transport tensor graph provided in the embodiments of the present invention.

[0015] Figure 5 This is a parameter sensitivity analysis diagram showing the changes in clustering performance under different parameter values ​​on the publicly available datasets BBC and ORL provided in this embodiment of the invention.

[0016] Figure 6 This is a graph showing the comparison of clustering performance of various algorithms provided in this embodiment of the invention on 12 benchmark datasets.

[0017] Figure 7 This is a schematic diagram of the ablation experiment provided in this embodiment of the invention: verification of the effectiveness of the core components. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0019] refer to Figure 1 , Figure 4 As shown, the first embodiment of the present invention discloses a multimodal data clustering method based on kernelized optimal transport tensor graphs, which can be executed by a multimodal data clustering device based on kernelized optimal transport tensor graphs (hereinafter referred to as clustering device), specifically, by one or more processors within the clustering device, to implement the following method: S1, Obtain the multimodal sample data to be processed, preprocess the multimodal sample data to be processed, and construct the kernel matrix of each mode; S2, input the kernel matrix of each modality into the pre-trained multimodal clustering model based on kernelized optimal transport tensor graph to generate clustering results.

[0020] Specifically, in this embodiment, the pre-trained multimodal clustering model has established a nonlinear mapping relationship between "kernelized low-dimensional representation features," "optimal transport graph structure features," and "cluster labels." By inputting the preprocessed kernel matrix data into the model, the cluster labels of the samples can be output, thereby achieving high-precision clustering under unsupervised conditions based on multimodal observation data, providing a reliable analytical basis for multimodal data scenarios such as image recognition and bioinformatics.

[0021] Please see Figure 2 , Figure 3 The training steps for the multimodal clustering model based on the kernelized optimal transport tensor graph are as follows: S01. Obtain the training multimodal dataset. Based on the preset kernel function (such as polynomial kernel, Gaussian kernel, etc.), map the original feature matrices of each modality in the training multimodal dataset to the regenerating kernel Hilbert space to construct the corresponding kernel matrix; thus implicitly capturing the nonlinear correlation within the data.

[0022] Specifically, step S01 further includes: obtaining a training multimodal dataset containing n samples and m modalities, wherein the original feature matrix under modality k is represented as: , For the set of real numbers, Let d be the feature dimension corresponding to modality k; different modalities can have different feature dimensions. For example, in an image recognition scenario, the same set of images can have color histogram features (d1=256), texture features (d2=128), and shape features (d3=64) extracted respectively.

[0023] Based on the preset kernel function, calculate the kernel matrix for k modes. Among them, the kernel matrix The Each element is defined as: , It is an implicit feature mapping function. Let i be the original eigenvector of the original feature matrix of sample i under mode k. Let J be the original eigenvector of sample j in the original feature matrix under mode k. The kernel is the inner product symbol. Kernel functions include, but are not limited to, polynomial kernels, Gaussian kernels, or linear kernels; they map the original data from the input space to a high-dimensional reproducing kernel Hilbert space. The inner product of the mapped samples is directly calculated using kernel tricks, without explicitly constructing an implicit feature mapping function, thus preserving nonlinear expressive power while avoiding the computational overhead of high-dimensional mapping. In simple terms, the constructed kernel matrix represents the nonlinear similarity relationship between samples in the kernel space, providing a richer foundation for subsequent graph learning relational metrics.

[0024] In this embodiment, the preset kernel function is preferably a polynomial kernel, defined as follows: This kernel function implicitly maps the original features to a high-dimensional reproducing kernel Hilbert space to capture second-order interactions and nonlinear structures between features in the data. It offers advantages such as computational simplicity and the elimination of parameter tuning requirements, effectively enhancing the model's ability to express nonlinear data.

[0025] S02 learns the corresponding kernelized low-dimensional representation based on the kernel matrix of each modality, and learns the similarity map of each modality based on the kernelized low-dimensional representation. This representation achieves dimensionality reduction and noise reduction while maintaining the nonlinear structure of the kernel space. It also combines self-transmission optimal transmission to learn the similarity map, and uses Wasserstein distance instead of traditional Euclidean distance to measure the relationship between samples, overcoming the shortcomings of traditional methods in characterizing complex data distributions.

[0026] Specifically, step S02 further includes: learning the kernelized low-dimensional representation under k modes by solving the kernel matrix factorization problem. Its objective function is: , Where d is the dimension after dimensionality reduction and (Preferred setting is the number of cluster categories c). For transpose, Let Frobenius norm be the square of I, and I be the identity matrix. This kernelized low-dimensional representation achieves dimensionality reduction and noise reduction while preserving the nonlinear structure of the kernel space, effectively suppressing the adverse effects of noise and unreliable similarity in the original high-dimensional kernel space on subsequent graph learning.

[0027] The physical meaning of this objective function is to find a low-dimensional representation such that the data in the original kernel space can be reconstructed from this low-dimensional representation while minimizing the reconstruction error; using kernel tricks, the kernel matrix factorization problem is simplified to... It is equivalent to maximizing By analyzing the kernel matrix under the k-mode Perform eigenvalue decomposition and select the eigenvectors corresponding to the top d largest eigenvalues ​​to construct the kernelized low-dimensional representation under the k modes. ,in, The trace of the matrix; Based on the kernelized low-dimensional representation under k-mode Solve the optimal transmission problem of self-transmission with entropy regularization and learn the similarity graph under k modes. The optimal transmission plan, also known as the optimal transmission plan, is defined by the following formula: , , , , Let be the low-dimensional representation vector of sample i under mode k. Let be the low-dimensional representation vector of sample j in mode k. The square of the L2 norm, Let be the probability quality of sample i being transmitted to sample j, reflecting the similarity relationship between samples in the k-th modality, and satisfying the double random constraint that both the row sum and column sum are 1. This is the entropy regularization coefficient (preferably set to 0.1), used to control the smoothness of the transmission plan. As the value approaches 0, the transmission plan approaches a deterministic allocation. As the number of transmissions increases, the transmission schedule tends to be more uniformly distributed.

[0028] In this embodiment, the first term is the transmission cost term, which measures the difference between samples in the low-dimensional representation space based on the Wasserstein distance. Compared with the traditional Euclidean distance, it can more accurately capture the geometric structure relationship of data in the high-dimensional space, overcoming the shortcomings of traditional methods in characterizing complex data distributions. Through the self-transmission mechanism, samples undergo quality redistribution within their own set, giving similar samples higher transmission weights, thereby learning a more discriminative similarity map.

[0029] S03. Construct a cross-modal tensor graph model, stack the auxiliary graphs of each modality along the modal dimension into a third-order tensor, and apply tensor low-rank constraints (such as tensor nuclear norm or tensor Tubal rank) to the third-order tensor based on the similarity graph; in order to capture the structural consistency shared within and between modalities, and realize cross-modal high-order information interaction.

[0030] Specifically, step S03 further includes: introducing an auxiliary diagram. , , , The auxiliary graphs for each of the k modes are used to refine the optimal transmission plan for each mode and ensure cross-modal consistency; and the auxiliary graphs for each mode are stacked along the modal dimension into a third-order tensor. , , For stacking operations, This is an auxiliary diagram for mode 1. This is an auxiliary diagram for mode 2. Auxiliary plot for mode m; Third-order tensor The three dimensions correspond to sample, modality, and sample, respectively, fully encoding the inter-sample relationships, inter-modal relationships, and cross-modal sample interactions in multimodal data; its corresponding overall optimization objective includes consistency constraints between the auxiliary graph and the optimal transport plan, Frobenius norm regularization of the auxiliary graph, and low-rank constraints of the tensor. , (Preferred setting is 1.0) (Preferred setting is 5.0), where, The trade-off / balancing parameter for controlling the regularization strength, The trade-off / balancing parameters for controlling the modal consistency strength, Let be the matrix nuclear norm, defined as the sum of all singular values ​​of the matrix.

[0031] In this embodiment, the first term ensures that the auxiliary graph does not deviate from the sample relationships learned by the optimal transmission plan; the second term is Frobenius norm regularization to prevent overfitting of the auxiliary graph; and the third term is the tensor Tubal nuclear norm, used to characterize the cross-modal low-rank structure and promote the sharing of consistent low-dimensional latent factors among auxiliary graphs of different modalities. The tensor low-rank constraint uses the tensor Tubal nuclear norm, i.e. , For tensor The block diagonal matrix obtained after performing a discrete Fourier transform along the third dimension has each diagonal block being a forward slice matrix of the tensor at the corresponding Fourier frequency. This constraint effectively captures the consistency of intramodal and intermodal high-order structures (cross-modal structures) by promoting the sharing of consistent low-dimensional latent factors among modal auxiliary graphs in the Fourier domain (the low rank in the Fourier domain implicitly forces each mode to share similar latent structure factors), enabling the model to fully utilize the complementarity and consistency information among multiple modalities.

[0032] S04 employs an alternating iterative optimization strategy to solve the sub-problems corresponding to the kernelized low-dimensional representation, similarity graph, and auxiliary graph, respectively, thereby achieving joint optimization of the three components. This allows each component to promote and enhance each other until convergence.

[0033] Specifically, step S04 further includes: the alternating iterative optimization strategy includes a T optimization step, a Z optimization step, and an A optimization step, and the three optimization steps are executed sequentially and cyclically until the change in the objective function value is less than a preset threshold or the maximum number of iterations is reached; The T optimization steps specifically involve: kernelized low-dimensional representation under fixed k modes. and auxiliary diagram The Sinkhorn-Knopp algorithm is used to solve the optimal transport problem with entropy regularization, resulting in an updated similarity graph for the k-mode. Where, the cost matrix C is defined, and its i-th Each element is defined as: , For auxiliary diagram The elements, of which, introduce The purpose of factorization is to use information from the current auxiliary graph to guide the cost definition, ensuring the non-negativity of the cost matrix, while allowing highly similar sample pairs in the auxiliary graph to obtain lower transmission costs; constructing Gibbs kernels. , The cost matrix is ​​transformed into a kernel matrix form, which is an exponential function. The scaling vector v is initialized as an all-one vector, and the scaling vector is updated iteratively. and Until convergence, each iteration essentially involves alternating row and column normalization operations to gradually satisfy the double stochastic constraint on the transfer matrix; after convergence, the updated similarity graph for the k modes is obtained. The plan satisfies the double random constraint that the row sum is 1 and the column sum is 1, and is sparsity-based, with each sample establishing strong connections with only a few samples; u and v are both scaling vectors. For diagonal matrix operators, This is an element-wise division operation; the essence of this step is to find the optimal inter-sample transfer scheme under the guidance of the current kernel representation and auxiliary graph.

[0034] The Z-optimization steps are as follows: Fix the similarity graph under the updated k-modal. By solving the trace maximization problem And satisfy orthogonal constraints The updated kernelized low-dimensional representation under the k-mode is obtained. The closed-form solution to this problem is obtained by applying the matrix... The eigenvalues ​​are obtained by performing eigenvalue decomposition, and the eigenvectors corresponding to the top d largest eigenvalues ​​are selected as the kernelized low-dimensional representations for the updated k-modes. This step incorporates nonlinear information from the kernel space (from...). ) and the sample relationships learned by optimal transmission (from This allows the low-dimensional representation to possess both nonlinearity preservation and discriminative properties.

[0035] Simply put, the objective function The maximization problem is transformed into a problem involving the matrix. Perform eigenvalue decomposition; this matrix incorporates the kernel matrix. Captured nonlinear relationships and optimal transfer matrix The learned sample similarity information enables the solution of the kernelized low-dimensional representation. It can simultaneously preserve the nonlinear structure of the original data and the discriminative relationships revealed by the Wasserstein distance; It consists of the feature vectors corresponding to the first d largest feature values, where d is a preset dimensionality reduction parameter, which is usually much smaller than the number of samples.

[0036] The optimization steps specifically involve alternating updates using the augmented Lagrange multiplier method and the tensor Tubal shrinkage operator. and (and accelerate convergence by introducing a penalty factor): fix the similarity graph under the updated k modes. and nucleation low-dimensional representation Introducing auxiliary tensors and Lagrange multipliers The problem is transformed into one with equality constraints using the augmented Lagrange multiplier method. The augmented Lagrange form is given by the formula: The process involves alternating updates (i.e., solving for closed-form solutions for each column j of each mode k; updating the auxiliary tensor using the tensor Tubal contraction operator; and updating the Lagrange multipliers and penalty factors) to obtain the updated auxiliary graph for mode k. and auxiliary tensor ,in, This is the auxiliary matrix for mode 1. This is the auxiliary matrix for mode 2. Let m be the auxiliary matrix for mode m. The penalty factor is used; this step promotes the sharing of a consistent structure among modal auxiliary graphs through tensor low-rank constraints, thereby realizing cross-modal information interaction and collaborative enhancement.

[0037] When it is determined that the change in the objective function value is less than a preset threshold or the maximum number of iterations has been reached, the auxiliary graph under the final updated k-mode is output. Similarity graph and nucleation low-dimensional representation ; When it is determined that the change in the objective function value is not less than a preset threshold (preferably set to 10), -5 If the maximum number of iterations (preferably set to 50) is not reached, a new round of alternating iteration optimization strategy is executed.

[0038] The update process includes an auxiliary graph update step and an auxiliary tensor update step; The specific steps for updating the auxiliary graph are as follows: fix the auxiliary tensor. and Lagrange multipliers To find the closed-form solution of the auxiliary graph corresponding to each column of samples j under mode k, the formula is: , The updated auxiliary graph under the k-mode is obtained. ,in, Auxiliary diagram for mode k The element corresponding to sample j in the middle, To be based on constraints The Lagrange multipliers, determined by solving the linear equations, are represented by J, which is an intermediate variable matrix. For each sample j in the intermediate variable matrix J under mode k, Similarity graph under mode k The element corresponding to sample j in the middle, P is an auxiliary tensor The matrix obtained after expanding along the modal direction This means setting negative values ​​to zero to ensure the non-negativity of the auxiliary plot; The update of the auxiliary tensor specifically involves solving for the updated auxiliary tensor using the tensor Tubal contraction operator. , , For tensors The results of t-SVD decomposition , For the contraction operation, It is a right orthogonal tensor. It is a left orthogonal tensor. For singular value tensors, Let f be an f-diagonal tensor, and its elements in the Fourier field be... This operation performs soft-threshold shrinking on the singular values ​​of each forward slice of the tensor, thereby achieving low-rank constraints. The formula for updating the Lagrange multipliers and penalty factors is as follows: , , , To accelerate convergence, a scaling factor (preferably set to 1.5) is used to force the equality constraints by gradually increasing the penalty factor. The conditions are strictly met. This alternating iterative optimization strategy enables the components to promote and enhance each other: the kernelized representation guides the optimal transport to learn more accurate sample relationships, the optimal transport graph assists in updating the kernelized representation to enhance discriminativeness, and the tensor graph refines the cross-modal consistent structure, forming a closed-loop optimization.

[0039] S05. Based on the optimized auxiliary graphs of each modality, a unified consensus graph is constructed through weighted averaging and symmetry processing to ensure the symmetry and non-negativity of the graph.

[0040] Specifically, step S05 further includes: updating the auxiliary graphs for each modality. After weighted averaging and symmetry processing, a consensus graph is obtained. The consensus graph integrates auxiliary graph information from all modalities with equal weights, and the symmetry processing ensures the graph's undirectedness, facilitating subsequent spectral clustering analysis. The edge weights in the consensus graph reflect the degree of similarity between samples after comprehensively considering information from each modality, providing a unified similarity metric for the final clustering.

[0041] S06. Input the consensus graph into the spectral clustering algorithm, obtain the low-dimensional spectral embedding by calculating the eigenvalue decomposition of the Laplacian matrix, and then use the K-means clustering algorithm to divide and obtain the final clustering result. Specifically, step S06 further includes: inputting the consensus graph into the spectral clustering algorithm to calculate its corresponding Laplacian matrix DL or normalized Laplacian matrix. D is the degree matrix; For the Laplacian matrix DL or normalized Laplacian matrix Perform eigenvalue decomposition, select the eigenvectors corresponding to the c smallest eigenvalues, and arrange them column-wise to form a spectral embedding matrix. c is the preset number of clusters (e.g., the number of clusters in a known dataset); preferably, a normalized Laplacian matrix is ​​used. The normalized Laplacian matrix can effectively handle situations where the degree of nodes in the graph varies greatly, thus improving clustering stability.

[0042] After L2 normalization of each row vector of the spectral embedding matrix, the K-means algorithm is used for clustering. Sample allocation and cluster center updates are iteratively performed until the cluster allocation no longer changes or the maximum number of iterations (e.g., 100) is reached, yielding the final cluster labels and completing the multimodal data clustering task, generating the final clustering result. Each row of the spectral embedding matrix corresponds to a low-dimensional representation of a sample in the c-dimensional spectral space, ensuring that samples belonging to the same cluster are close to each other in the spectral space. By employing multiple rounds of random initialization and selecting the optimal result, the K-means solution effectively avoids getting trapped in local optima.

[0043] S07. The final clustering results are evaluated according to preset evaluation metrics. When the evaluation results are determined to have reached the preset performance threshold, training is completed, and a trained multimodal clustering model based on kernelized optimal transport tensor graph is obtained. That is, the clustering performance is evaluated on a multimodal benchmark dataset. By calculating three metrics—clustering accuracy (ACC), normalized mutual information (NMI), and purity (PUR)—the model is comprehensively judged to determine whether it has reached the preset performance threshold until a pre-trained clustering model is obtained.

[0044] Specifically, step S07 further includes: evaluating clustering performance on a multimodal benchmark dataset, comparing the final clustering results with the true labels, and calculating three indicators: clustering accuracy, normalized mutual information, and purity based on the confusion matrix; Analyze the above metrics to determine if the evaluation results meet the preset performance threshold (i.e., if ACC, NMI, and PUR are all higher than the preset threshold (e.g., ACC > 0.85)). If yes, complete the training and determine that the model is qualified. If not, adjust the hyperparameters (e.g., dimensionality reduction dimension d, regularization parameter). and (etc.), retrain the model.

[0045] The formulas for the three indicators—clustering accuracy, normalized mutual information, and purity—are as follows: , , , , ,in, For clustering accuracy, To normalize mutual information, For purity, This is an indicator function; it takes the value 1 if the two are equal, and 0 otherwise. For the true label of sample i, The predicted cluster label for sample i, The mutual information between the true label distribution and the predicted label distribution is represented by Y, where Y is the true label of the sample. For the predicted clustering labels of the samples, The information entropy of the real label, To predict the information entropy of the label, This represents the number of samples belonging to category c. For those that simultaneously belong to both the true category c and the predicted category The number of samples, Let c be the set of samples of the true class c. For predicting categories The sample set. Purity measures the proportion of samples of the majority true class in each predicted cluster, and the value ranges from [0,1]. The closer to 1, the higher the cluster purity.

[0046] Finally, for large-scale datasets, the Nyström approximation method is used to accelerate the solution of kernelized low-dimensional representations by randomly sampling from n samples. Using samples as landmarks, an approximate kernel matrix is ​​constructed, and a low-rank approximation is used to obtain a complete eigenvector estimate, reducing the computational complexity of singular value decomposition from... Reduce to This allows the method to be extended to large-scale multimodal data scenarios.

[0047] Please see Figure 5 Specifically, in this embodiment, to further verify the effectiveness of the proposed model and its solution, 12 publicly available benchmark multimodal datasets covering multiple fields such as image, text, and bioinformatics were selected for comprehensive experimental evaluation. These 12 datasets cover different data types, representing diverse real-world application scenarios of multimodal clustering. Detailed information for each dataset is shown in Table 1.

[0048] Table 1 Summary of multimodal benchmark datasets:

[0049] The experimental hyperparameter settings for the method of this invention are shown in Table 2 below.

[0050] Table 2. Main hyperparameter settings for the experiment:

[0051] This method was used to train and test the model, and its performance was compared with other cutting-edge mainstream multimodal clustering methods on 12 publicly available benchmark datasets. The comparison methods cover a variety of representative multimodal clustering strategies, including: graph learning-based methods (GMC, RGMVC, TBGL, CTGL, SLMVGC), matrix factorization-based methods (OMSC), kernel learning-based methods (OPMC, ONALK), subspace learning-based methods (SURER), and deep clustering methods (DCMVC). Specific macroscopic test results are shown below. Figure 6 As shown, the optimal result is highlighted in bold red, and the second-best result is highlighted in bold blue. This is based on experimental comparisons. Figure 6 The results lead to the following observations: (1) As a multimodal clustering method based on kernelized optimal transport tensor graphs, the method of this invention achieves the best clustering performance on most datasets in twelve benchmark datasets (such as Scene-15, MSRC, Yale, Flowers17, etc.), and significantly outperforms existing cutting-edge multimodal clustering methods based on graph learning, subspace learning, and deep networks in various scenarios. In particular, on the Scene-15 dataset, the ACC of the method of this invention reaches 81.76%, which is 37.90 percentage points higher than the second-best tensor graph method CTGL (43.86%); on the Flowers17 dataset, the ACC reaches 91.47%, which is 28.38 percentage points higher than the second-best method CTGL (63.09%).

[0052] (2) On challenging datasets with complex feature distributions (such as BUAA and Yale), the method of this invention demonstrates superior clustering ability. On the BUAA dataset, the ACC of the method of this invention reaches 87.11% and the NMI reaches 95.46%, while the best performing method among all comparison methods, CTGL, only achieves 64.22% (ACC) and 81.82% (NMI). The ACC performance of the method of this invention is improved by more than 22 percentage points, which fully demonstrates the absolute advantage of "kerneled low-dimensional representation" in processing original complex nonlinear data and effectively achieving dimensionality reduction and noise reduction.

[0053] (3) Compared with other cutting-edge methods that also utilize tensor low-rank constraints to mine cross-modal consistent structures (such as TBGL and CTGL), the method of this invention has achieved higher clustering accuracy in more than ten of the twelve datasets (only a very small difference is considered suboptimal on a few datasets such as HandWritten), which verifies the effectiveness of introducing Wasserstein distance metric (optimal transport graph learning) and kernelized representation learning, as well as the complementary advantages of the three modules working together and promoting each other under the alternating iterative optimization framework.

[0054] To further verify the effectiveness of the two core components of this invention, kernel representation learning (KR) and Wasserstein graph learning (WG), ablation experiments were conducted. The complete model WKGL was compared with two simplified variants: (1) w / o KR: kernel representation learning was removed, and optimal transport graph learning and tensor quantization were performed directly on the original feature space; (2) w / o WG: Wasserstein graph metric was removed, and tensor quantized graph learning was performed directly on the kernel representation. The ablation experiment results on the ORL, Yale, Flowers17, and 100Leaves datasets are as follows: Figure 7 As shown. According to Figure 7 The ablation experiment results lead to the following conclusions: (1) Removing kernelized representation learning (w / oKR) resulted in a significant decrease in performance on all four datasets. On the Yale dataset, the ACC dropped from 96.97% to 87.88%, a decrease of 9.09 percentage points; on the 100Leaves dataset, the ACC dropped from 97.31% to 92.13%, a decrease of 5.18 percentage points. This fully demonstrates that kernelized low-dimensional representation can effectively capture the nonlinear relationships within the data and suppress noise while reducing dimensionality, which is crucial for improving the discriminative ability of graph learning.

[0055] (2) After removing Wasserstein graph metric learning (w / oWG), the performance on each dataset degraded more significantly. On the Flowers17 dataset, the ACC plummeted from 91.47% to 52.57%, a drop of 38.90 percentage points; on the Yale dataset, the ACC dropped from 96.97% to 70.30%, a drop of 26.67 percentage points. This verifies the key role of replacing traditional Euclidean distance with Wasserstein distance to measure the relationship between samples, indicating that optimal transport graph learning can more accurately capture sample similarity under complex data distributions.

[0056] (3) On all four datasets, the performance degradation caused by removing the Wasserstein graph metric (w / oWG) was significantly greater than that caused by removing kernelized representation learning (w / oKR), indicating that the Wasserstein distance metric plays a more central discriminative role in the overall framework. However, the performance of both variants was significantly lower than that of the complete method, verifying that kernelized representation and optimal transmission are both indispensable and work synergistically.

[0057] (4) The complete method of this invention (WKGL) achieved the best results on all metrics of the four datasets, verifying the synergistic effect of the three core modules of kernel representation learning, optimal transport graph learning and tensor graph learning and the effectiveness of the overall design.

[0058] In summary, this method can automatically extract nonlinear relationships from multimodal raw features and provide high-precision clustering results. Addressing the problems of existing multimodal clustering methods such as over-reliance on Euclidean distance, difficulty in capturing nonlinear data structures, and insufficient modeling of cross-modal high-order consistency, this method innovatively integrates kernelized representation learning, optimal transport graph learning, and tensor graph learning into a unified optimization framework. First, the raw data is mapped to a regenerating kernel Hilbert space through kernel functions, implicitly capturing the nonlinear relationships within the data. Second, optimal transport theory is introduced, replacing traditional Euclidean distance with Wasserstein distance to measure sample similarity, improving graph discrimination capabilities under complex data distributions. Third, a cross-modal tensor graph model is constructed and low-rank constraints are applied, effectively capturing high-order structural consistency within and between modalities by promoting the sharing of consistent low-dimensional latent factors among auxiliary graphs of each modality. Finally, an alternating iterative optimization strategy enables mutual promotion and synergistic enhancement among the components. Thus, this invention successfully achieves a modeling leap from "single-modal linear Euclidean metric" to "multi-modal nonlinear collaborative enhancement", significantly improving clustering accuracy and robustness on diverse benchmark datasets.

[0059] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A multimodal data clustering method based on kernelized optimal transport tensor graphs, characterized in that, include: Acquire the multimodal sample data to be processed, preprocess the multimodal sample data to be processed, and construct the kernel matrix of each mode; The kernel matrix of each modality is input into a pre-trained multimodal clustering model based on kernelized optimal transport tensor graph to generate clustering results; The training steps for the multimodal clustering model based on the kernelized optimal transport tensor graph are as follows: Obtain the training multimodal dataset, and based on the preset kernel function, map the original feature matrices of each modality in the training multimodal dataset to the regenerated kernel Hilbert space to construct the corresponding kernel matrix; Based on the kernel matrix of each modality, the corresponding kernelized low-dimensional representation is learned, and the similarity graph of each modality is learned based on the kernelized low-dimensional representation. Construct a cross-modal tensor graph model, stack the auxiliary graphs of each modality into a third-order tensor along the modal dimension, and apply a low-rank constraint to the third-order tensor based on the similarity graph; An alternating iterative optimization strategy is adopted to solve the sub-problems corresponding to the kernelized low-dimensional representation, the similarity graph, and the auxiliary graph respectively, thereby achieving joint optimization of the three. Based on the optimized auxiliary graphs of each modality, a unified consensus graph is constructed through weighted averaging and symmetry processing. The consensus graph is input into the spectral clustering algorithm. The low-dimensional spectral embedding is obtained by calculating the eigenvalue decomposition of the Laplacian matrix. Then, the K-means clustering algorithm is used to divide the data to obtain the final clustering result. The final clustering results are evaluated according to the preset evaluation indicators. When the evaluation results are determined to reach the preset performance threshold, the training is completed, and a trained multimodal clustering model based on the kernelized optimal transport tensor graph is obtained.

2. The multimodal data clustering method based on kernelized optimal transport tensor graphs according to claim 1, characterized in that, Obtain the training multimodal dataset. Based on a preset kernel function, map the original feature matrices of each modality in the training multimodal dataset to the regenerating kernel Hilbert space to construct the corresponding kernel matrix, specifically: Obtain a training multimodal dataset containing n samples and m modalities, where the original feature matrix for modality k is represented as: , For the set of real numbers, Let k be the feature dimension corresponding to mode k. Based on the preset kernel function, calculate the kernel matrix for k modes. Among them, the kernel matrix The Each element is defined as: , It is an implicit feature mapping function. Let i be the original eigenvector of the original feature matrix of sample i under mode k. Let J be the original eigenvector of sample j in the original feature matrix under mode k. This is the inner product symbol.

3. The multimodal data clustering method based on kernelized optimal transport tensor graphs according to claim 2, characterized in that, Based on the kernel matrix of each modality, the corresponding kernelized low-dimensional representation is learned, and the similarity graph of each modality is learned based on the kernelized low-dimensional representation, specifically: By solving the kernel matrix decomposition problem, we learn the kernelized low-dimensional representation under k modes. Its objective function is: , Where d is the dimension after dimensionality reduction and , For transpose, Let I be the square of the Frobenius norm, and I be the identity matrix. The kernel matrix decomposition problem is simplified to It is equivalent to maximizing By analyzing the kernel matrix under the k-mode Perform eigenvalue decomposition and select the eigenvectors corresponding to the top d largest eigenvalues ​​to construct the kernelized low-dimensional representation under the k modes. ,in, The trace of the matrix; Based on the kernelized low-dimensional representation under k-mode Solve the optimal transmission problem of self-transmission with entropy regularization and learn the similarity graph under k modes. Its formula is: , , , , Let be the low-dimensional representation vector of sample i under mode k. Let be the low-dimensional representation vector of sample j in mode k. The square of the L2 norm, Let i be the probability mass of being transmitted from sample i to sample j. is the entropy regularization coefficient.

4. The multimodal data clustering method based on kernelized optimal transport tensor graph according to claim 3, characterized in that, A cross-modal tensor graph model is constructed by stacking the auxiliary graphs of each modality into a third-order tensor along the modal dimension, and applying a low-rank constraint to the third-order tensor based on the similarity graph, specifically: Introducing auxiliary graphs , , , The auxiliary graphs for each of the k modes are then stacked along the modal dimension to form a third-order tensor. , , For stacking operations, This is an auxiliary diagram for mode 1. This is an auxiliary diagram for mode 2. Auxiliary plot for mode m; Third-order tensor The three dimensions correspond to samples, modalities, and samples, respectively. The overall optimization objective includes the consistency constraint between the auxiliary graph and the optimal transmission plan, the Frobenius norm regularization of the auxiliary graph, and the low-rank constraint of the tensor. , , ,in, To control the trade-off parameters of regularization strength, The trade-off parameters for controlling modal uniformity strength, Let be the matrix nuclear norm.

5. The multimodal data clustering method based on kernelized optimal transport tensor graph according to claim 4, characterized in that, The alternating iterative optimization strategy includes a T optimization step, a Z optimization step, and an A optimization step. The three optimization steps are executed sequentially and cyclically until the change in the objective function value is less than a preset threshold or the maximum number of iterations is reached. The T optimization steps specifically involve: kernelized low-dimensional representation under fixed k modes. and auxiliary diagram The Sinkhorn-Knopp algorithm is used to solve the optimal transport problem with entropy regularization, resulting in an updated similarity graph for the k-mode. Where, the cost matrix C is defined, and its i-th Each element is defined as: , For auxiliary diagram The Each element constructs the Gibbs core. , The cost matrix is ​​transformed into a kernel matrix form, which is an exponential function. The scaling vector v is initialized as an all-one vector, and the scaling vector is updated iteratively. and Until convergence, after iterative convergence, the updated similarity graph for the k modes is obtained. u and v are both scaling vectors. For diagonal matrix operators, This is an element-wise division operation; The Z-optimization steps are as follows: Fix the similarity graph under the updated k-modal. By solving the trace maximization problem And satisfy orthogonal constraints The updated kernelized low-dimensional representation under the k-mode is obtained. The closed-form solution to this problem is obtained by applying the matrix... The eigenvalues ​​are obtained by performing eigenvalue decomposition, and the eigenvectors corresponding to the top d largest eigenvalues ​​are selected as the kernelized low-dimensional representations for the updated k-modes. ; The specific optimization steps are as follows: fix the similarity graph under the updated k-modalities. and nucleation low-dimensional representation Introducing auxiliary tensors and Lagrange multipliers The problem is transformed into one with equality constraints using the augmented Lagrange multiplier method. The augmented Lagrange form is given by the formula: Then, perform alternating update processing to obtain the updated auxiliary graph under the k-mode. and auxiliary tensor ,in, This is the auxiliary matrix for mode 1. This is the auxiliary matrix for mode 2. Let m be the auxiliary matrix for mode m. As a penalty factor; When it is determined that the change in the objective function value is less than a preset threshold or the maximum number of iterations has been reached, the auxiliary graph under the final updated k-mode is output. Similarity graph and nucleation low-dimensional representation ; When it is determined that the change in the objective function value is not less than the preset threshold or the maximum number of iterations has not been reached, a new round of alternating iteration optimization strategy is executed.

6. The multimodal data clustering method based on kernelized optimal transport tensor graph according to claim 5, characterized in that, Alternating update processing includes an auxiliary graph update step and an auxiliary tensor update step; The specific steps for updating the auxiliary graph are as follows: fix the auxiliary tensor. and Lagrange multipliers To find the closed-form solution of the auxiliary graph corresponding to each column of samples j under mode k, the formula is: , The updated auxiliary graph under the k-mode is obtained. ,in, Auxiliary diagram for mode k The element corresponding to sample j in the middle, To be based on constraints The Lagrange multipliers, determined by solving the linear equations, are represented by J, which is an intermediate variable matrix. For each sample j in the intermediate variable matrix J under mode k, Similarity graph under mode k The element corresponding to sample j in the middle, P is an auxiliary tensor The matrix obtained after expanding along the modal direction This means setting negative values ​​to zero to ensure the non-negativity of the auxiliary plot; The update of the auxiliary tensor specifically involves solving for the updated auxiliary tensor using the tensor Tubal contraction operator. , , , , For the contraction operation, It is a right orthogonal tensor. It is a left orthogonal tensor. For singular value tensors, It is an f-diagonal tensor; The formula for updating the Lagrange multipliers and penalty factors is as follows: , , , The scaling factor is used to accelerate convergence.

7. The multimodal data clustering method based on kernelized optimal transport tensor graphs according to claim 6, characterized in that, Based on the optimized auxiliary graphs of each modality, a unified consensus graph is constructed through weighted averaging and symmetry processing. Specifically, this involves updating the auxiliary graphs of each modality. After weighted averaging and symmetry processing, a consensus graph is obtained. .

8. The multimodal data clustering method based on kernelized optimal transport tensor graph according to claim 7, characterized in that, The consensus graph is input into a spectral clustering algorithm. Low-dimensional spectral embeddings are obtained by calculating the eigenvalue decomposition of the Laplacian matrix. Then, the K-means clustering algorithm is used to partition the data to obtain the final clustering results. Specifically: Input the consensus graph into the spectral clustering algorithm to calculate its corresponding Laplacian matrix (DL) or normalized Laplacian matrix. D is the degree matrix; For the Laplacian matrix DL or normalized Laplacian matrix Perform eigenvalue decomposition, select the eigenvectors corresponding to the c smallest eigenvalues, and arrange them column-wise to form a spectral embedding matrix. c is the preset number of cluster categories; After L2 normalization of each row vector of the spectral embedding matrix, the K-means algorithm is used for clustering. The sample allocation and cluster center update are performed iteratively until the cluster allocation no longer changes or the maximum number of iterations is reached, and the final cluster labels are obtained, thus completing the clustering task of multimodal data and generating the final clustering results.

9. The multimodal data clustering method based on kernelized optimal transport tensor graph according to claim 1, characterized in that, The final clustering results are evaluated based on preset evaluation indicators, specifically as follows: Clustering performance was evaluated on a multimodal benchmark dataset. The final clustering results were compared with the true labels, and three metrics, namely clustering accuracy, normalized mutual information, and purity, were calculated based on the confusion matrix. Analyze the above indicators to determine whether the evaluation results have reached the preset performance threshold. If yes, complete the training and determine that the model is qualified. If not, adjust the hyperparameters and retrain the model.

10. The multimodal data clustering method based on kernelized optimal transport tensor graph according to claim 9, characterized in that, The formulas for the three indicators—clustering accuracy, normalized mutual information, and purity—are as follows: , , , , ,in, For clustering accuracy, To normalize mutual information, For purity, For indicator functions, For the true label of sample i, The predicted cluster label for sample i, The mutual information between the true label distribution and the predicted label distribution is represented by Y, where Y is the true label of the sample. For the predicted clustering labels of the samples, The information entropy of the real label, To predict the information entropy of the label, This represents the number of samples belonging to category c. For those that simultaneously belong to both the true category c and the predicted category The number of samples, Let c be the set of samples of the true class c. For predicting categories The sample set.

Images