federated multi-modal clustering method based on self-supervised multi-scale kernel diffusion graph learning

Abstract

The application provides a federated multi-modal clustering method based on self-supervised multi-scale kernel diffusion graph learning, relates to the technical field of distributed machine learning and data mining, and proposes an end-cloud collaborative framework: a client calculates an intra-class and inter-class scatter matrix by using consistent pseudo-labels issued by a server, learns Mahalanobis distance measurement, generates a square anchor graph of discriminative norm and uploads the same, and protects data privacy; a server calculates a graph Laplacian matrix on each modal anchor graph, generates a heat kernel representation on multiple diffusion time scales to filter noise, obtains a consensus kernel matrix through entropy regularization weighted fusion, and then performs logarithmic pullback and Laplace cone projection with sparsity and algebraic connectivity constraints to construct a global consensus graph and update the pseudo-labels. Iterative optimization is performed until convergence, and high-quality cross-modal clustering results are output. The application achieves better clustering performance than existing methods on multiple benchmark datasets.

Description

Technical Field

[0001] This invention relates to the field of distributed machine learning and data mining technology, specifically to a federated multimodal clustering method based on self-supervised multi-scale kernel diffusion graph learning. Background Technology

[0002] Multimodal data is widely used in real-world applications such as bioinformatics, computer vision, and social media analytics. For example, a single patient may possess multiple omics data, including DNA methylation, RNA expression, miRNA, and medical imaging data; similar news events are often presented in multiple formats, such as text, images, and videos. This data, originating from different sources or possessing different feature representations, is called multimodal data, each capturing different aspects of the observed object. Effectively utilizing the complementarity and consistency information among heterogeneous features is key to improving learning performance.

[0003] Multimodal clustering aims to partition data into different clusters in an unsupervised manner, leveraging the consistency and complementarity of multimodal data. To achieve this goal, researchers have proposed various representative methods, including matrix factorization-based, kernel-based, and graph-based methods. Among these, graph-based methods demonstrate superior clustering performance by mining the underlying common structures of multimodal data and capturing the similarity relationships between data points to learn the optimal graph. For example, one graph learning method uses a modality-weighted graph learning strategy to construct a unified graph, capturing intra-modal and inter-modal relationships to improve clustering accuracy; another graph-based method simultaneously and explicitly mines the consistency and diversity of multimodal data within a unified framework. In recent years, some graph-based learning methods have further explored using manifold topology for multimodal graph clustering.

[0004] Despite the remarkable performance of existing graph learning methods in multimodal clustering tasks, they still face several significant challenges. First, most existing methods are based on the assumption of centralized data processing, requiring the aggregation of raw data from all participants to a central server for joint modeling. However, due to increasingly stringent data privacy regulations (such as the General Data Protection Regulation (GDPR)) and data silos between institutions, centralized multimodal clustering is often difficult to implement in practice, and the phenomenon of data silos severely restricts the effective utilization of multimodal data. Second, from the perspective of graph learning algorithm mechanisms, traditional methods typically rely on similarity measures based on Euclidean distance, which struggles to fully exploit the complex nonlinear relationships and inherent distribution structures of real-world data. This results in insufficient discriminative power in the learned graph structure, failing to accurately reflect the true relationships between data points. Furthermore, in distributed scenarios, the data modalities of each client are heterogeneous. How to construct a structurally effective, noise-resistant, and interconnected robust global consensus graph from these heterogeneous local graph structures is another core challenge facing federated multimodal clustering.

[0005] In view of the above, this application is hereby submitted. Summary of the Invention

[0006] This invention provides a federated multimodal clustering method based on self-supervised multi-scale kernel diffusion graph learning, which can at least partially improve the above-mentioned problems.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A federated multimodal clustering method based on self-supervised multi-scale kernel diffusion graph learning includes: Step S1: Control each client to construct a mode-specific anchor graph based on its local corresponding single-modal data, and upload the anchor graph to the server. Control the server to perform preprocessing and spectral clustering based on the anchor graph to generate a global consensus graph and consistent pseudo-labels. Step S2: The driver server sends the consistency pseudo-label to each client, controls each client to relearn based on the consistency pseudo-label and the end-to-cloud collaboration mechanism, updates the anchor map, and uploads the updated anchor map to the server. Step S3: Drive the server to perform multi-scale kernel diffusion fusion and Laplacian cone projection processing on the updated anchor graph, update the global consensus graph and consistency pseudo-label, and obtain the updated global consensus graph and updated consistency pseudo-label. Step S4: Repeat steps S2 to S3 for iterative optimization until the preset convergence condition is met, and extract and generate cross-modal clustering results based on the global consensus graph obtained from the last update.

[0009] In summary, this invention addresses the problems of insufficient discriminative power of client-side local graph structures and the difficulty of the server side in constructing a robust global consensus graph from heterogeneous multimodal graph structures in existing federated multimodal clustering, and proposes an end-to-cloud collaborative clustering framework scheme.

[0010] At the client level, each client stores data of different modalities. After receiving the consistency pseudo-labels from the server, the client first uses the pseudo-labels to calculate the anchor-class affinity matrix, and then constructs the intra-class scatter matrix and inter-class scatter matrix accordingly. In addition, the client learns a Mahalanobis distance metric with semantic discriminative power. Based on the Mahalanobis metric, each client solves the constrained anchor graph optimization problem, generates a modality-specific anchor graph with enhanced discriminative power, and uploads it to the server. In this way, cross-modal consistency information is fully explored while protecting the privacy of the original data. At the server level, after receiving anchor graphs from different modalities, the server first constructs the graph Laplacian matrix corresponding to each modality and calculates its hot kernel representation at multiple preset diffusion time scales, using hot kernel operators to filter high-frequency noise. Next, by learning optimal scale weights, it performs weighted fusion of hot kernels from all modalities and all scales to obtain a preliminary cross-modal consensus kernel matrix. A logarithmic pullback operation is performed on the consensus kernel matrix, mapping it back from the kernel space to the Laplacian space. Then, through Laplacian cone projection with sparsity and algebraic connectivity constraints, a structurally effective, noise-resistant, and connectivity-preserving global consensus graph is finally generated. The server updates the consistency pseudo-label based on this consensus graph and distributes it to each client, entering the next iteration until convergence. This invention effectively improves the discriminative ability of single-modal local graph structures through client-side self-supervised Markov anchor graph learning; and significantly enhances the robustness and consistency of the global cross-modal consensus graph through multi-scale kernel diffusion fusion and structural constraint projection on the server side. Attached Figure Description

[0011] Figure 1 This is a flowchart illustrating the federated multimodal clustering method based on self-supervised multi-scale kernel diffusion graph learning provided in this embodiment of the invention.

[0012] Figure 2 This is a schematic diagram of the update process of the global consensus graph and the consistency pseudo-label provided in the embodiments of the present invention.

[0013] Figure 3 This is a graph showing the changes in clustering performance under different parameter settings on the MSRC and ORL datasets provided in this embodiment of the invention.

[0014] Figure 4 This is a graph showing the comparison of clustering performance of various algorithms provided in this embodiment of the invention on 10 benchmark datasets.

[0015] Figure 5These are ablation experiment results on five datasets for various variants of the federated multimodal clustering method based on self-supervised multi-scale kernel diffusion graph learning, as provided in this embodiment of the invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0017] refer to Figure 1 As shown, the first embodiment of the present invention discloses a federated multimodal clustering method based on self-supervised multi-scale kernel diffusion graph learning, which can be executed by a federated multimodal clustering device based on self-supervised multi-scale kernel diffusion graph learning (hereinafter referred to as clustering device), specifically, by one or more processors within the clustering device, to implement the following method: Step S1: Control each client to construct a mode-specific anchor graph based on its local corresponding single-modal data, and upload the anchor graph to the server. Control the server to perform preprocessing and spectral clustering based on the anchor graph to generate a global consensus graph and consistent pseudo-labels. Specifically, step S1 further includes: acquiring the corresponding local single-mode data, directly using Euclidean distance. Construct anchor graphs, where, For the sample In modality The feature matrix of local single-modal data under, anchor point In modality The anchor point set below, It is the square of the L2 norm.

[0018] Anchor weights are solved by the following closed-form formula. The server calculates an initial global consensus graph based on the initial anchor graphs uploaded by each client, through multi-scale kernel diffusion fusion and Laplacian cone projection, and performs spectral clustering to obtain initial consistent pseudo-labels, thus initiating the subsequent self-supervised iterative optimization process.

[0019] In this embodiment, during the first round of communication, each client selects an anchor point set based on its local single-modal data feature matrix, calculates the similarity between the data sample and the anchor point using Euclidean distance, solves an optimization problem with non-negativity constraints and row sum-one constraints, constructs a modality-specific initial anchor graph, and uploads it to the server. After receiving the multimodal initial anchor graphs uploaded by each client, the server constructs a complete graph for each modality and calculates the normalized symmetric graph Laplacian matrix. At multiple preset diffusion time scales, the server calculates the heat kernel representation corresponding to the Laplacian matrix of each modality graph. By solving an optimization problem with entropy regularization, the server adaptively weights and fuses the heat kernels of all modalities and all scales to obtain a consensus kernel matrix. The server performs a logarithmic pullback operation on the consensus kernel matrix to map it to the graph Laplacian space and projects it onto a Laplacian cone constrained by sparsity and algebraic connectivity to generate an initial global consensus graph. The server performs spectral clustering based on this initial global consensus graph to obtain initial consistency pseudo-labels. It should be noted that the client and server only exchange anchor graphs and consistent pseudo-labels, without transmitting any raw data, sample indexes, or raw features, in order to ensure the data privacy of all participants.

[0020] In step S2, the server sends the consistency pseudo-label to each client, controls each client to relearn based on the consistency pseudo-label and the end-to-cloud collaboration mechanism, updates the anchor graph, and uploads the updated anchor graph to the server. Throughout the federated learning process, the client and server only interact with the anchor graph matrix and consistency pseudo-label, without transmitting any original data, sample indexes, or original features, thereby ensuring the data privacy of all participants.

[0021] Specifically, step S2 further includes: for mode k, controlling the client to obtain the feature matrix of the local single-mode data. In this process, the K-means clustering algorithm is used to select cluster centers as the anchor point set. , This is the feature matrix of local single-mode data under mode k. For the anchor point set under mode k, For the set of real numbers, For the sample set, For feature dimension, This refers to the number of anchor points; Based on the consistent pseudo-label y issued by the server, generate the one-hot encoding matrix under mode k. Wherein, the consistent pseudo-label corresponding to sample i If and only if At that time, the one-hot encoded value of sample i for class j under mode k c is the preset number of categories; Anchor graph based on mode k One-hot encoding matrix under mode k This is used to quantify the support strength of each anchor point for each category; the anchor point-category affinity matrix under mode k is calculated. , , This is the transpose; the matrix quantifies the support strength of each anchor point for each category.

[0022] Based on the anchor-class affinity matrix under mode k Calculate the corresponding intra-class scatter matrix respectively. and inter-class scatter matrix ,in, For the sample In modality A set of multiple nearest neighbor anchor points (preferably 5). In modal anchor point Consistent pseudo-labels for sample i The affinity is used to accumulate the local geometric deviation between the sample and anchor points of the same class with affinity as the weight, capturing the intra-class manifold structure; the inter-class scatter matrix accumulates the inter-class structural difference between the sample and anchor points of different classes with complementary weights.

[0023] Combined with regularization parameters , (Preferred setting is 10) -3 Construct a Mahalanobis distance metric matrix with semantic discriminative power. , It is the identity matrix; Calculate Mahalanobis distance using the Mahalanobis distance metric matrix. Using Mahalanobis distance instead of Euclidean distance, we solve the anchor graph optimization problem with non-negativity constraints and row sum-to-one constraints, for the anchor graph under mode k. Update.

[0024] The anchor graph optimization problem has a closed-form solution. Under mode k, sample i and anchor point... The formula for calculating the anchor weights between them is: , For the sample under mode k After sorting the Mahalanobis distances to all anchor points in ascending order, the [number]th Smallest value (only the nearest one) Anchor points (preferred) (The weights of the anchor points are non-zero, while the weights of the remaining anchor points are zero.) This closed-form solution guarantees the sparsity and efficient computation of the anchor graph, while avoiding the communication latency caused by iterative optimization. K is the number of anchor points in the nearest neighbor set. For anchor point K+1 in mode The anchor point set below, anchor point In modality The anchor point set below, For the sample under mode k After sorting the Mahalanobis distances to all anchor points in ascending order, the [number]th Small value, For the sample under mode k After sorting the Mahalanobis distances to all anchor points in ascending order, the [number]th Small value.

[0025] In this embodiment, the server distributes the consistency pseudo-labels for the current round to each client. Each client calculates the anchor-class affinity matrix based on the current anchor graph and the received pseudo-labels, and accordingly calculates the intra-class scatter matrix and inter-class scatter matrix. A Mahalanobis distance metric matrix is ​​then constructed using regularization parameters. The client replaces the Euclidean distance with the Mahalanobis distance to resolve the anchor graph optimization problem, generating an updated modality-specific anchor graph and uploading it to the server. It should be noted that during the initialization phase of the first round of communication, the clients have not yet received the consistency pseudo-labels from the server; therefore, Euclidean distance is used to construct the initial anchor graph. In subsequent iterative optimization phases, each client uses the consistency pseudo-labels distributed by the server to construct a Mahalanobis distance metric, replacing the traditional Euclidean distance for self-supervised anchor graph learning. Throughout the federated learning process, the client and server only exchange anchor graph matrices and consistency pseudo-labels, without transmitting any original data, sample indexes, or original features, thus ensuring the data privacy of all participants.

[0026] Please see Figure 2 Step S3: Drive the server to perform multi-scale kernel diffusion fusion and Laplacian cone projection processing on the updated anchor graph, update the global consensus graph and consistency pseudo-label, and obtain the updated global consensus graph and updated consistency pseudo-label. Specifically, step S3 further includes: S31. Construct the corresponding complete graph based on the updated anchor graph, calculate the normalized symmetric graph Laplacian matrix corresponding to the complete graph, and calculate the normalized symmetric graph Laplacian matrix for each graph on multiple preset diffusion time scales to generate multiple heat kernel representations. Specifically, step S31 further includes: receiving anchor graphs for all modes. Construct a complete graph for the anchor graph of each modality. , m is the number of modes. For modality The complete image below, For degree matrix, This represents the operation of converting a vector into a diagonal matrix, where 1 represents a vector consisting entirely of 1s; the anchor graph describes the similarity relationship between samples and anchor points, while the full graph recovers the pairwise similarity relationships between all samples.

[0027] According to mode The complete image below Calculate the corresponding normalized symmetric graph Laplacian matrix. , For modality The normalized symmetric Laplace matrix is ​​obtained below; this normalized Laplace matrix has good spectral properties, and its eigenvalues ​​are in the range of [0,2], which is beneficial to the numerical stability of subsequent thermonuclear diffusion.

[0028] A set of preset diffusion timescales is defined, using equal or logarithmic intervals (to account for both local and global graph structure information across multiple scales); preferably, it is set to {0.1, 0.5, 1.0, 2.0, 5.0, 10.0}, for a total of [number missing]. Scale. Diffusion at multiple scales can take into account both local and global graph structure information: small scales preserve fine-grained local structure, while large scales capture coarse-grained global topology; Based on a pre-set diffusion timescale set For each mode and the s-th diffusion timescale The formula for calculating the heat nucleus representation is as follows: S is the number of diffusion timescales. For modality The thermal nucleus representation corresponding to the s-th diffusion timescale. This is a matrix exponential operation. The hot kernel operator utilizes the low-pass filtering properties to suppress high-frequency noise components corresponding to larger eigenvalues ​​in the graphical Laplacian matrix through exponential decay, while retaining low-frequency components that reflect the global structure, effectively improving robustness to noise modes.

[0029] S32, by solving the alternating optimization problem with entropy regularization, learns the optimal weights for the diffusion time scale, and adaptively weights and fuses the hot kernel representations under all modes and all scales to obtain the final consensus kernel matrix; Specifically, step S32 further includes: constructing an entropy-regularized optimization problem as the objective function for adaptively fusing hot kernels across all modalities and scales, with the following formula: , , , The weights corresponding to the s-th diffusion timescale are: For consensus kernel matrix, As the weight of the diffusion timescale, It is the Frobenius norm. For matrix logarithm operations, To control the regularization parameter of the weight distribution entropy (preferably set to 1.0); entropy regularization term Encourage uniform weight distribution to avoid the model from relying too much on a single scale.

[0030] The optimization problem with entropy regularization is solved using an alternating optimization strategy (i.e., scale weights are introduced to weight and fuse hot kernels of all modes and all scales to obtain a preliminary consensus kernel matrix). The solution includes the first update and the second update. The first update is: when the weights are fixed on the diffusion timescale. Update consensus kernel matrix When the objective function degenerates into a weighted average problem, its closed-form solution is: ; The second update is: when the consensus kernel matrix is ​​fixed. Update the weights of the diffusion timescale At that time, the reconstruction loss at each scale is defined as Using the Lagrange multiplier method, the closed expression is obtained as follows: , Let be the reconstruction loss corresponding to the s-th diffusion timescale. For the first Reconstruction loss corresponding to each diffusion timescale; The first and second updates are performed alternately until the weight change on the diffusion timescale is lower than a preset threshold (preferably 10). -4 If the maximum number of alternation iterations is reached (preferably 20), the consensus kernel matrix updated last time will be used. As the final consensus core matrix.

[0031] S33, perform a logarithmic pullback operation on the consensus kernel matrix to restore it from the kernel space mapping to the corresponding intermediate graph Laplace matrix; Specifically, step S33 further includes: [The text abruptly ends here, likely due to an incomplete sentence or a formatting error.] Mapping back from the kernel space to the graph Laplacian space to impose graph structure constraints on the final consensus kernel matrix. Perform a logarithmic pullback operation to calculate the intermediate plot Laplacian matrix. , , This is the weighted average diffusion time. This operation is for thermonuclear mapping. The inverse operation will restore the consensus representation in kernel space to the graph structure representation in Laplace space, laying the foundation for applying structural constraints to the graph subsequently.

[0032] S34. Project the intermediate graph Laplacian matrix onto the Laplacian cone constrained by sparsity and algebraic connectivity, and solve it by alternating direction multiplier method to obtain the final global consensus graph. Specifically, step S34 further includes: constructing a constrained Laplacian cone projection optimization problem as the optimization objective function, and generating a structurally efficient, sparse, and connected global consensus graph. Its formula is: , , , , ,in, For the Laplace cone constraint set, The sparsity regularization parameter (preferably set to 10) -3 ), Element level Norms are used to encourage the generated consensus graph to have a sparse edge structure, thereby improving the interpretability and computational efficiency of the graph. Lower bound for algebraic connectivity (preferably set to 10) -4 To ensure the connectivity of the global consensus graph, L is a placeholder in the constraint set. For the first Line number Column elements, constraints This indicates that the graph is symmetric and constrained. This indicates a constraint that guarantees the row sum is zero (a property of the Laplace matrix). This indicates that off-diagonal elements are guaranteed to be non-positive, constraining... This indicates that the second smallest eigenvalue is not less than the lower bound of algebraic connectivity. That is, the graph is connected; The constrained Laplace cone projection optimization problem is solved using the Alternating Direction Method of Multipliers (ADMM) (i.e., decoupling the original problem into an alternating update of the spectral projection subproblem and the sparse Laplace projection subproblem). An auxiliary variable Z and a dual variable U are introduced, and the constrained Laplace cone projection optimization problem is restated as follows: , , For Laplace cone constraint set Indicator functions, It is an indicator function for element-level non-positive constraints; The corresponding augmented Lagrangian function is obtained: , , This is the penalty parameter (preferably set to 1.0). The solution process of the alternating direction multiplier method requires alternating execution of the following three updates until the update convergence condition is met (preferably, both the original residual and the dual residual are less than 1). 10-4 ), and the last updated global consensus graph As the final global consensus graph; The first update addresses the spectral projection problem by updating the global consensus graph. With the auxiliary variable Z and the dual variable U fixed, the solution formula is: ,make and perform feature decomposition on it. Projecting the eigenvalues ​​as: Reconstruction yields , For the first The global consensus graph of the next iteration. For spectral constraint set, For the first Auxiliary variables for the next iteration For the first The dual variable of the next iteration, A, is the intermediate matrix. The eigenvector matrix, It is an eigenvalue diagonal matrix. For the first 1 eigenvalue, For the first The projection yields a set of eigenvalues; this projection guarantees that the first eigenvalue is zero (corresponding to an all-1 eigenvector), and the remaining eigenvalues ​​are not less than [value missing]. This ensures the connectivity of the graph.

[0033] In simple terms, the spectral projection subproblem is: after eigenvalue decomposition of the target matrix, project the eigenvalues ​​as... The refactoring yields the updated version. The sparse Laplace projection subproblem involves applying a soft thresholding operator to off-diagonal elements in conjunction with non-positive constraints for updating, while diagonal elements are determined by the condition that the row sum is zero.

[0034] The second update addresses the sparse Laplace projection problem by updating the auxiliary variable Z and fixing the global consensus graph. For the dual variable U, the soft threshold operator combined with non-positive constraints is applied to off-diagonal elements, and the formula is as follows: The diagonal elements are determined by the condition that the sum of the rows is zero, and the formula is: , For the first The iteration of the ... Line number The auxiliary variables corresponding to the columns, For soft thresholding operators, For the first The iteration of the ... Line number The corresponding global consensus graph, For the first The iteration of the ... Line number The dual variable corresponding to the column, For the first The iteration of the ... Line number The auxiliary variables corresponding to the columns, For the first The iteration of the ... Line number The auxiliary variables corresponding to the columns; among them, the formula for the soft threshold operator is: This step also achieves sparsity (through...) Soft thresholds and off-diagonal non-positive constraints.

[0035] The third update is to update the dual variable U, and its formula is: , For the first The dual variable of the next iteration For the first Auxiliary variables for each iteration.

[0036] S35 performs spectral clustering on the final global consensus graph to obtain updated consistency pseudo-labels.

[0037] Specifically, in this embodiment, the updated global consensus graph is decomposed into features. The feature vectors corresponding to the c smallest feature values ​​are used to form a spectral embedding matrix. Then, K-means clustering is performed on the row vectors of this matrix to obtain the updated consistency pseudo-labels. Subsequently, the server distributes the updated consistency pseudo-labels to each client. After receiving the updated pseudo-labels, each client uses them to recalculate the anchor-class affinity matrix, intra-class scatter matrix, and inter-class scatter matrix, updates the Mahalanobis distance metric matrix, and then resolves the anchor graph optimization problem to generate the next round of modality-specific anchor graphs.

[0038] In this embodiment, the server receives the updated multimodal anchor graph and repeatedly performs complete graph construction, graph Laplacian matrix calculation, multi-scale hot kernel representation generation, weighted fusion, and logarithmic pullback operations. It then alternately solves the spectral projection subproblem and the sparse Laplacian projection subproblem using the alternating direction multiplier method to complete the Laplacian cone projection and generate an updated global consensus graph. The server then performs spectral clustering based on this updated consensus graph to obtain updated consistency pseudo-labels. It should be noted that in the first round of communication, the server performs multi-scale kernel diffusion fusion and Laplacian cone projection on the initial anchor graphs uploaded by each client to generate an initial global consensus graph. Spectral clustering is then performed on this initial consensus graph to obtain initial consistency pseudo-labels. In each subsequent iteration, the server re-executes spectral clustering based on the global consensus graph generated in the current round to update the consistency pseudo-labels.

[0039] Step S4: Repeat steps S2 to S3 for iterative optimization until the preset convergence condition is met, and extract and generate cross-modal clustering results based on the global consensus graph obtained from the last update.

[0040] In this embodiment, the steps in the iterative optimization phase (i.e., steps S2 to S3) are repeated until the convergence conditions of the pseudo-label change rate of adjacent rounds being lower than a preset threshold, the change amount of the global consensus graph being lower than a preset threshold, or the maximum number of communication rounds being reached are met. Based on the finally generated global consensus graph, the cross-modal clustering results are extracted and output.

[0041] Specifically, step S4 further includes: the preset convergence condition includes: the rate of change of the consistent pseudo-label between two adjacent iterations is lower than a preset threshold (preferably 10). -3 ), or the change in the global consensus graph Below a preset threshold (preferably 10) -3 The process continues until the maximum number of communication rounds is reached (preferably 50 rounds). After iterative convergence, the server outputs cross-modal clustering results based on the final generated global consensus graph.

[0042] In this embodiment, the performance of the cross-modal clustering results output by the global consensus graph can be evaluated based on evaluation metrics. These metrics include at least one of clustering accuracy (ACC), normalized mutual information (NMI), and purity (PUR). When a preset threshold is reached for an evaluation metric, the model is deemed to have passed training. The specific calculation rules for each evaluation metric are as follows: Clustering accuracy formula: , The optimal label mapping obtained by the Hungarian algorithm; normalized mutual information formula: , For mutual information, Information entropy; Purity formula: , and These are the real cluster and the predicted cluster, respectively.

[0043] Please see Figure 3 Specifically, in this embodiment, to verify the effectiveness of the proposed method, ten publicly available benchmark datasets in the field of multimodal clustering were selected for comprehensive experimental evaluation. Detailed information for each dataset is shown in Table 1.

[0044] Table 1 Summary of multimodal benchmark datasets:

[0045] These ten datasets cover a variety of data types, including images, text, and biological data, such as MSRC (multi-feature images), ORL (face images), Flowers17 (flower images), 100Leaves (plant leaves), BBCSport (sports news text), BBC (news text), NewsGroup (newsgroup text), Prokaryotic (prokaryotes), 3Sources (news text), and BDGP (Drosophila gene expression), representing diverse real-world applications of multimodal clustering. The experimental hyperparameter settings for this method are shown in Table 2 below.

[0046] Table 2. Main hyperparameter settings for the experiment:

[0047] Thanks to the self-supervised Mahalanobis anchor graph learning mechanism employed in this invention, the client only needs to upload the anchor graph matrix to achieve the sharing of local structural information, eliminating the need to transmit the original high-dimensional data and significantly reducing communication overhead. Simultaneously, the server-side multi-scale kernel diffusion fusion and ADMM cone projection possess closed-form solutions or fast convergence properties, allowing the overall framework to run efficiently on conventional computing devices and demonstrating good engineering deployment feasibility. Using this method, we compared its performance with existing cutting-edge centralized and federated multimodal clustering methods on ten public datasets. The comparison methods include: centralized methods SLMVGC, LMTC, OMSC, GMC, and OPMC, and federated methods FedMVFL, FedMVFCM, FedMVFPC, and CeFMC. Specific macroscopic test results are as follows: Figure 4 As shown, the best result is marked in red, and the second-best result is marked in blue.

[0048] according to Figure 4The results lead to the following observations: (1) As a federated multimodal clustering method, this method achieves the best clustering performance on seven out of ten datasets (such as MSRC, 3Sources, Flowers17, etc.), and significantly outperforms existing state-of-the-art centralized and federated multimodal clustering methods in most scenarios. In particular, on the MSRC dataset, the ACC of this method reaches 90.48%, which is 5.72 percentage points higher than the second-best centralized method OPMC (84.76%) and 3.34 percentage points higher than the federated method CeFMC (87.14%). (2) On challenging high-class datasets, this method demonstrates excellent clustering ability. On the 100Leaves dataset (100 categories), the ACC of this method reached 87.11% and the NMI reached 95.46%, while the best-performing FedMVFCM among all compared federated methods was only 42.69% (ACC) and 71.87% (NMI). The ACC performance of this method is more than doubled, which fully demonstrates the advantages of the multi-scale kernel diffusion fusion mechanism in handling complex fine-grained clustering tasks. (3) Compared with CeFMC, a federated method that also utilizes self-supervised pseudo-label information, this method achieved higher clustering accuracy on nine out of ten datasets (only slightly better on the BDGP dataset), which verifies the effectiveness of client-side Mahalanobis distance metric learning and server-side multi-scale hot kernel diffusion fusion, as well as the complementary advantages of their collaborative work.

[0049] To further verify the effectiveness of each key module in this method, ablation experiments were conducted. The ablation experiment variants were set as follows: (1) w / o SpML: The client's self-supervised Mahalanobis metric learning module was removed, and the client always used Euclidean distance to construct anchor graphs, retaining only the server-side multi-scale kernel diffusion fusion; (2) w / o MsKD: The server-side multi-scale kernel diffusion fusion mechanism was removed, and the graph Laplacian matrix constructed by fusing each modality anchor graph was replaced with simple averaging; (3) w / o MS: The multi-scale mechanism was removed, and only kernel diffusion fusion with a single diffusion scale (set S=1) was retained. The ablation experiments were conducted on five representative datasets: Prokaryotic, MSRC, ORL, 3Sources, and BDGP. The results are as follows. Figure 5 As shown.

[0050] according to Figure 5The ablation experiment results lead to the following conclusions: (1) Removing client-side self-supervised Mahalanobis metric learning (w / o SpML) significantly reduced performance on all five datasets. On the Prokaryotic dataset, the ACC dropped from 69.33% to 54.45%, a decrease of 14.88 percentage points. This fully demonstrates that the client's use of server pseudo-labels to learn Mahalanobis distance metric can effectively enhance the semantic discriminative power of anchor graphs, which is crucial for improving overall clustering performance. (2) Removing the server-side multi-scale kernel diffusion fusion mechanism (w / o MsKD) also resulted in a significant performance degradation on each dataset, verifying the key role of multi-scale hot kernel diffusion in filtering high-frequency noise and extracting cross-modal consensus structures. (3) Removing the multi-scale mechanism and retaining only a single diffusion scale (w / o MS) resulted in performance that was better than the first two variants (w / o SpML and w / o MsKD) on most datasets, but was still significantly lower than the complete method, indicating that multi-scale diffusion makes an important contribution to the final clustering performance. (4) The complete method (Ours) achieved the best results on all metrics of the five datasets, verifying the synergistic effect of the two core modules of client-side self-supervised Markov learning and server-side multi-scale kernel diffusion fusion and the effectiveness of the overall design.

[0051] In summary, this method effectively improves the discriminative ability of single-modal local graph structures through client-side self-supervised Mahalanobis anchor graph learning, and significantly enhances the robustness and consistency of the global cross-modal consensus graph through server-side multi-scale kernel diffusion fusion and structural constraint projection. It achieves superior performance compared to existing centralized and federated clustering methods on multiple benchmark multimodal datasets. It aims to address at least one of the technical problems in existing technologies, such as the risk of data privacy leakage caused by centralized processing in multimodal clustering, insufficient discriminative ability of client-side local graph structures, and the difficulty for the server to construct a structurally effective and noise-resistant global consensus graph from heterogeneous local graph structures in distributed scenarios. Specifically, this method is applied to distributed multimodal data scenarios where each client independently stores different modal data (such as image features, text features, biometric features, etc.) for the same observation object, and does not share the original data. During the method execution, the client and server only exchange anchor graph matrices and consistency pseudo-labels, thereby strictly ensuring data privacy while completing cross-modal collaborative clustering.

[0052] Compared with existing technologies, this method innovatively designs a self-supervised Mahalanobis anchor graph learning mechanism at the client level. It uses consistent pseudo-labels issued by the server as supervision signals to construct an anchor-class affinity matrix and calculate the intra-class scatter matrix and inter-class scatter matrix. It learns a Mahalanobis distance metric with semantic discriminative power to replace the traditional Euclidean distance for anchor graph learning, which effectively improves the discriminative ability of single-modal local graph structures. At the same time, it can achieve strict protection of the privacy of the original data by only uploading the anchor graph matrix. At the server level, an innovative multi-scale kernel diffusion fusion mechanism is introduced. This mechanism calculates the hot kernel representation of the Laplacian matrix of each modality graph across multiple preset diffusion timescales, effectively filtering high-frequency noise using the low-pass filtering properties of the hot kernel. By learning the optimal scale weights, adaptive weighted fusion of hot kernels across all modalities and scales is achieved, overcoming the limitation of traditional simple average fusion in distinguishing view quality. Furthermore, a logarithmic pullback operation maps the consensus kernel matrix from kernel space back to Laplacian space and projects it onto a Laplacian cone with applied sparsity and algebraic connectivity constraints. This ultimately generates a structurally effective, noise-resistant, and connectivity-preserving global consensus graph, significantly enhancing the robustness and consistency of cross-modal consensus representations. Through an iterative optimization framework with end-cloud collaboration, the client and server only need to exchange anchor graph matrices and consistency pseudo-labels, fully leveraging the consistency and complementarity information of multimodal data while strictly protecting data privacy. Experiments show that this method outperforms existing centralized and federated multimodal clustering methods in terms of clustering accuracy, normalized mutual information, and purity on multiple benchmark multimodal datasets such as MSRC, ORL, Flowers17, 100Leaves, BBCSport, BBC, NewsGroup, Prokaryotic, BDGP, ​​and 3Sources.

[0053] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A federated multimodal clustering method based on self-supervised multi-scale kernel diffusion graph learning, characterized in that, Includes the following steps: Step S1: Control each client to construct a mode-specific anchor graph based on its local corresponding single-modal data, and upload the anchor graph to the server. Control the server to perform preprocessing and spectral clustering based on the anchor graph to generate a global consensus graph and consistent pseudo-labels. Step S2: The driver server sends the consistency pseudo-label to each client, controls each client to relearn based on the consistency pseudo-label and the end-to-cloud collaboration mechanism, updates the anchor map, and uploads the updated anchor map to the server. Step S3: Drive the server to perform multi-scale kernel diffusion fusion and Laplacian cone projection processing on the updated anchor graph, update the global consensus graph and consistency pseudo-label, and obtain the updated global consensus graph and updated consistency pseudo-label. Step S4: Repeat steps S2 to S3 for iterative optimization until the preset convergence condition is met, and extract and generate cross-modal clustering results based on the global consensus graph obtained from the last update.

2. The federated multimodal clustering method based on self-supervised multi-scale kernel diffusion graph learning according to claim 1, characterized in that, Each client is controlled to construct a mode-specific anchor graph based on its local single-modal data. Specifically, the local single-modal data is obtained, and Euclidean distance is directly used. Construct anchor graphs, where, For the sample In modality The feature matrix of local single-modal data under, anchor point In modality The anchor point set below, It is the square of the L2 norm.

3. The federated multimodal clustering method based on self-supervised multi-scale kernel diffusion graph learning according to claim 2, characterized in that, Each client is controlled to relearn and update the anchor graph based on the consistent pseudo-label and the end-to-cloud collaboration mechanism, specifically as follows: For mode k, the control client obtains the feature matrix from the local single-mode data. In this process, the K-means clustering algorithm is used to select cluster centers as the anchor point set. , This is the feature matrix of local single-mode data under mode k. For the anchor point set under mode k, For the set of real numbers, For the sample set, For feature dimension, This refers to the number of anchor points; Based on the consistent pseudo-label y issued by the server, generate the one-hot encoding matrix under mode k. Wherein, the consistent pseudo-label corresponding to sample i If and only if At that time, the one-hot encoded value of sample i for class j under mode k c is the preset number of categories; Anchor graph based on mode k One-hot encoding matrix under mode k Calculate the anchor-class affinity matrix under mode k. , , For transpose; Based on the anchor-class affinity matrix under mode k Calculate the corresponding intra-class scatter matrix respectively. and inter-class scatter matrix ,in, For the sample In modality The set of multiple nearest neighbor anchor points below, In modal anchor point Consistent pseudo-labels for sample i Affinity; Combined with regularization parameters , Construct a Mahalanobis distance metric matrix with semantic discriminative power. , It is the identity matrix; Calculate Mahalanobis distance using the Mahalanobis distance metric matrix. Using Mahalanobis distance instead of Euclidean distance, we solve the anchor graph optimization problem with non-negativity constraints and row sum-to-one constraints, for the anchor graph under mode k. Update.

4. The federated multimodal clustering method based on self-supervised multi-scale kernel diffusion graph learning according to claim 3, characterized in that, The anchor graph optimization problem has a closed-form solution. Under mode k, sample i and anchor point... The formula for calculating the anchor weights between them is: , For the sample under mode k After sorting the Mahalanobis distances to all anchor points in ascending order, the [number]th The smaller value, K, represents the number of anchor points in the nearest neighbor set. For anchor point K+1 in mode The anchor point set below, anchor point In modality The anchor point set below, For the sample under mode k After sorting the Mahalanobis distances to all anchor points in ascending order, the [number]th Small value, For the sample under mode k After sorting the Mahalanobis distances to all anchor points in ascending order, the [number]th Small value.

5. The federated multimodal clustering method based on self-supervised multi-scale kernel diffusion graph learning according to claim 4, characterized in that, The driver server performs multi-scale kernel diffusion fusion and Laplacian cone projection processing on the updated anchor graph to update the global consensus graph and consistency pseudo-labels, resulting in the updated global consensus graph and updated consistency pseudo-labels, specifically: Construct the corresponding complete graph based on the updated anchor graph, calculate the normalized symmetric graph Laplacian matrix corresponding to the complete graph, and calculate the normalized symmetric graph Laplacian matrix for each graph at multiple preset diffusion time scales to generate multiple heat kernel representations; By solving the alternating optimization problem with entropy regularization, the optimal weights for the diffusion time scale are learned, and the hot kernel representations of all modes and all scales are adaptively weighted and fused to obtain the final consensus kernel matrix. Perform a logarithmic pullback operation on the consensus kernel matrix to restore it from the kernel space to the corresponding intermediate graph Laplacian matrix; The intermediate graph's Laplacian matrix is ​​projected onto a Laplacian cone constrained by sparsity and algebraic connectivity, and solved using the alternating direction multiplier method to obtain the final global consensus graph. Perform spectral clustering on the final global consensus graph to obtain updated consistency pseudo-labels.

6. The federated multimodal clustering method based on self-supervised multi-scale kernel diffusion graph learning according to claim 5, characterized in that, Based on the updated anchor graph, construct the corresponding complete graph, calculate the normalized symmetric graph Laplacian matrix corresponding to the complete graph, and calculate the normalized symmetric graph Laplacian matrix for each normalized symmetric graph at multiple preset diffusion time scales to generate multiple heat kernel representations, specifically: Receive anchor graphs for all modes Construct a complete graph for the anchor graph of each modality. , m is the number of modes. For modality The complete image below, For degree matrix, This represents the operation of converting a vector into a diagonal matrix, where 1 represents a vector consisting entirely of 1s. According to mode The complete image below Calculate the corresponding normalized symmetric graph Laplacian matrix. , For modality The normalized symmetric graph Laplace matrix under the following conditions; Based on a pre-set diffusion timescale set For each mode and the s-th diffusion timescale The formula for calculating the heat nucleus representation is as follows: S is the number of diffusion timescales. For modality The thermal nucleus representation corresponding to the s-th diffusion timescale. This is for matrix exponentiation operations.

7. The federated multimodal clustering method based on self-supervised multi-scale kernel diffusion graph learning according to claim 6, characterized in that, By solving an alternating optimization problem with entropy regularization, the optimal weights for the diffusion timescale are learned. Adaptive weighted fusion of the hot kernel representations across all modalities and scales is then performed to obtain the final consensus kernel matrix, specifically: Construct an optimization problem with entropy regularization, using the objective function, whose formula is: , , , The weights corresponding to the s-th diffusion timescale are: For consensus kernel matrix, As the weight of the diffusion timescale, It is the Frobenius norm. For matrix logarithm operations, To control the regularization parameter of the weight distribution entropy; An alternating optimization strategy is used to solve an optimization problem with entropy regularization, wherein the solution includes a first update and a second update. The first update is: when the weights are fixed on the diffusion timescale. Update consensus kernel matrix When the objective function degenerates into a weighted average problem, its closed-form solution is: ; The second update is: when the consensus kernel matrix is ​​fixed. Update the weights of the diffusion timescale At that time, the reconstruction loss at each scale is defined as Using the Lagrange multiplier method, the closed expression is obtained as follows: , Let be the reconstruction loss corresponding to the s-th diffusion timescale. For the first Reconstruction loss corresponding to each diffusion timescale; The first and second updates are performed alternately until the weight change on the diffusion timescale is below a preset threshold or the maximum number of alternating iterations is reached. The consensus kernel matrix is ​​then updated using the last update. As the final consensus core matrix.

8. The federated multimodal clustering method based on self-supervised multi-scale kernel diffusion graph learning according to claim 7, characterized in that, Perform a logarithmic pullback operation on the consensus kernel matrix to restore it from the kernel space mapping to the corresponding intermediate graph Laplacian matrix, specifically: The final consensus kernel matrix Mapping from kernel space back to graph Laplace space, for the final consensus kernel matrix Perform a logarithmic pullback operation to calculate the intermediate plot Laplacian matrix. , , This represents the weighted average diffusion time.

9. The federated multimodal clustering method based on self-supervised multi-scale kernel diffusion graph learning according to claim 8, characterized in that, Projecting the intermediate graph's Laplacian matrix onto a Laplacian cone constrained by sparsity and algebraic connectivity, and solving using the alternating direction multiplier method, yields the final global consensus graph, specifically: Construct a constrained Laplacian cone projection optimization problem as the optimization objective function to generate a global consensus graph. Its formula is: , , , , ,in, For the Laplace cone constraint set, For sparsity regularization parameters, Element level Norm, L is a placeholder in the constraint set, representing a lower bound on algebraic connectivity. For the first Line number Column elements, constraints This indicates that the graph is symmetric and constrained. This indicates a constraint that guarantees the row sum is zero. This indicates that off-diagonal elements are guaranteed to be non-positive, constraining... This indicates that the second smallest eigenvalue is not less than the lower bound of algebraic connectivity. ; The constrained Laplace cone projection optimization problem is solved using the alternating direction multiplier method. An auxiliary variable Z and a dual variable U are introduced, and the constrained Laplace cone projection optimization problem is restated as follows: , , For Laplace cone constraint set Indicator functions, It is an indicator function for element-level non-positive constraints; The corresponding augmented Lagrangian function is obtained: , , For penalty parameters; The solution process of the alternating direction multiplier method requires alternating execution of the following three updates until the update convergence condition is met, and then updating the global consensus graph of the last update. As the final global consensus graph; The first update addresses the spectral projection problem by updating the global consensus graph. With the auxiliary variable Z and the dual variable U fixed, the solution formula is: ,make and perform feature decomposition on it. Projecting the eigenvalues ​​as: Reconstruction yields , For the first The global consensus graph of the next iteration. For spectral constraint set, For the first Auxiliary variables for the next iteration For the first The dual variable of the next iteration, A, is the intermediate matrix. The eigenvector matrix, It is an eigenvalue diagonal matrix. For the first 1 eigenvalue, For the first Each projected eigenvalue; The second update addresses the sparse Laplace projection problem by updating the auxiliary variable Z and fixing the global consensus graph. For the dual variable U, the soft threshold operator combined with non-positive constraints is applied to off-diagonal elements, and the formula is as follows: The diagonal elements are determined by the condition that the sum of the rows is zero, and the formula is: , For the first The iteration of the ... Line number The auxiliary variables corresponding to the columns, For soft thresholding operators, For the first The iteration of the ... Line number The corresponding global consensus graph, For the first The iteration of the ... Line number The dual variable corresponding to the column, For the first The iteration of the ... Line number The auxiliary variables corresponding to the columns, For the first The iteration of the ... Line number Auxiliary variables corresponding to the columns; The third update is to update the dual variable U, and its formula is: , For the first The dual variable of the next iteration For the first Auxiliary variables for each iteration.

10. The federated multimodal clustering method based on self-supervised multi-scale kernel diffusion graph learning according to claim 1, characterized in that, The preset convergence conditions include: the rate of change of the consistent pseudo-label between two adjacent iterations is lower than a preset threshold, or the change in the global consensus graph is lower than a preset threshold, or the preset maximum number of communication rounds is reached.

Images