ART-based multi-modal federated clustering method

Abstract

An ART-based multi-modal federated clustering method belongs to the field of federated learning, and comprises the following steps: constructing a dual-channel fusion ART model and embedding the dual-channel fusion ART model into a client and a server, the dual-channel fusion ART model comprising a feature vector channel and a label channel, for a classification task, transmitting a feature vector through the feature vector channel, and transmitting a label vector through the label channel; for a clustering task, closing a label channel, and transmitting a feature vector only through a feature vector channel; and the dual-channel fusion ART model adopts an improved CAE algorithm to realize a classification task and a clustering task. According to the invention, through a dual-channel mechanism of a clustering and classification task unification framework, the method is suitable for simultaneously processing a complex scene with a known task and an unknown mode; by integrating data features of different dimensions, limitation of single-mode information is broken through, complementarity of multi-mode data is utilized, features are captured from different angles, and clustering quality, robustness and generalization ability are improved.

Description

Technical Field

[0001] This invention belongs to the field of federated learning technology, specifically relating to an ART-based multimodal federated clustering method. Background Technology

[0002] With the advent of the big data era, the storage and processing of distributed data has become one of the hot research topics. However, with the increasing demand for cross-device and cross-organizational collaborative learning, traditional centralized learning methods face a series of challenges, including privacy leaks, data heterogeneity, and computational bottlenecks. In order to fully explore the potential information in distributed data while ensuring data privacy, federated learning has been proposed as an emerging technology and has gradually gained widespread attention.

[0003] Clustering federated learning combines clustering methods with a federated learning framework to effectively address heterogeneity and privacy issues in distributed data. Specifically, by keeping data on local devices or within an organization, clustering federated learning avoids the privacy risks associated with centralized data storage. Simultaneously, through a collaborative learning mechanism, it fully leverages the data characteristics from different devices or organizations for clustering, solving the problem of uneven data distribution and heterogeneity in distributed data, and achieving efficient model training.

[0004] Basic clustering algorithms in federated learning typically use centroid-based clustering algorithms, such as K-Means clustering, Fuzzy C-Means clustering, and Gaussian Mixture Models (GMM). In 2019, Ghosh et al. proposed a K-Means-based federated clustering algorithm (A. Ghosh, J. Hong, D. Yin, et al. Robust Federated Learning in a Heterogeneous Environment [EB / OL]. arXiv preprint arXiv:1906.06629, 2019.). The pre-trained client parameters are used as input to the K-Means clustering algorithm, and then within each cluster, clients perform federated training.

[0005] In 2021, Liu et al. proposed clustering into K clusters based on the similarity of sparse vectors among different clients (BYLiu, Y.Guo, XQChen. PFA: Privacy-preserving Federated Adaptation for Effective Model Personalization[C]. Proceedings of the Web Conference 2021, 2021: 923-934.). Another drawback of the K-Means clustering algorithm is that if the client's cluster identity is incorrectly classified during clustering, the subsequent federated process cannot correct this identity, thus limiting the performance of the federated model. In 2020, Briggs et al. proposed a general federated learning hierarchical clustering algorithm (C. Briggs, Z. Fan, P. Andras. Federated Learning with Hierarchical Clustering of Local Updates to Improve Training on Non-IID Data[C]. 2020 International Joint Conference on Neural Networks(IJCNN), 2020: 1-9.), which trains a subset of clients during each round of federated model training. Subsequently, Ghosh et al. proposed an improved K-Means clustering federated algorithm (Iterative Federated Clustering Algorithm, IFCA) (A. Ghosh, J. Chung, D. Yin, et al. An Efficient Framework for Clustered Federated Learning[J]. Advances in Neural Information Processing Systems, 2020, 33: 19586-19597.), which dynamically assigns client clustering identities by minimizing the client loss function during each iteration. The algorithm's drawback lies in the high computational cost on the local client. In each round, the cluster model needs to be sent to the local machine, which then trains all cluster models and assigns them to the cluster corresponding to the model with the smallest loss value. Furthermore, the IFCA algorithm still suffers from performance limitations, related to its requirement for initial client identity assignment. While centroid-based clustering algorithms are simple and widely applicable, they require the number of centroids to be specified beforehand. This limitation makes it difficult to apply these algorithms to data with unknown or constantly changing distributions.Therefore, in order to improve the performance of clustered federated models, some scholars have proposed more advanced adaptive iterative clustered federated learning models. For example, Sattler et al. proposed the binary iterative method (Clustered Federated Learning, CFL) (F.Sattler,KRMuller,W.Samek.Clustered Federated Learning: Model-Agnostic Distributed Multitask Optimization Under Privacy Constraints[J].IEEE Transactions on Neural Networks and Learning Systems,2020,32(8):3710-3722.). In this method, the number of clusters does not need to be specified when performing clustering. In each iteration, binary clustering is performed based on the similarity of the clients, and the number of clusters changes adaptively with each iteration. Although the CFL method has better performance guarantees, it still requires a huge computational cost.

[0006] In clustering algorithms, the Growing Neural Gas (GNG) proposed by Fritzke et al. (B. Fritzke. A Growing Neural Gas Network Learns Topologies[J]. Advances in Neural Information Processing Systems, 1994, 7: 625-632.) and the Adjusted Self-Organizing Incremental Neural Network (ASOINN) proposed by Shen et al. (F. Shen, O. Hasegawa. A Fast Nearest Neighbor Classifier Based on Self-organizing Incremental Neural Network[J]. Neural Networks, 2008, 21(10): 1537-1547.) adaptively generate topological networks (i.e., nodes and edges) to represent the distribution of given data. The SOINN+ method proposed by Berrar et al. is an algorithm based on the ASOINN network (C. Wiwatcharakoses, D. Berrar. SOINN+, A Self-organizing Incremental Neural Network for Unsupervised Learning from Noisy Data Streams[J]. Expert Systems with Applications, 2020, 143:113069.), which can handle arbitrary data distributions in noisy data streams without any predefined parameters. However, because these algorithms learn new information by inserting new nodes and edges, they are prone to forgetting previously learned information, i.e., catastrophic forgetting. As an algorithm based on GNG networks, the Grow When Required Network (GWR) proposed by Marsland et al. (S. Marsland, J. Shapiro, U. Nehmzow. A Self-organizing Network that Grows when Required[J]. Neural Networks, 2002, 15(8-9): 1041-1058.) successfully avoids catastrophic forgetting, adding a node only when the current network state is insufficient to match a new instance.A common problem with GWR and SOINN+ methods is that the threshold computation cost per node increases with the number of nodes in the topology, thus reducing learning efficiency. In clustering algorithms, Masuyama et al. proposed a clustering method based on Adaptive Resonance Theory (ART), which can adaptively generate centroid nodes according to the distribution of given data and perform continuous learning without causing catastrophic forgetting (N. Masuyama, CK Loo, H. Ishibuchi, et al. Topological Clustering via Adaptive Resonance Theory with Information Theoretic Learning[J].IEEE Access,2019,7:76920-76936.). In addition, Liu et al. proposed a Corren's Entropy-Induced Measure (CIM) (W. Liu, P. P. K. Harel, J. C. P. Rincipe. Correntropy: Properties and Applications in Non-Gaussian Signal Processing [J]. IEEE Transactions on Signal Processing, 2007, 55(11): 5286-5298.), and the ART clustering algorithm based on this similarity measure has better clustering performance than other clustering algorithms.

[0007] Traditional federated learning, such as FedAvg and FedProx, aggregates the global model by averaging client model parameters through multiple rounds of iteration. However, in heterogeneous data with skewed label distributions, local models drift due to differences in data distribution, causing the global model to favor certain clients and reducing generalization performance. Existing federated clustering methods only support single tasks, either supervised classification or unsupervised clustering. Mainstream federated clustering algorithms, such as k-FED, FedFCM, and MUFC, are based on k-means or fuzzy C-means, requiring pre-specification of the number or size of clusters, and cannot dynamically adjust the cluster structure, making them difficult to handle scenarios with unknown or continuously changing data distributions. Furthermore, traditional federated clustering methods rely on only single modal features, making them susceptible to noise interference. Summary of the Invention

[0008] Existing federated clustering methods require specifying the number of nodes in each cluster and cannot achieve continuous learning. When the client encounters new data, it needs to retrain the entire dataset, resulting in high computational overhead and catastrophic forgetting. Static clustering structures cannot adapt to dynamic data distributions. To address the problems of low clustering accuracy and unstable clustering results commonly found in existing federated clustering methods when dealing with heterogeneous data, this invention provides a multimodal federated clustering method based on ART.

[0009] This invention presents an ART-based multimodal federated clustering method that eliminates the need to specify the number of cluster nodes. Furthermore, leveraging the incremental learning characteristics of ART, the client-side local model dynamically generates or adjusts category nodes, absorbing new data in real-time without forgetting old knowledge, thus avoiding global retraining. Edge management and the removal of isolated nodes during topology evolution dynamically maintain node relationships, uncovering data distribution structure, filtering noise interference, and improving model robustness. Additionally, this invention assesses node diversity through a similarity metric, replacing a fixed threshold and enabling adaptive adjustment of clustering granularity. Moreover, differential privacy is introduced to ensure data security, providing reliable support for real-world scenarios.

[0010] The technical solution adopted by this invention to solve the technical problem is as follows:

[0011] This invention provides a multimodal federated clustering method based on ART, which specifically includes the following steps:

[0012] A dual-channel fusion ART model is constructed and embedded into the client and server respectively. The dual-channel fusion ART model includes a feature vector channel and a label channel. For classification tasks, feature vectors are transmitted through the feature vector channel and label vectors are transmitted through the label channel. For clustering tasks, the label channel is turned off, and feature vectors are transmitted only through the feature vector channel. The dual-channel fusion ART model uses an improved CAE algorithm to implement classification and clustering tasks.

[0013] Furthermore, in the dual-channel fusion ART model, the D-dimensional input feature vector received by the feature vector channel is represented as follows: The tag vector received by the tag channel is represented as follows: C represents the number of categories; according to the complementary coding method, each input feature vector will be associated with its complementary vector. Combination, feature vector channels (k=1) In the tag channel (k=2) The final input feature vector is The original input vector, These are the complementary values ​​of the original input vector.

[0014] Furthermore, in the dual-channel fusion ART model, four key hyperparameters applicable to all channels are used to determine the dynamic process of fusion ART, namely: selecting hyperparameter α k >0, contribution hyperparameter γ k ∈[0,1], warning value ρ k ∈[0,1] and learning rate β k ∈[0,1]; Warning value ρ of the label channel 2 =1.0, learning rate β 2 =1.0.

[0015] Furthermore, in the dual-channel fusion ART model, the input feature vector and label vector are clustered into category nodes through a category layer; each category node j represents a category code. Let X be the category code vector for channel k, whose dimension is the same as the input feature vector X. k Consistent.

[0016] Furthermore, the improved CAE algorithm includes the following steps:

[0017] (1) The diversity of node sets is evaluated by using the determinant point process approximation method and correlation measure;

[0018] (2) The similarity threshold is determined by calculating the average of the minimum paired CIM values ​​in the set of active nodes A;

[0019] (3) Each time a data point x is input, select the two most similar nodes from the current node set as the winning nodes;

[0020] (4) Evaluate the relationship between data point x and the two winning nodes through an alertness test;

[0021] (5) Create and update nodes and edges based on the results of the alertness test;

[0022] (6) Estimate the edge deletion threshold based on the age of the current edge and the age of the deleted edges;

[0023] (7) When the age of an edge exceeds the edge deletion threshold, the edge will be deleted, and the age set of all deleted edges will be updated.

[0024] Furthermore, in step (1), diversity D is estimated using the determinant of matrix R, where matrix R is the pairwise similarity matrix between nodes; the bandwidth of the correlation measure is given by the formula... The calculation yields the following: where d is the vector dimension; D = 0 indicates that the node set is not diverse, and D > 0 indicates that the node set is diverse; when the value of diversity D is close to 0, it means that the creation of new nodes is close to existing nodes.

[0025] Furthermore, in step (4), the alertness test is divided into three cases:

[0026] ① If the similarity between data point x and the first winning node s1 is greater than the similarity threshold V threshold V threshold <V s1 ≤V s2 ;

[0027] ② If the similarity between data point x and the first winning node s1 is less than the similarity threshold V threshold The similarity with the second winning node s2 is greater than the similarity threshold V. threshold V s1 ≤V threshold <V s2 ;

[0028] ③ If the similarity between data point x and both winning nodes is less than the threshold, i.e., V s1 ≤V s2 ≤V threshold ;

[0029] Among them, V s1 V represents the similarity between data point x and the first winning node s1. s2 Let x be the similarity between data point x and the second winning node s2.

[0030] Furthermore, in step (5), the specific implementation process for creating and updating the nodes and edges is as follows:

[0031] 1) If data point x belongs to case ① in the alertness test, it means that the data point x has insufficient similarity with the existing nodes. In this case, a new node is created and added to the new node set, and the oldest node in the active node set A is replaced.

[0032] 2) If data point x belongs to case ② in the alertness test, it means that data point x matches the first winning node s1. Then update the first winning node, increase its winning count, and update the node position. y s1 M represents the position of node s1. s1 To determine the number of wins for node s1, replace the oldest node in the active node set A with the updated node, and increment the age of each edge connected to node s1 by 1.

[0033] 3) If data point x belongs to case ③ in the alertness test, it means that data point x matches both winning nodes. Then, perform a similar operation to step 2), and check if nodes s1 and s2 are connected by an edge. If there is an edge, reset the age of that edge to 1; if there is no edge, create a new edge and set its age to 1. After updating the edge information, update all neighboring nodes of node s1. y k M represents the location of the neighboring node. k λ represents the number of times the neighboring node has won; every λ data points, nodes without any connected edges are deleted and removed from the active node set A.

[0034] Furthermore, the formula for calculating the edge deletion threshold is as follows:

[0035]

[0036] Where, α del Let |α| be the set of ages of all edges deleted during the learning process. del |For the age set α del The number of elements in For the age set α del The arithmetic mean of the edges is given, where α is the age set of the edges connected to node s1, and |α| is the number of elements in the age set α. The edge deletion threshold α is increased each time the age of an edge increases. max It will update;

[0037] Furthermore, in the improved CAE algorithm, the class code activation equation of the global model is modified, and its calculation formula is as follows:

[0038]

[0039] in, The contribution hyperparameter of the k-th channel in the server model is used to adjust the weight of this channel in the activation value calculation, and its value range is [0,1]. This is the vector of the j-th local category code of client n in the k-th channel, used to represent the feature or label information of a certain cluster in the client's local data; Let g be the category code vector of the global category node g in the k-th channel. It is the global cluster representation obtained by the server after aggregating the local codes of multiple clients. is the hyperparameter for selecting the k-th channel in the server model, which affects the sensitivity of activation value calculation, and takes a value greater than 0; w (n,j') Let g be the local category code of client n excluding the j-th category; g is the global category node; if any category code w of client n (n,j')If a category code (n,j) from the same client is already associated with the global category node g, then another category code (n,j) cannot be associated with the same global category node, and its activation value is set to 0.

[0040] The beneficial effects of this invention are:

[0041] 1. Traditional federated learning, such as FedAvg and FedProx, aggregates the global model by averaging client model parameters through multiple rounds of iteration. However, with heterogeneous data exhibiting skewed label distribution, local models drift due to differences in data distribution, causing the global model to favor certain clients and reducing generalization performance. To address this issue, this invention replaces parameter averaging with clustering aggregation using category codes. Clients generate local category codes using ART, and the server aggregates them into a global code using ART, preserving the diversity of heterogeneous data.

[0042] 2. Existing federated clustering methods only support a single task, either supervised classification or unsupervised clustering, and can only achieve one of the clustering or classification tasks. Different architectures need to be designed for the two types of tasks, leading to system complexity, high maintenance costs, and the inability of classification models to utilize unlabeled data, while clustering models cannot associate label information, forming "data silos." To address this problem, this invention implements a dual-channel mechanism that unifies the framework for clustering and classification tasks by switching the label channel on and off within the same ART model. This mechanism is suitable for complex scenarios that simultaneously handle known tasks and unknown patterns. Furthermore, by prohibiting multiple prototypes from the same client from clustering into the same cluster, client diversity is preserved, client prototype overfitting is avoided, and the ability to handle heterogeneous data is enhanced.

[0043] 3. Mainstream federated clustering algorithms, such as k-FED, FedFCM, and MUFC, are based on k-means or fuzzy C-means, requiring pre-specification of the number of clusters or cluster size. Furthermore, they cannot dynamically adjust the cluster structure, making them unsuitable for scenarios with unknown or continuously changing data distributions. To address this issue, this invention proposes a federated clustering model that does not require specifying the number of clusters and can continuously learn, based on the incremental learning characteristics and topology maintenance of the Adaptive Resonance Theory (ART) algorithm.

[0044] 4. In addition, existing federated clustering methods rely on only single-modal features and are susceptible to noise interference. To address this issue, this invention combines multimodal technology to expand the application scenarios of federated clustering models in multimodal data. By integrating data features from different dimensions, it overcomes the limitations of single-modal information, fully utilizes the complementarity of multimodal data, captures features from different angles, and improves clustering quality, robustness, and generalization ability using multi-dimensional features. Attached Figure Description

[0045] Figure 1 The flowchart illustrates an ART-based multimodal federated clustering method provided by this invention.

[0046] Figure 2 This is a structural diagram of a dual-channel fused ART model. Detailed Implementation

[0047] The present invention will be further described in detail below with reference to the accompanying drawings.

[0048] Based on Adaptive Resonance Theory (ART), this invention proposes a multimodal federated clustering method that combines supervised and unsupervised learning frameworks and is capable of continuous learning.

[0049] This invention combines the ART algorithm to achieve federated learning for clustering. The ART algorithm performs clustering in an adaptive manner, dynamically adjusting based on patterns in different data within the data feature space to handle data heterogeneity and complexity. During server-side clustering, adaptive clustering replaces parameter averaging, preserving the diversity of client-side data, avoiding model drift, and improving robustness in heterogeneous environments. Furthermore, when dealing with dynamically changing distributed data, the incremental learning characteristic of the ART algorithm effectively avoids the "catastrophic forgetting" phenomenon, maintaining the stability and accuracy of the clustering model while achieving continuous learning.

[0050] At the application level, clustering federated learning based on the ART algorithm can adapt to scenarios with high data privacy requirements, complex distributed data environments, and dynamically changing data. This invention, based on the general learning mechanism of ART, utilizes the characteristic of the ART algorithm to simultaneously process labeled and unlabeled data, constructing a unified framework that supports both federated classification and clustering tasks. Furthermore, by combining multimodal technology, this invention can be extended to multimodal application scenarios. The client uses dedicated encoders for each modality to extract features, supports heterogeneous label systems, forms category codes, and performs local single-modal data clustering. The server, through an early cross-modal fusion mechanism, concatenates features to form multidimensional vectors, achieving multimodal information integration, clustering to form global category codes, and upgrading from "partial perception" to "three-dimensional cognition" through multidimensional feature complementarity.

[0051] In this invention, a dual-channel fusion ART model is first constructed and embedded into the client and server respectively. The dual-channel fusion ART models in both the client and server use the improved CAE algorithm to implement classification and clustering tasks.

[0052] During training, supervised learning is performed on labeled data, where the category code is the feature vector and label vector; unsupervised learning is performed on unlabeled data, where the label channel weights are set to 0, and the category code is only the feature vector. For example... Figure 1As shown, the client clusters local single-modal data (category codes) and uploads the clustered category codes to the server. The server first performs feature fusion, concatenating the features of each modality into a multi-dimensional vector, and then performs global clustering on the category codes to generate global category codes. During the server's multimodal data clustering process, an edge deletion mechanism and the removal of isolated nodes are used to maintain the cluster topology, adapting to dynamic data distribution, enabling continuous learning, and filtering noisy nodes to improve clustering quality. Finally, the server distributes the global category codes to each client, and each client replaces its local category code with the global category code. During the test data testing, the label channel is closed, unsupervised clustering is used, and the test results are compared to calculate the corresponding metrics.

[0053] Specifically, the dual-channel fusion ART model constructed in this invention includes a feature vector channel and a label channel, which flexibly supports labeled heterogeneous data clustering through the label channel. In clustering scenarios, the label channel will be removed. The structure is as follows: Figure 2 As shown, its core content is as follows:

[0054] (1) Input Layer (F1): In the dual-channel fusion ART model, the D-dimensional input feature vector received by the feature vector channel is represented as follows: The tag vector received by the tag channel is represented as Where C represents the number of categories. Based on the complementary coding method proposed by Carpenter, Grossberg, and Rosen (1991), each input feature vector is coupled with its complementary vector. Combination, where the feature vector channels (k=1) are... In the tag channel (k=2) The final input feature vector is The original input vector, These are the complementary values ​​of the original input vector. For clustering tasks, the label channels will be removed.

[0055] (2) Hyperparameters: The dynamic process of fused ART is determined by four key hyperparameters applicable to all channels, namely: selecting hyperparameter α k >0, contribution hyperparameter γ k ∈[0,1], warning value σ k ∈[0,1] and learning rate β k ∈[0,1]. The warning value ρ for the label channel. 2 =1.0, used for strict label matching, learning rate β 2 =1.0, used for quickly updating the tag prototype.

[0056] (3) Category Coding: The category layer (F2) clusters the input feature vector and label vector into category nodes. Each category node j represents a category code. in Let X be the category code vector for channel k, whose dimension is the same as the input feature vector X. k Consistent. Category codes represent clusters of a set of similar input instances.

[0057] This invention proposes a dual-channel mechanism for a unified framework of clustering and classification tasks. The same dual-channel fused ART model supports both types of tasks by switching the label channel on and off, without structural adjustments. It uses category codes as the abstract representation of client uploads, which consist of feature vectors and label vectors. The server clusters heterogeneous prototypes into a global prototype, preserving both supervised and unsupervised information. Furthermore, multiple prototypes from the same client are prohibited from being clustered into the same global cluster. By limiting the contribution of a single client to a single global cluster, the server is forced to learn finer-grained knowledge representations, preventing the loss of diversity within the client, avoiding overfitting of the global model, and effectively improving aggregation accuracy in extremely heterogeneous data.

[0058] According to this invention, each client uses its local data to train a local dual-channel fused ART model. For classification tasks, the feature vector channel is used to transmit feature vectors, and the label channel is used to transmit label vectors. For clustering tasks, the dual-channel fused ART model only accepts feature vectors as input, i.e., single-channel. After all clients have completed learning their respective local models, they send their category codes to the server for aggregation.

[0059] According to the present invention, during the server aggregation process, the class code (X) received from the client is... 1 ,X 2 ), and the existing category codes in the dual-channel fusion ART model. Perform matching. The activation value T of each global category node. j The specific calculation formula based on CIM values ​​is as follows:

[0060]

[0061] Where, γ k α represents the channel weights, used to adjust the influence of labels on clustering. k To select hyperparameters, this formula will replace the similarity threshold calculation formula in the improved CAE algorithm.

[0062] According to this invention, both the client-side and server-side dual-channel fused ART models employ an improved CAE algorithm. CAE (CIM-based ART with Edge) is a parameter-free topological clustering algorithm based on ART, capable of continuous learning. Typically, ART algorithms have a data-dependent parameter, such as a similarity threshold. In the CAE algorithm, the similarity threshold is calculated based on the pairwise similarity between a certain number of nodes. The number of nodes used to calculate the similarity threshold is estimated using the determinant point process (DPP) approximation method. Furthermore, the edge deletion threshold is estimated based on the age of each edge. Experimental studies on synthetic and real datasets show that the CAE algorithm outperforms existing parameter-free fixed algorithms in clustering. When the node set is large, the computational cost of the determinant becomes very high; therefore, a DPP approximation method can be introduced to reduce computational complexity. A common DPP approximation method approximates the determinant calculation through sampling. Specifically, DPP-based sampling algorithms, such as orthogonal sampling, can be used to select a smaller subset from the similarity matrix that represents the diversity of the original set. DPP-based sampling algorithms approximate the initial determinant by selecting a set of points in space, with a computational complexity typically of O(n^2). 2 (The computational complexity of DPP approximates the sampling determinant) or lower, making it suitable for large-scale datasets.

[0063] The improved CAE algorithm constructed in this invention is a continuous clustering algorithm based on Adaptive Resonance Theory (ART). As a preferred embodiment, the specific implementation flow of the improved CAE algorithm is as follows:

[0064] (1) Node diversity estimation;

[0065] The diversity of node sets is assessed using the determinant-based point process (DPP) approximation method and the correlation measure (CIM). Specifically, diversity D is estimated using the determinant of matrix R, which is the pairwise similarity matrix between nodes. The bandwidth of the correlation measure is given by the formula... The calculation yields a vector dimension, where d is the vector dimension. D = 0 indicates that the node set is not diverse, while D > 0 indicates that the node set is diverse. When the diversity value D is close to 0, it means that the creation of new nodes is closer to existing nodes.

[0066] When the diversity D satisfies the condition that D is less than the diversity threshold, it indicates that the diversity of the node set is very low, and the node set size λ is set to twice the original size. In the original CAE algorithm, the diversity threshold was set to 1.0 × 10⁻⁶. -6This often results in the generation of too many nodes. Therefore, in this invention, the diversity threshold needs to be adjusted based on experiments, and the threshold value should be appropriately reduced. Furthermore, a change rate threshold ∈ is set. When the change in diversity D is less than the change rate threshold ∈, the adjustment of the node set size λ is stopped to prevent the generation of too many nodes.

[0067] (2) Similarity threshold calculation;

[0068] Similarity threshold (V) threshold The value is determined by averaging the minimum paired CIM values ​​in the set of active nodes A. The specific calculation formula is as follows:

[0069]

[0070] Among them, y i Let y be the i-th node in the set of active nodes A; j For the set of active nodes A, excluding y i The j-th node outside of the active node set A; S is the bandwidth set of the Gaussian kernel function, which contains the bandwidth parameter corresponding to each node in the active node set A; means(S) represents the average value of the bandwidth set S, which is used to calculate the kernel function bandwidth when CIM.

[0071] (3) Selection of winning node;

[0072] Each time a data point x is input, the two most similar nodes are selected from the current node set as winning nodes. The first winning node s1 is the node most similar to data point x, and the second winning node s2 is the node most similar to data point x after excluding the first winning node s1. The first winning node s1 is added to the active node set A, while the oldest node in the active node set A is replaced.

[0073] (4) Alertness test;

[0074] The alertness test is used to evaluate the relationship between data point x and the two winning nodes, and it is divided into three cases:

[0075] ① If the similarity between data point x and the first winning node s1 is greater than the similarity threshold (V) threshold ), that is, V threshold <V s1 ≤V s2 ;

[0076] ② If the similarity between data point x and the first winning node s1 is less than the similarity threshold (V threshold The similarity with the second winning node s2 is greater than the similarity threshold (V). threshold ), that is, V s1 ≤V threshold <V s2 ;

[0077] ③ If the similarity between data point x and both winning nodes is less than the threshold, i.e., V s1 ≤V s2 ≤V threshold .

[0078] Among them, V s1 V represents the similarity between data point x and the first winning node s1. s2 Let x be the similarity between data point x and the second winning node s2.

[0079] (5) Creation and updating of nodes and edges;

[0080] The creation and updating of nodes and edges are based on the results of the alertness test. The specific steps are as follows:

[0081] 1) If data point x falls under case ① in the alertness test, indicating insufficient similarity between data point x and existing nodes, a new node will be created and added to a new node set. This new node becomes the active node and replaces the oldest node in the active node set A. The bandwidth of the new node is calculated using the active node set A, and the number of wins is initialized to 1.

[0082] 2) If data point x belongs to the second case ② in the alertness test, it means that data point x matches the first winning node s1. Then update the first winning node, increase its winning count, and update the node position. Where y s1 M represents the position of node s1. s1 To determine the number of wins for node s1, replace the oldest node in the active node set A with the updated node, and increment the age of each edge connected to node s1 by 1.

[0083] 3) If data point x falls under case ③ in the alertness test, meaning it matches both winning nodes, then perform operations similar to step 2) above, and check if nodes s1 and s2 are connected by an edge. If an edge exists, reset its age to 1; otherwise, create a new edge and set its age to 1. After updating the edge information, all neighboring nodes of node s1 are updated. Where y k M represents the location of the neighboring node. k This represents the number of times the neighboring node won.

[0084] In addition, to reduce the impact of noise, every λ data points, nodes without edge connections are deleted and removed from the active node set A.

[0085] (6) Estimation of the edge deletion threshold;

[0086] The edge deletion threshold is estimated based on the age of the current edge and the ages of the deleted edges, a process similar to the edge deletion mechanism of the SOINN+ algorithm. The edge deletion threshold α... max The definition is as follows:

[0087]

[0088] Where, α del Let |α| be the set of ages of all edges deleted during the learning process. del |For the age set α del The number of elements in For the age set α del The arithmetic mean of the edges is given, where α is the age set of edges connected to node s1, and |α| is the number of elements in age set α. The edge deletion threshold α is increased each time the age of an edge increases. max It will be updated.

[0089] (7) Edge deletion mechanism;

[0090] When the age of an edge exceeds the edge deletion threshold, the edge will be deleted, and the age set α of all deleted edges will be updated. del .

[0091] According to this invention, although both the client and server-side dual-channel fused ART models employ the aforementioned improved CAE algorithm, the difference lies in that the client does not require a topology structure. Furthermore, to enhance the ability to process heterogeneous data, the CAE algorithm is improved in the following two aspects:

[0092] (1) Since the category codes of the same client may represent different categories at the local level, in order to avoid clustering category codes from the same client together, the activation equation of the category code of the global model is modified. The specific calculation formula is as follows:

[0093]

[0094] in, The contribution hyperparameter of the k-th channel in the server model is used to adjust the weight of this channel in the activation value calculation, and its value range is [0,1]. This is the vector of the j-th local category code of client n in the k-th channel, used to represent the feature or label information of a certain cluster in the client's local data; Let g be the category code vector of the global category node g in the k-th channel. It is the global cluster representation obtained by the server after aggregating the local codes of multiple clients. is the hyperparameter for selecting the k-th channel in the server model, which affects the sensitivity of activation value calculation, and takes a value greater than 0; w (n,j')is the local category code for client n excluding the j-th category; g is the global category node.

[0095] If client n has any category code w (n,j') If a class code (n,j) from the same client is already associated with a global class node g, then another class code (n,j) cannot be associated with the same global class node. Therefore, its activation value is set to 0.

[0096] (2) Cluster sizes must be small enough to distinguish class codes representing heterogeneous client data. A greater vigilance is used for the global model. The vigilance hyperparameter ρ, ranging from 0.0 to 1.0, determines the size of the clusters represented by various class codes during the learning process by controlling the class code matching criteria. Smaller vigilance hyperparameter values ​​relax the matching criteria so that inputs can easily match existing class codes. Therefore, each class code cluster may contain a large number of inputs. On the other hand, larger vigilance hyperparameter values ​​lead to more mismatches between input class codes and existing class codes. When mismatches occur, new class codes are added to accommodate the input. Therefore, using a larger vigilance hyperparameter ρ, the global model means that heterogeneous class codes received from various clients can be clustered into different global categories, preventing them from merging into one or a few clusters.

[0097] According to the present invention, the feature vector channel uses the DPP approximation method in the improved CAE algorithm to dynamically calculate the warning hyperparameter ρ, the label channel uses a fixed value, the client uses a higher warning hyperparameter ρ, and the server should use a higher warning hyperparameter ρ than the client to deal with heterogeneous data and prevent the category codes of different clients from being merged into a few clusters.

[0098] This invention proposes an improved CAE algorithm. Leveraging the incremental learning characteristics of the ART algorithm, each client can adjust its clustering model at any time to adapt to the emergence of new data, achieving continuous learning and avoiding catastrophic forgetting. Furthermore, by implementing continuous evolution of the topology, edge management, and periodic deletion of isolated nodes, it adapts to continuous learning and dynamic environments, avoiding noise interference in the continuous learning process. The topology structure can also further mine the distribution information of unlabeled data. The diversity of the node set is evaluated using the Determinant Point Process (DPP) approximation method and the Correlation Metric (CIM). The size of the node set is adjusted based on this diversity, exhibiting dynamic adaptability compared to setting a fixed value for the warning hyperparameter ρ. When aggregating nodes on the server side, high-importance nodes with a high number of wins are prioritized, improving the stability of the algorithm in the initial learning phase.

[0099] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A multimodal federated clustering method based on ART, characterized in that, Includes the following steps: A dual-channel fusion ART model is constructed and embedded into the client and server respectively. The dual-channel fusion ART model includes a feature vector channel and a label channel. For classification tasks, feature vectors are transmitted through the feature vector channel and label vectors are transmitted through the label channel. For clustering tasks, the label channel is turned off, and feature vectors are transmitted only through the feature vector channel. The dual-channel fusion ART model uses an improved CAE algorithm to implement classification and clustering tasks.

2. The multimodal federated clustering method based on ART according to claim 1, characterized in that, In the dual-channel fusion ART model, the D-dimensional input feature vector received by the feature vector channel is represented as follows: The tag vector received by the tag channel is represented as follows: C represents the number of categories; according to the complementary coding method, each input feature vector will be associated with its complementary vector. Combination, feature vector channels (k=1) In the tag channel (k=2) The final input feature vector is The original input vector, These are the complementary values ​​of the original input vector.

3. The multimodal federated clustering method based on ART according to claim 1, characterized in that, In the dual-channel fusion ART model, four key hyperparameters applicable to all channels are used to determine the dynamic process of the fusion ART, namely: selecting hyperparameter α k >0, contribution hyperparameter γ k ∈[0,1], warning value ρ k ∈[0,1] and learning rate β k ∈[0,1]; Warning value ρ of the label channel 2 =1.0, learning rate β 2 =1.

0.

4. The multimodal federated clustering method based on ART according to claim 1, characterized in that, In the dual-channel fusion ART model, the input feature vector and label vector are clustered into category nodes through a category layer; each category node j represents a category code. Let X be the category code vector for channel k, whose dimension is the same as the input feature vector X. k Consistent.

5. The multimodal federated clustering method based on ART according to claim 1, characterized in that, The improved CAE algorithm includes the following steps: (1) The diversity of node sets is evaluated by using the determinant point process approximation method and correlation measure; (2) The similarity threshold is determined by calculating the average of the minimum paired CIM values ​​in the set of active nodes A; (3) Each time a data point x is input, select the two most similar nodes from the current node set as the winning nodes; (4) Evaluate the relationship between data point x and the two winning nodes through an alertness test; (5) Create and update nodes and edges based on the results of the alertness test; (6) Estimate the edge deletion threshold based on the age of the current edge and the age of the deleted edges; (7) When the age of an edge exceeds the edge deletion threshold, the edge will be deleted, and the age set of all deleted edges will be updated.

6. The multimodal federated clustering method based on ART according to claim 5, characterized in that, In step (1), diversity D is estimated using the determinant of matrix R, where matrix R is the pairwise similarity matrix between nodes; the bandwidth of the correlation measure is given by the formula... The calculation yields the following: where d is the vector dimension; D = 0 indicates that the node set is not diverse, and D > 0 indicates that the node set is diverse; when the value of diversity D is close to 0, it means that the creation of new nodes is close to existing nodes.

7. The multimodal federated clustering method based on ART according to claim 5, characterized in that, In step (4), the alertness test is divided into three cases: ① If the similarity between data point x and the first winning node s1 is greater than the similarity threshold V thrshold V thrshold <V s1 ≤V s2 ; ② If the similarity between data point x and the first winning node s1 is less than the similarity threshold V threshold The similarity with the second winning node s2 is greater than the similarity threshold V. threshold V s1 ≤V threshold <V s2 ; ③ If the similarity between data point x and both winning nodes is less than the threshold, i.e., V s1 ≤V s2 ≤V threshold ; Among them, V s1 V represents the similarity between data point x and the first winning node s1. s2 Let x be the similarity between data point x and the second winning node s2.

8. The ART-based multimodal federated clustering method according to claim 7, characterized in that, In step (5), the specific implementation process for creating and updating nodes and edges is as follows: 1) If data point x belongs to case ① in the alertness test, it means that the data point x has insufficient similarity with the existing nodes. In this case, a new node is created and added to the new node set, and the oldest node in the active node set A is replaced. 2) If data point x belongs to case ② in the alertness test, it means that data point x matches the first winning node s1. Then update the first winning node, increase its winning count, and update the node position. y s1 M represents the position of node s1. s1 To determine the number of wins for node s1, replace the oldest node in the active node set A with the updated node, and increment the age of each edge connected to node s1 by 1. 3) If data point x belongs to case ③ in the alertness test, it means that data point x matches both winning nodes. Then, perform a similar operation to step 2), and check if nodes s1 and s2 are connected by an edge. If there is an edge, reset the age of that edge to 1; if there is no edge, create a new edge and set its age to 1. After updating the edge information, update all neighboring nodes of node s1. y k M represents the location of the neighboring node. k λ represents the number of times the neighboring node has won; every λ data points, nodes without any connected edges are deleted and removed from the active node set A.

9. The multimodal federated clustering method based on ART according to claim 5, characterized in that, The formula for calculating the edge deletion threshold is: Where, α del Let |α| be the set of ages of all edges deleted during the learning process. del |For the age set α del The number of elements in For the age set α del The arithmetic mean of the edges is given, where α is the age set of the edges connected to node s1, and |α| is the number of elements in the age set α. The edge deletion threshold α is increased each time the age of an edge increases. max It will be updated.

10. The ART-based multimodal federated clustering method according to claim 5, characterized in that, In the improved CAE algorithm, the class code activation equation of the global model is modified, and its calculation formula is as follows: in, The contribution hyperparameter of the k-th channel in the server model is used to adjust the weight of this channel in the activation value calculation, and its value range is [0,1]. This is the vector of the j-th local category code of client n in the k-th channel, used to represent the feature or label information of a certain cluster in the client's local data; Let g be the category code vector of the global category node g in the k-th channel. It is the global cluster representation obtained by the server after aggregating the local codes of multiple clients. is the hyperparameter for selecting the k-th channel in the server model, which affects the sensitivity of activation value calculation, and takes a value greater than 0; w (n,j') Let g be the local category code of client n excluding the j-th category; g is the global category node; if any category code w of client n (n,j') If a category code (n,j) from the same client is already associated with the global category node g, then another category code (n,j) cannot be associated with the same global category node, and its activation value is set to 0.

Images