Multi-client small sample classification method and system based on decoupling element learning and mutual information maximization
By decoupling meta-learning and mutual information maximization strategies, the sensitivity and accuracy issues of small sample classification in multi-client collaborative training are solved, achieving efficient knowledge transfer and model generalization, and highly adaptable medical and biological image recognition and classification.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-03
- Publication Date
- 2026-03-13
AI Technical Summary
In multi-client collaborative training scenarios, traditional federated learning faces challenges such as local model overfitting and global model aggregation bias caused by small sample sizes and non-independent identically distributed data, as well as device heterogeneity and communication resource constraints. Furthermore, when meta-learning is combined with federated learning, there are issues of high model coupling and low knowledge transfer efficiency.
A decoupled meta-learning framework is adopted to separate the server model and the client model. By combining the mutual information maximization strategy and the partial knowledge distillation strategy, local meta-knowledge is transferred to the global model and global knowledge is fed back. The decoupled meta-learning framework learns global knowledge and meta-knowledge respectively. The mutual information maximization strategy promotes knowledge transfer, and the partial knowledge distillation strategy transmits key knowledge from the global model to the local model.
In a multi-client environment, the model's adaptability and accuracy were improved, and its utilization efficiency and generalization ability on small sample data were enhanced.
Smart Images
Figure CN121659012A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of machine learning technology, and in particular to a multi-client few-sample classification method and system based on decoupled meta-learning and mutual information maximization. Background Technology
[0002] With the development of edge computing and IoT technologies, multi-client collaborative training has become an important paradigm for artificial intelligence applications. However, traditional federated learning faces two major challenges in multi-client scenarios: First, client-side local data often exhibits small sample sizes and non-independent identically distributed characteristics, leading to overfitting of local models and bias in global model aggregation. Second, device heterogeneity and communication resource constraints further reduce training efficiency and model accuracy. Existing solutions such as the Federated Avg algorithm alleviate the data heterogeneity problem through local multi-round training and parameter aggregation, but are not optimized for small sample scenarios. While meta-learning can improve the model's rapid adaptability, its direct combination with federated learning still suffers from high coupling between client and server models and low knowledge transfer efficiency. Furthermore, traditional methods lack effective mechanisms for local and global knowledge interaction, resulting in insufficient sensitivity of models to small sample tasks.
[0003] Decoupling meta-learning reduces interference from global aggregation on local meta-knowledge by separating the learning processes of the client model and the server model. The server model, consisting of a server model encoder and a server model classifier, is responsible for learning client-invariant features and aggregating client parameters using the FedAvg strategy to form the global model. The client model, consisting of a client model encoder and a client model classifier, updates only locally and does not participate in aggregation, focusing on learning meta-knowledge from limited sample data. The client model encoder takes the output of the server model encoder as input, reducing computational resource requirements. Local training employs an episodic learning framework, where each meta-task includes a support set S and a query set Q, using the MAML strategy for rapid adaptation and meta-updates.
[0004] Mutual information is a measure of the degree of interdependence between random variables. The commonly used condition for maximizing mutual information is to maximize the correlation between two random events; in a dataset, this means maximizing the correlation between the probability distributions fitted to two data sets. Mutual information maximization strategies are used to enhance local-to-global knowledge transfer and address the problem of insufficient meta-knowledge learning due to insufficient local data. This strategy forces the server model to absorb local feature information from the client by maximizing the mutual information between the output representations of the server and client models.
[0005] However, there are currently no reports on improving the sensitivity and accuracy of small-sample classification in multi-client scenarios by decoupling the learning process of client and server models and combining mutual information maximization and partial knowledge distillation. Summary of the Invention
[0006] The purpose of this invention is to provide a multi-client small sample classification method and system based on decoupled meta-learning and mutual information maximization, which has high sample data utilization efficiency, high knowledge transfer efficiency, and strong model generalization ability. It achieves maximum knowledge extraction on a small number of samples, thereby improving the sensitivity and accuracy of small sample classification in multi-client scenarios.
[0007] The technical solution to achieve the purpose of this invention is: a multi-client few-sample classification method based on decoupled meta-learning and mutual information maximization, comprising the following steps:
[0008] Step 1: By decoupling the meta-learning framework, the server model and the client model are separated, and global knowledge and meta-knowledge are learned respectively based on existing medical biological images;
[0009] Step 2: Utilize the mutual information maximization strategy to transfer local meta-knowledge to the global server model;
[0010] Step 3: Employ a partial knowledge distillation strategy to distill key knowledge from the global server model to the local client model;
[0011] Step 4: Train and evaluate the model based on the meta-task, and output global and local small sample classification models to achieve medical biological image recognition and classification.
[0012] Furthermore, in step 1, the decoupled meta-learning framework separates the server model and the client model, and learns global knowledge and meta-knowledge respectively based on existing medical biological images, as detailed below:
[0013] Step 1.1: The server model consists of a server model encoder and a server model classifier. The server model encoder maps the input samples to a low-dimensional representation space and uses ResNet12 as the backbone network. The server model classifier performs base class classification.
[0014] The server model aggregates knowledge from multiple clients using a federated averaging strategy to form a global model. The aggregation formula is as follows:
[0015]
[0016] in Indicates the first Server model parameters after round aggregation It is the first The client in the first Parameters of the locally updated server model This represents the total number of clients; the server model is distributed to all clients at the start of each training round as the basis for local training.
[0017] Step 1.2: The client model consists of a client model encoder and a client model classifier. The client model encoder takes the output of the server model encoder as input, and the client does not need to process the high-dimensional features of the original data. The client model is updated only locally and does not participate in server aggregation.
[0018] Local updates are based on meta-tasks, each containing a support set S and a query set Q, sampled from the client's local base class dataset. Client-side model updates are performed in two steps: first, the support set S is used to quickly adapt and update the parameters of the client model.
[0019]
[0020] in It supports the learning rate of the set phase. It is the cross-entropy loss function on the support set. Indicates the first The new parameter values for each client after gradient descent adjustment. Refers to the first The original parameter values for each client; Indicates parameters The gradient is used to indicate the direction of parameter updates; Refers to the first The original server parameter values for each client;
[0021] Then, the client model parameters are updated using the query set Q to optimize the meta-knowledge:
[0022]
[0023] in It is the meta-learning rate; It is a loss function on the query set, combining the loss from cross-entropy and knowledge distillation; Indicates the first After gradient descent adjustment, each client New parameter values for the wheel, Refers to the first The first client's The original parameter values of the wheel, Indicates parameters The gradient is used to indicate the direction of parameter updates. Refers to the first The first client's The original server parameter values for the wheel.
[0024] Furthermore, the transfer of local meta-knowledge to the global server model using the mutual information maximization strategy described in step 2 is as follows:
[0025] Calculate the mutual information between the support set samples and the output representations of the server and client models, and promote knowledge transfer by maximizing the mutual information loss function; to balance meta-knowledge transfer and basic classification tasks, the mutual information loss... Cross-entropy loss on the support set Combined, the total loss function of the server model is formed. :
[0026]
[0027]
[0028] in These are hyperparameters that control the weights of mutual information loss; It is a server model for samples In category The predicted probability, It is a sample The true label, Cross-entropy loss on the support set; The support set refers to a small dataset used for rapid adaptation in meta-learning or knowledge distillation. This represents the total number of samples supporting the set. Indicates the number of categories. This represents the category number, indicating that all possible categories are iterated over.
[0029] Furthermore, step 3 employs a partial knowledge distillation strategy to distill key knowledge from the global server model to the local client model, as detailed below:
[0030] By selectively utilizing global knowledge from the server model, the meta-knowledge learning ability of the client model is enhanced, while reducing interference from irrelevant categories; knowledge distillation loss function. Based on KL divergence, the difference in probability distributions between the server model and the client model across relevant categories is measured:
[0031]
[0032] in This refers to the number of samples in the query set. Indicates the number of categories. This represents the category number, indicating that all possible categories have been traversed. Let represent the predicted probability of the server model that the i-th sample belongs to the j-th class, where Indicates server model parameters; Let represent the predicted probability of the client model that the i-th sample belongs to the j-th class, where Represents client-side model parameters; knowledge distillation loss function This forces the client model to mimic the prediction behavior of the server model on key categories, thereby achieving knowledge transfer.
[0033] Furthermore, step 4 involves training and evaluating the model based on a meta-task, outputting global and local few-sample classification models to achieve medical biological image recognition and classification, as detailed below:
[0034] Step 4.1: At the beginning of each training round, the server distributes the current global server model parameters to all clients. Each client performs updates locally based on the meta-task sampling, adapts the client model using the support set, and updates the parameters.
[0035] Step 4.2: Combine cross-entropy loss and knowledge distillation loss, use the query set to calculate the loss, perform meta-update, and at the same time use the mutual information maximization strategy to update the server model locally.
[0036] Step 4.3: After training is complete, the global server aggregates the server model parameters of all clients using FedAvg to obtain the global server model and the local client model of each client.
[0037] A multi-client few-shot classification system based on decoupled meta-learning and mutual information maximization is proposed. This system implements the aforementioned multi-client few-shot classification method and includes a decoupled meta-learning module, a local meta-knowledge transfer module, a knowledge distillation module, and a model training and evaluation module, wherein:
[0038] The decoupled meta-learning module separates the server model and the client model through the decoupled meta-learning framework, and learns global knowledge and meta-knowledge respectively based on existing medical biological images;
[0039] The local meta-knowledge transfer module utilizes a mutual information maximization strategy to transfer local meta-knowledge to the global server model.
[0040] The knowledge distillation module employs a partial knowledge distillation strategy to distill key knowledge from the global server model to the local client model.
[0041] The model training and evaluation module trains and evaluates the model based on the meta-task, and outputs global and local few-sample classification models.
[0042] A mobile terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multi-client small sample classification method.
[0043] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the multi-client few-sample classification method.
[0044] Compared with the prior art, the significant advantages of this invention are: (1) By using a client-server model decoupling architecture, the model can fully utilize the collective wisdom of distributed data while maintaining personalized learning capabilities, and shows significant advantages in non-independent and identically distributed data scenarios; (2) By using a mutual information maximization strategy to achieve local to global knowledge transfer, the meta-knowledge learned by the client can be effectively integrated into the global model; (3) At the same time, a partial knowledge distillation strategy is adopted to achieve global to local knowledge feedback, so that each client can benefit from the knowledge of other clients; (4) The utilization efficiency of small sample data is improved, and the maximum knowledge extraction is achieved on only a small number of samples. Attached Figure Description
[0045] Figure 1 This is a flowchart illustrating the multi-client few-sample classification method based on decoupled meta-learning and mutual information maximization of the present invention. Detailed Implementation
[0046] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.
[0047] like Figure 1 As shown, this invention presents a multi-client few-sample classification method based on decoupled meta-learning and mutual information maximization. The few samples are a small number of image samples, such as animal images, industrial product images, and medical images, with medical images being the primary focus. The method is applied to medical biological image recognition and classification, and includes the following steps:
[0048] Step 1: By decoupling the meta-learning framework, the server model and the client model are separated, and global knowledge and meta-knowledge are learned respectively based on existing medical biological images;
[0049] Step 2: Utilize the mutual information maximization strategy to transfer local meta-knowledge to the global server model;
[0050] Step 3: Employ a partial knowledge distillation strategy to distill key knowledge from the global server model to the local client model;
[0051] Step 4: Train and evaluate the model based on the meta-task, and output global and local small sample classification models to achieve medical biological image recognition and classification.
[0052] As a specific example, step 1 describes a decoupling meta-learning framework that separates the server model from the client model. Based on existing medical biological images, it learns global knowledge and meta-knowledge separately, thus solving the "global data variance" problem and preventing server aggregation operations from interfering with the client's local meta-knowledge learning. Specifically, the steps are as follows:
[0053] Step 1.1: The server model consists of a server model encoder and a server model classifier. The server model encoder maps the input samples to a low-dimensional representation space and uses ResNet12 as the backbone network. The server model classifier performs base class classification.
[0054] The server model aggregates knowledge from multiple clients using a federated averaging strategy to form a global model. The aggregation formula is as follows:
[0055]
[0056] in Indicates the first Server model parameters after round aggregation It is the first The client in the first Parameters of the locally updated server model This represents the total number of clients; this aggregation mechanism ensures that the server model learns client-invariant characteristics, reducing bias caused by the distribution of non-IID data.
[0057] The server model is distributed to all clients at the start of each training round, serving as the basis for local training.
[0058] Step 1.2: The client model consists of a client model encoder and a client model classifier. The client model encoder takes the output of the server model encoder as input, and the client does not need to process the high-dimensional features of the original data, thereby reducing the demand for computing resources. The client model is only updated locally and does not participate in server aggregation, thereby avoiding interference from global aggregation on meta-knowledge learning.
[0059] Local updates are based on meta-tasks, each containing a support set S and a query set Q, sampled from the client's local base class dataset. Client-side model updates are performed in two steps: first, the support set S is used to quickly adapt and update the parameters of the client model.
[0060]
[0061] in It supports the learning rate of the set phase. It is the cross-entropy loss function on the support set. Indicates the first The new parameter values for each client after gradient descent adjustment. Refers to the first The original parameter values for each client; Indicates parameters The gradient is used to indicate the direction of parameter updates; Refers to the first The original server parameter values for each client;
[0062] Then, the client model parameters are updated using the query set Q to optimize the meta-knowledge:
[0063]
[0064] in It is the meta-learning rate; It is a loss function on the query set, combining the loss from cross-entropy and knowledge distillation; Indicates the first After gradient descent adjustment, each client New parameter values for the wheel, Refers to the first The first client's The original parameter values of the wheel, Indicates parameters The gradient is used to indicate the direction of parameter updates. Refers to the first The first client's The original server parameter values for the wheel.
[0065] This design ensures that each client can independently learn meta-knowledge from local small sample data, while obtaining global information through the server model.
[0066] As a specific example, step 2 utilizes the mutual information maximization strategy to transfer local meta-knowledge to the global server model, ensuring that valuable meta-knowledge learned by the client can be effectively integrated into the global model, enriching the global model's knowledge base, as detailed below:
[0067] Calculate the mutual information between the support set samples and the output representations of the server and client models, and promote knowledge transfer by maximizing the mutual information loss function; to balance meta-knowledge transfer and basic classification tasks, the mutual information loss... Cross-entropy loss on the support set Combined, the total loss function of the server model is formed. :
[0068]
[0069]
[0070] in These are hyperparameters that control the weights of mutual information loss; It is a server model for samples In category The predicted probability, It is a sample The true label, Cross-entropy loss on the support set; The support set refers to a small dataset used for rapid adaptation in meta-learning or knowledge distillation. This represents the total number of samples supporting the set. Indicates the number of categories. This represents the category number, indicating that all possible categories are traversed. This strategy enables the server model to learn both client-invariant features and meta-knowledge simultaneously, improving its adaptability to few-shot tasks.
[0071] As a specific example, step 3 employs a partial knowledge distillation strategy to distill key knowledge from the global server model to the local client model, addressing the "insufficient local data" problem. This leverages the rich knowledge of the global model to guide and improve the learning of the local client model, as detailed below:
[0072] By selectively utilizing global knowledge from the server model, the meta-knowledge learning ability of the client model is enhanced, while reducing interference from irrelevant categories; knowledge distillation loss function. Based on KL divergence, the difference in probability distributions between the server model and the client model across relevant categories is measured:
[0073]
[0074] in This refers to the number of samples in the query set. Indicates the number of categories. This represents the category number, indicating that all possible categories have been traversed. Let represent the predicted probability of the server model that the i-th sample belongs to the j-th class, where Indicates server model parameters; Let represent the predicted probability of the client model that the i-th sample belongs to the j-th class, where Represents client-side model parameters; knowledge distillation loss function This forces the client model to mimic the prediction behavior of the server model on key categories, thereby achieving knowledge transfer.
[0075] As a specific example, step 4 involves training and evaluating the model based on a meta-task, outputting global and local few-shot classification models to achieve medical biological image recognition and classification. The model is trained and its performance is tested by simulating real few-shot classification scenarios, ultimately yielding a usable high-performance model, as detailed below:
[0076] Step 4.1: At the beginning of each training round, the server distributes the current global server model parameters to all clients. Each client performs updates locally based on the meta-task sampling, and uses the support set to quickly adapt the client model and update the parameters.
[0077] Step 4.2: Combine cross-entropy loss and knowledge distillation loss, use the query set to calculate the loss, perform meta-update, and at the same time use the mutual information maximization strategy to update the server model locally.
[0078] Step 4.3: After training is completed, the global server aggregates the server model parameters of all clients through FedAvg to obtain the global server model and the local client model of each client. These models are used for few-shot classification tasks and support N-way K-shot learning scenarios.
[0079] This invention also provides a multi-client few-shot classification system based on decoupled meta-learning and mutual information maximization. This system is used to implement the aforementioned multi-client few-shot classification method. The system includes a decoupled meta-learning module, a local meta-knowledge transfer module, a knowledge distillation module, and a model training and evaluation module, wherein:
[0080] The decoupled meta-learning module separates the server model and the client model through the decoupled meta-learning framework, and learns global knowledge and meta-knowledge respectively based on existing medical biological images;
[0081] The local meta-knowledge transfer module utilizes a mutual information maximization strategy to transfer local meta-knowledge to the global server model.
[0082] The knowledge distillation module employs a partial knowledge distillation strategy to distill key knowledge from the global server model to the local client model.
[0083] The model training and evaluation module trains and evaluates the model based on the meta-task, and outputs global and local few-sample classification models.
[0084] The present invention also provides a mobile terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multi-client small sample classification method.
[0085] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the multi-client few-sample classification method.
[0086] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A multi-client few-sample classification method based on decoupled meta-learning and mutual information maximization, characterized in that, Includes the following steps: Step 1: By decoupling the meta-learning framework, the server model and the client model are separated, and global knowledge and meta-knowledge are learned respectively based on existing medical biological images; Step 2: Utilize the mutual information maximization strategy to transfer local meta-knowledge to the global server model; Step 3: Employ a partial knowledge distillation strategy to distill key knowledge from the global server model to the local client model; Step 4: Train and evaluate the model based on the meta-task, and output global and local small sample classification models to achieve medical biological image recognition and classification.
2. The multi-client few-sample classification method based on decoupled meta-learning and mutual information maximization according to claim 1, characterized in that, Step 1 describes a decoupling meta-learning framework that separates the server model from the client model. Based on existing medical biological images, it learns global knowledge and meta-knowledge separately, as detailed below: Step 1.1: The server model consists of a server model encoder and a server model classifier. The server model encoder maps the input samples to a low-dimensional representation space and uses ResNet12 as the backbone network. The server model classifier performs base class classification. The server model aggregates knowledge from multiple clients using a federated averaging strategy to form a global model. The aggregation formula is as follows: ； in Indicates the first Server model parameters after round aggregation It is the first The client in the first Parameters of the locally updated server model This represents the total number of clients; the server model is distributed to all clients at the start of each training round as the basis for local training. Step 1.2: The client model consists of a client model encoder and a client model classifier. The client model encoder takes the output of the server model encoder as input, and the client does not need to process the high-dimensional features of the original data. The client model is updated only locally and does not participate in server aggregation. Local updates are based on meta-tasks, each containing a support set S and a query set Q, sampled from the client's local base class dataset. Client-side model updates are performed in two steps: first, the support set S is used to quickly adapt and update the parameters of the client model. ； in It supports the learning rate of the set phase. It is the cross-entropy loss function on the support set. Indicates the first The new parameter values for each client after gradient descent adjustment. Refers to the first The original parameter values for each client; Indicates parameters The gradient is used to indicate the direction of parameter updates; Refers to the first The original server parameter values for each client; Then, the client model parameters are updated using the query set Q to optimize the meta-knowledge: ； in It is the meta-learning rate; It is a loss function on the query set, combining the loss from cross-entropy and knowledge distillation; Indicates the first After gradient descent adjustment, each client New parameter values for the wheel, Refers to the first The first client's The original parameter values of the wheel, Indicates parameters The gradient is used to indicate the direction of parameter updates. Refers to the first The first client's The original server parameter values for the wheel.
3. The multi-client few-sample classification method based on decoupled meta-learning and mutual information maximization according to claim 1, characterized in that, Step 2 describes the use of mutual information maximization to transfer local meta-knowledge to the global server model, as detailed below: Calculate the mutual information between the support set samples and the output representations of the server and client models, and promote knowledge transfer by maximizing the mutual information loss function; to balance meta-knowledge transfer and basic classification tasks, the mutual information loss... Cross-entropy loss on the support set Combined, the total loss function of the server model is formed. : ； ； in These are hyperparameters that control the weights of mutual information loss; It is a server model for samples In category The predicted probability, It is a sample The true label, Cross-entropy loss on the support set; The support set refers to a small dataset used for rapid adaptation in meta-learning or knowledge distillation. This represents the total number of samples supporting the set. Indicates the number of categories. This represents the category number, indicating that all possible categories are iterated over.
4. The multi-client few-sample classification method based on decoupled meta-learning and mutual information maximization according to claim 1, characterized in that, Step 3 describes the use of a partial knowledge distillation strategy to distill key knowledge from the global server model to the local client model, as detailed below: By selectively utilizing global knowledge from the server model, the meta-knowledge learning ability of the client model is enhanced, while reducing interference from irrelevant categories; knowledge distillation loss function. Based on KL divergence, the difference in probability distributions between the server model and the client model across relevant categories is measured: ； in This refers to the number of samples in the query set. Indicates the number of categories. This represents the category number, indicating that all possible categories have been traversed. Let represent the predicted probability of the server model that the i-th sample belongs to the j-th class, where Indicates server model parameters; Let represent the predicted probability of the client model that the i-th sample belongs to the j-th class, where Represents client-side model parameters; knowledge distillation loss function This forces the client model to mimic the server model's predictive behavior on key categories, thereby achieving knowledge transfer.
5. The multi-client few-sample classification method based on decoupled meta-learning and mutual information maximization according to claim 1, characterized in that, Step 4 describes training and evaluating the model based on meta-tasks, outputting global and local few-sample classification models to achieve medical biological image recognition and classification, as detailed below: Step 4.1: At the beginning of each training round, the server distributes the current global server model parameters to all clients. Each client performs updates locally based on the meta-task sampling, adapts the client model using the support set, and updates the parameters. Step 4.2: Combine cross-entropy loss and knowledge distillation loss, use the query set to calculate the loss, perform meta-update, and at the same time use the mutual information maximization strategy to update the server model locally. Step 4.3: After training is complete, the global server aggregates the server model parameters of all clients using FedAvg to obtain the global server model and the local client model of each client.
6. A multi-client few-sample classification system based on decoupled meta-learning and mutual information maximization, characterized in that, This system is used to implement the multi-client few-sample classification method according to any one of claims 1 to 5. The system includes a decoupled meta-learning module, a local meta-knowledge transfer module, a knowledge distillation module, and a model training and evaluation module, wherein: The decoupled meta-learning module separates the server model and the client model through the decoupled meta-learning framework, and learns global knowledge and meta-knowledge respectively based on existing medical biological images; The local meta-knowledge transfer module utilizes a mutual information maximization strategy to transfer local meta-knowledge to the global server model. The knowledge distillation module employs a partial knowledge distillation strategy to distill key knowledge from the global server model to the local client model. The model training and evaluation module trains and evaluates the model based on the meta-task, and outputs global and local few-sample classification models.
7. A mobile terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the multi-client few-sample classification method as described in any one of claims 1 to 5.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the multi-client few-sample classification method as described in any one of claims 1 to 5.