A federated model training method and classification method for federation

Abstract

The present invention provides a federation-oriented model training method for sub-federation models of multiple sub-federations, wherein each sub-federation is a federation composed of multiple clients of the same organization, and the method includes: S1, in each sub-federation, using the training set of the sub-federation to locally initialize and train its sub-federation model, wherein the classification space of the sub-federation models corresponding to different sub-federations is the same; S2, transferring the sub-federation model of the previous sub-federation to the next adjacent sub-federation in a predetermined transfer order, and performing multiple rounds of federation training respectively to obtain sub-federation models containing public knowledge; S3, transferring the sub-federation model containing public knowledge to each sub-federation as a final teacher model, and each sub-federation uses the knowledge of the final teacher model and the local training set to train its sub-federation model to obtain the final sub-federation model of each sub-federation.

Description

Technical Field

[0001] The present invention relates to the field of neural network technology, specifically to the field of federated learning in neural network technology, and more specifically to a federated model training method and a classification method oriented to federation. Background Art

[0002] Deep learning models possess powerful representation and learning capabilities, capable of autonomously learning useful representations of high-dimensional data. This significantly reduces the effort required for manual feature design and extraction, enabling the training of highly accurate machine learning models. Due to their powerful performance, deep learning models have been widely applied in technical fields such as speech recognition, object detection, image recognition, and healthcare. However, training deep learning models often requires acquiring vast amounts of data, which translates into significant data costs for model builders. With increasing attention to data privacy, centralizing large amounts of data or directly transmitting data has become increasingly difficult. Therefore, leveraging distributed data to aggregate useful information while protecting data privacy and security, and to build robust deep learning models, has become a new research trend.

[0003] Furthermore, various countries, organizations, and institutions have introduced numerous laws, regulations, and systems for protecting data privacy and security, such as the European Union's General Data Protection Regulation (GDPR) and China's Data Security Law. These regulations further complicate direct access to large amounts of data. In this environment, federated learning (FL) technology has emerged.

[0004] Federated learning technology often uses encrypted model transmission instead of direct data transmission, preventing the original data from leaving the local machine, thereby protecting data privacy and security. Federated learning has become a new learning paradigm, attracting numerous scholars and researchers. Chinese patent application publication number CN114745253A discloses a traffic early warning method based on Transformer and federated learning. This method employs a general horizontal federated learning architecture, combined with a Transformer model, to detect anomalies in traffic flow at each base station. Ultimately, it develops a model to monitor the network performance of each CPE device. Chinese patent application publication number CN114595396A discloses a sequential recommendation method and system based on federated learning. This method stores user personal information, real-time, and historical data locally. Each client can represent its state based on its historical data, capturing the interaction between users and items. Based on deep reinforcement learning, the recommendation process is treated as a sequential decision-making process. Furthermore, an attention mechanism is introduced into the federated learning aggregation algorithm to consider the relevance and diversity of client data, increasing data personalization and enabling personalized recommendations for users while protecting personal data privacy. A Chinese patent with authorization announcement number CN114462626B discloses a federated model training method, apparatus, terminal device, and storage medium. Homomorphic property calculations are performed using a first plaintext model weight random number and a second plaintext model random number ciphertext, enabling more secure encryption of transmitted information and improving the security of federated model training.

[0005] Recently, Professor Yang Qiang, a renowned expert in the field of artificial intelligence, categorized federated learning into three scenarios: horizontal federation, vertical federation, and migration federation. While these three scenarios cover most federated learning applications in scientific research settings, real-world scenarios are far more complex. For example, multiple clients within a data-sharing organization form a sub-federation, while multiple clients within another organization form another federation. This can be repeated, creating multiple equal sub-federations with no central aggregation point. These federations can exchange private data within their respective scopes, but not between different federations. Numerous such scenarios exist in reality, for example, with different hospitals forming a sub-federation, different nursing homes forming another, and home care facilities forming yet another. The key issue is how to efficiently conduct federated learning across these federations. While various federation approaches have been applied in the real world, they still have several limitations in terms of versatility, specificity, and applicability.

[0006] 1) In federated learning scenarios, different federations are often equal and in a supreme position, and there is rarely a more advanced server. In the absence of a server, many existing federation methods cannot be directly applied.

[0007] 2) In federated learning scenarios, data distribution differences often exist across federations, requiring a sufficiently personalized high-precision model for each sub-federation. However, direct application of existing federation methods may result in reduced accuracy of individual federations.

[0008] Therefore, there is an urgent need to design a new federated learning method to cope with federated scenarios and solve the above technical problems. Summary of the Invention

[0009] Therefore, the purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a federation-oriented model training method and classification method.

[0010] The purpose of the present invention is achieved through the following technical solutions:

[0011] According to a first aspect of the present invention, a federation-oriented model training method is provided for training sub-federation models of multiple sub-federations, wherein each sub-federation is a federation composed of multiple clients of the same organization, and all clients of the sub-federation aggregate collected samples on a device representing the sub-federation to form a data set for the sub-federation including a training set and a validation set. The method comprises: S1, in each sub-federation, using the training set of the sub-federation to locally initialize and train its sub-federation model, wherein the classification space of the sub-federation models corresponding to different sub-federations is the same; S2, transferring the sub-federation model of the previous sub-federation to the next adjacent sub-federation in a predetermined transfer order, and performing multiple rounds of federated training on each sub-federation to obtain a sub-federation model containing common knowledge, wherein each complete transfer process is executed once in the predetermined transfer order to complete a round of federated training, and each sub-federation completes the training of its own sub-federation model using the transferred previous sub-federation model and the local training set; S3, transferring the sub-federation model containing common knowledge to each sub-federation as a final teacher model, and each sub-federation respectively trains its sub-federation model using the knowledge of the final teacher model and the local training set to obtain the final sub-federation model of each sub-federation.

[0012] In some embodiments of the present invention, in step S2, each round of federation training includes: starting from the sub-federation ranked first, in accordance with a predetermined transfer order, the sub-federation model of the previous sub-federation after training in this round is sequentially transferred to the next adjacent sub-federation as an intermediate teacher model to complete the training of the sub-federation model of the next sub-federation in this round, wherein the sub-federation model of the sub-federation ranked first in the initial round transfers the initialized sub-federation model to the next sub-federation, and the sub-federation ranked first in other rounds uses the sub-federation model transferred from the last sub-federation in the previous round to complete the training of its own sub-federation model and then transfers it to the next sub-federation.

[0013] In some embodiments of the present invention, each sub-federation model includes a feature extraction layer for extracting sample features and a classification prediction layer for classifying according to the sample features. Different sub-federations can adopt heterogeneous or homogeneous sub-federation models, wherein, when the teacher model of a sub-federation (an intermediate teacher model or a final teacher model) and its sub-federation model are isomorphic models, the parameters of the sub-federation model are updated at least based on the total loss determined by the classification loss of the sub-federation model and the distance loss between the feature extraction layer of the teacher model and the sample features extracted by the feature extraction layer of the sub-federation model; or when the teacher model of a sub-federation (an intermediate teacher model or a final teacher model) and its sub-federation model are heterogeneous models, the parameters of the sub-federation model are updated at least based on the total loss determined by the classification loss of the sub-federation model and the distance loss between the output of the teacher model and the sub-federation model.

[0014] In some embodiments of the present invention, in step S2, when the accuracy of the sub-federated model on the local validation set is greater than a predetermined first threshold, the sub-federated model is trained in the following manner: when the intermediate teacher model is isomorphic to the sub-federated model, the parameters of the sub-federated model are updated according to a first total loss determined by a first distance loss between the classification loss of the sub-federated model for the sample and the sample features extracted by the feature extraction layer of the teacher model and the sample features extracted by the feature extraction layer of the sub-federated model; when the intermediate teacher model is heterogeneous to the sub-federated model, the parameters of the sub-federated model are updated according to a second total loss determined by a second distance loss between the classification loss of the sub-federated model for the sample and the classification output by the classification prediction layer of the teacher model and the classification output by the classification prediction layer of the federated model.

[0015] In some embodiments of the present invention, the first total loss is determined as follows:

[0016]

[0017] in, Indicates the The first total loss corresponding to the sub-federation is Indicates the The number of samples in the training set of each federation, Represents a sample The corresponding label ,Label Indicative sample The category to which it belongs, Indicates the The training set of the federation, Indicates the The classification loss of the sub-federation model for the sample, Indicates the A sub-federation model of sub-federations, represents the first distance loss, Indicates the The feature extraction layer of the teacher model of the federation, Indicates the The feature extraction layer of the sub-federation model of the sub-federation, Represents the weight pre-set for the first distance loss.

[0018] In some embodiments of the present invention, the second total loss is determined as follows:

[0019]

[0020] in, Indicates the The second total loss corresponding to the sub-federation is Indicates the The number of samples in the training set of each federation, Represents a sample The corresponding label ,Label Indicative sample The category to which it belongs, Indicates the The training set of the federation, Indicates the The classification loss of the sub-federation model for the sample, represents the second distance loss, represents the weight pre-set for the second distance loss, Indicates the A sub-federation model of sub-federations, Indicates the The teacher model in the middle of the federation, Represents the preset temperature hyperparameter. The values ​​in the classification prediction layer of the intermediate teacher model and the sub-federated model before being processed by their softmax functions are divided by the preset temperature hyperparameter and used as the input of their softmax functions.

[0021] In some embodiments of the present invention, in step S2, when the accuracy of the sub-federated model on the local validation set is less than or equal to a predetermined first threshold, the current sub-federated model trains its sub-federated model in the following manner: when the intermediate teacher model is isomorphic to the current sub-federated model, after replacing the parameters of the sub-federated model with the parameters of the intermediate teacher model, the parameters of the replaced sub-federated model are updated using the classification loss of the sample; when the intermediate teacher model is heterogeneous to the current sub-federated model, the parameters of the sub-federated model are updated based on a third total loss determined by a second distance loss between the classification loss of the sub-federated model for the sample and the classification output by the classification prediction layer of the intermediate teacher model and the classification output by the classification prediction layer of the sub-federated model.

[0022] In some embodiments of the present invention, the third total loss is determined as follows:

[0023]

[0024] in, Indicates the The third total loss corresponding to the sub-federation is Indicates the The number of samples in the training set of each federation, Represents a sample The corresponding label ,Label Indicative sample The category to which it belongs, Indicates the The training set of the federation, Indicates the The classification loss of the sub-federation model for the sample, represents the second distance loss, represents the weight pre-set for the second distance loss, represents the reinforcement factor used to make more use of the teacher model’s knowledge, , Indicates the A sub-federation model of sub-federations, Indicates the The teacher model of the Federation, Represents the preset temperature hyperparameter. The values ​​in the classification prediction layer of the intermediate teacher model and the sub-federated model before being processed by their softmax functions are divided by the preset temperature hyperparameter and used as the input of their softmax functions.

[0025] In some embodiments of the present invention, in step S3, when the accuracy of the final teacher model on the validation set of the current sub-federation is less than a predetermined second threshold and less than the accuracy of the sub-federation model of the current sub-federation on the validation set, the current sub-federation trains its sub-federation model in the following manner: only using the training set of the current sub-federation to train its sub-federation model once or multiple times to obtain the final sub-federation model of the sub-federation.

[0026] In some embodiments of the present invention, in step S3, when the accuracy of the final teacher model on the validation set of the current sub-federation is greater than or equal to a predetermined second threshold and / or greater than or equal to the accuracy of the current sub-federation model on the validation set, the current sub-federation trains its sub-federation model in the following manner: when the final teacher model is isomorphic to the sub-federation model of the current sub-federation, the parameters of the sub-federation model of the current sub-federation are updated according to a fourth total loss determined by a first distance loss between the classification loss of the sub-federation model of the current sub-federation and the sample features extracted by the feature extraction layer of its teacher model and the sample features extracted by the feature extraction layer of the sub-federation model of the current sub-federation; when the final teacher model is heterogeneous to the sub-federation model of the current sub-federation, the parameters of the sub-federation model of the current sub-federation are updated according to a fifth total loss determined by a second distance loss between the classification loss of the sub-federation model of the current sub-federation and the classification output by the classification prediction layer of the teacher model and the classification output by the classification prediction layer of the sub-federation model of the current sub-federation.

[0027] In some embodiments of the present invention, in step S3, each sub-federation determines the adaptive weight set for the first distance loss or the second distance loss in the sub-federation based on the accuracy of the final teacher model on the validation set of the sub-federation and the accuracy of its sub-federation model on the validation set.

[0028] In some embodiments of the present invention, the adaptive weight is determined in the following manner:

[0029]

[0030] in, represents the adaptive weight, Represents the weight corresponding to the first distance loss or the second distance loss, Indicates that the final teacher model is The accuracy on the validation set of the federation, Indicates the The accuracy of the sub-federation model of each sub-federation on its validation set.

[0031] In some embodiments of the present invention, the fourth total loss is calculated as follows:

[0032]

[0033] in, Indicates the The fourth total loss corresponding to the sub-federation is Indicates the The number of samples in the training set of each federation, Represents a sample The corresponding label ,Label Indicative sample The category to which it belongs, Indicates the The training set of the federation, Indicates the The classification loss of the sub-federation model for the sample, Indicates the A sub-federation model of sub-federations, represents the first distance loss, represents the feature extraction layer of the final teacher model, Indicates the The feature extraction layer of the sub-federation model of the sub-federation, Represents the adaptive weight.

[0034] In some embodiments of the present invention, the fifth total loss is calculated as follows:

[0035]

[0036] in, Indicates the The fifth total loss corresponding to the sub-federation, Indicates the The number of samples in the training set of each federation, Represents a sample The corresponding label ,Label Indicative sample The category to which it belongs, Indicates the The training set of the federation, Indicates the The classification loss of the sub-federation model for the sample, represents the second distance loss, represents the adaptive weight, represents the final teacher model, Indicates the A sub-federation model of sub-federations, Represents the preset temperature hyperparameter, where the values ​​in the classification prediction layer of the final teacher model and the sub-federated model before being processed by their softmax functions are divided by the preset temperature hyperparameter and serve as the input of their softmax functions.

[0037] In some embodiments of the present invention, all sub-federation models are used in one of the following classification scenarios:

[0038] Used to predict the user's behavior category based on the collected motion sensor data, including: lying down, sitting, standing, walking, running, cycling, Nordic walking, watching TV, working on a computer, driving a car, going upstairs, going downstairs, skipping rope, playing football, cleaning the house, folding clothes, ironing clothes, using a vacuum cleaner; or

[0039] Used to predict the user's lung health category based on the collected lung images, including: healthy, viral pneumonia, bacterial pneumonia; or

[0040] It is used to segment the patient's organs based on the collected CT images. The segmented categories include heart, left lung, right lung, liver, spleen, pancreas, left kidney, right kidney, bladder, left femoral head, and right femoral head.

[0041] According to a second aspect of the present invention, a classification method is provided, which is applied to a client, wherein the client belongs to a sub-federation, and the method comprises: obtaining a sub-federation model of the sub-federation where the client is located and deploying it as a client prediction model, wherein the sub-federation model is trained using the method described in the first aspect; inputting a sample to be predicted into the client prediction model to obtain a classification result.

[0042] Compared with the prior art, the advantages of the present invention are:

[0043] The present invention provides sub-federation model training for multiple sub-federations in a central server-less scenario, wherein each sub-federation is a federation composed of multiple clients of the same organization. All clients of the sub-federation aggregate the collected samples to the device representing the sub-federation, and each sub-federation uses its own training set to locally train the sub-federation model; the sub-federation model is sequentially transferred between sub-federations in a predetermined transfer order to use its knowledge to help the sub-federation model of the next sub-federation complete training. Knowledge transfer and accumulation can be completed between sub-federations without the aid of an external server. After several predetermined sequential transfers, a sub-federation model containing public knowledge can be obtained as a final teacher model and transferred to each sub-federation. All sub-federations are trained again using the local training set and the final teacher model to be more suitable for predicting the personalized data of the sub-federation, thereby improving the prediction accuracy of the model; at the same time, during training, only the sub-federation model is transferred between sub-federations without transferring samples, and the local data set of a sub-federation will not flow out of the sub-federation, thereby ensuring data privacy. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The embodiments of the present invention are further described below with reference to the accompanying drawings, in which:

[0045] Figure 1 Schematic diagram of a flow chart of a federated model training method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below through specific embodiments in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0047] As mentioned in the background technology section, in federated learning scenarios, many existing federation methods cannot be directly applied without a server. Moreover, data distribution differences often exist in different federations, and the direct application of existing federation methods may lead to reduced accuracy of federated individuals. Therefore, the present invention provides a federation-oriented model training method for training sub-federation models of multiple sub-federations, wherein each sub-federation is a federation composed of multiple clients of the same organization. All clients of the sub-federation aggregate the collected samples on the device representing the sub-federation, and each sub-federation uses its own training set to locally train the sub-federation model; the sub-federation model is sequentially transferred between sub-federations in a predetermined transfer order to use its knowledge to help the sub-federation model of the next sub-federation complete training. Knowledge transfer and accumulation can be completed between sub-federations without the aid of an external server. After several predetermined sequential transfers, a sub-federation model containing public knowledge can be obtained as the final teacher model and transferred to each sub-federation. All sub-federations are trained again using the local training set and the final teacher model to be more suitable for predicting the personalized data of the sub-federation, thereby improving the prediction accuracy of the model; at the same time, during training, only the sub-federation model is transferred between sub-federations without transferring samples. The local data set of a sub-federation will not flow out of the sub-federation, thereby ensuring data privacy.

[0048] Before describing the embodiments of the present invention in detail, some of the terms used therein are explained as follows:

[0049] A sub-federation is a federation composed of multiple clients from the same organization. Data can be shared within a sub-federation, but not between sub-federations. For example, all clients in a sub-federation can aggregate collected data samples (referred to as samples) to a device representing the sub-federation (such as a predetermined server or a client designated from the organization's multiple clients) to achieve data sharing and form a dataset for the sub-federation. Data samples do not interact between sub-federations. For example, clients from the same hospital constitute a sub-federation, with different hospitals corresponding to different sub-federations. For another example, the same unit studying the exercise status of users constitutes a sub-federation, with different units studying the exercise status of users corresponding to different sub-federations.

[0050] A federation of federations is a general federation composed of multiple sub-federations. Within the general federation, each sub-federation is organized to train its sub-federation model according to the model training method for the federation of federations.

[0051] To better illustrate the principles of the present invention, the following is an explanation from four aspects: model structure, training samples used, training process, and application scenarios.

[0052] 1. Model Structure

[0053] According to one embodiment of the present invention, a sub-federated model is a neural network model (or neural network classification model) used to perform classification tasks. Each sub-federated model includes a feature extraction layer for extracting sample features and a classification prediction layer for classifying based on the sample features. Depending on the specific classification task being performed, the sub-federated model can employ any neural network model suitable for the sub-federated classification task, and the present invention imposes no limitations on this. For example, for overall image classification, ResNet models (such as ResNet18, ResNet34, or ResNet50), AlexNet models, VGG models (such as VGG16, VGG19), and GoogleNet models can be used. For classification of different image regions (also referred to as segmentation in some literature), UNet models, SegNet models, and FCN models can be used. For time series classification, LSTM models and Transformer models can be used. Importantly, the sub-federation models used by different sub-federations can be the same neural network model or different neural network models, but the classification space used by the sub-federation models is the same (or, in other words: the classification task is the same, the label space is the same, and the set of all possible output categories is the same). It should be understood that for implementations using existing neural network models, those skilled in the art can easily divide the feature extraction layer and the classification prediction layer. The following uses the structural division of two existing models as an example, and the rest is not elaborated on. For example, in the ResNet model, conv1, conv2_x, conv3_x, conv4_x, conv5_x, and the average pooling layer are collectively used as the feature extraction layer, and the output of conv5_x is used as the sample feature; the fully connected layer (1000-d fc) and the softmax function are collectively used as the classification prediction layer. For another example, in the VGG model, the penultimate fully connected layer (FC-4096) and all layers before it are used as the feature extraction layer, and the last fully connected layer and the softmax function are collectively used as the classification prediction layer.

[0054] 2. Training samples used

[0055] According to one embodiment of the present invention, samples collected by all clients of a sub-federation (referred to as samples) are aggregated to the device representing the sub-federation to form a local dataset for the sub-federation. The samples are labeled to indicate the category to which the sample corresponds (the label can be added by the client or after being transferred to the device representing the sub-federation). The category indicated by the label can be determined based on the classification task to be implemented. For example, when predicting user behavior categories based on collected motion sensor data, the labels may include: lying down, sitting, standing, walking, running, cycling, Nordic walking, watching TV, working on a computer, driving, going upstairs, going downstairs, skipping rope, playing soccer, cleaning the house, folding clothes, ironing clothes, and using a vacuum cleaner. For another example, when predicting a user's lung health category based on collected lung images, the labels may include: healthy, viral pneumonia, and bacterial pneumonia. For another example, when performing organ segmentation on a patient's organs based on collected CT images, the segmented categories may include: heart, left lung, right lung, liver, spleen, pancreas, left kidney, right kidney, bladder, left femoral head, and right femoral head. It should be understood that the neural network classification model can be applied to classification tasks in a variety of fields. The above classification scenarios are only illustrative. Those skilled in the art can adjust and set them as needed. The present invention does not impose any restrictions on this.

[0056] Before training, the dataset can be further divided into training set and validation set, or into training set, validation set and test set; usually in a certain ratio, such as 8:2, or 8:1:1 or 7:2:1. For example, suppose there are N different sub-federations, denoted as ,in, Indicates the sub-federations; the dataset corresponding to each sub-federation is ,in, Indicates the The datasets of each sub-federation are divided into three parts: training set, ... , validation set , test set .in, represents the jth sample of the training set of the i-th sub-federation (the input of the training data), represents the label of the jth sample in the training set of the i-th sub-federation, Indicates that the training set of the i-th sub-federation contains samples (sample number 1- ); represents the jth sample of the validation set of the i-th sub-federation (the input of the validation data), represents the label of the jth sample in the validation set of the i-th sub-federation, Indicates that the validation set of the ith sub-federation contains samples (sample number 1- ); represents the jth sample of the test set of the i-th sub-federation (the input of the test data), represents the label of the jth sample in the test set of the i-th sub-federation, Indicates that the test set of the i-th sub-federation contains samples (sample number 1- ). Obviously, the total number of samples in the dataset of the i-th sub-federation is , the dataset of the i-th sub-federation .

[0057] Because collecting training samples is cumbersome and costly, this paper uses the following existing dataset as an example to illustrate dataset formats. For experiments, the dataset below can be divided into multiple equal parts, with each part serving as a sub-federated dataset. In actual sample collection, the following datasets can also be used to collect and label samples according to their format requirements.

[0058] Example 1

[0059] Dataset name: PAMAP2

[0060] Task: Time Series Classification

[0061] Literature: Reiss, Attila, and Didier Stricker. "Introducing a newbenchmarked dataset for activity monitoring." 2012 16th internationalsymposium on wearable computers. IEEE, 2012.

[0062] Each input sample format: 27 channels of time series data (collected by three 9-axis motion sensors)

[0063] Classification space: Contains 18 behavioral categories, namely: lying down, sitting, standing, walking, running, cycling, Nordic walking, watching TV, working on a computer, driving a car, going upstairs, going downstairs, skipping rope, playing football, cleaning the house, folding clothes, ironing clothes, using a vacuum cleaner

[0064] Example 2

[0065] Dataset name: COVID pneumonia

[0066] Task: Image Classification

[0067] Literature: Sait, Unais, et al. "Curated dataset for COVID-19 posterior-anterior chest radiography images (X-Rays)." Mendeley Data 1 (2020).

[0068] Sample format: 2D X-ray image

[0069] Classification space: Contains three categories: health, viral pneumonia, and bacterial pneumonia

[0070] Example 3

[0071] NameOrganSMNIST

[0072] Task: Image Classification

[0073] Literature: Yang, Jiancheng, Rui Shi, and Bingbing Ni. "Medmnistclassification decathlon: A lightweight automl benchmark for medical image analysis." 2021 IEEE 18th International Symposium on Biomedical Imaging(ISBI). IEEE, 2021.

[0074] Sample format: 2D CT image

[0075] Classification space: Contains 11 organ categories, namely: heart, left lung, right lung, liver, spleen, pancreas, left kidney, right kidney, bladder, left femoral head, and right femoral head.

[0076] 3. Training Process

[0077] The present invention provides a federation-oriented model training method, see Figure 1 The method includes steps: S1, S2, and S3. In order to better understand the present invention, each step will be described in detail below in conjunction with specific embodiments.

[0078] In step S1, in each sub-federation, the sub-federation model is initialized and trained locally using the training set of the sub-federation, wherein the classification space of the sub-federation models corresponding to different sub-federations is the same.

[0079] According to one embodiment of the present invention, when the sub-federated model is locally initialized for training, samples in the training set are input into the sub-federated model, the classification of the samples is output, and the parameters of the sub-federated model are updated based on the loss value calculated based on the classification of the output samples and the corresponding labels.

[0080] In step S2, the sub-federation model of the previous sub-federation is transferred to the next adjacent sub-federation according to a predetermined transfer order, and multiple rounds of federated training are performed on each of the sub-federations to obtain sub-federation models containing common knowledge. Each complete transfer process performed according to the predetermined transfer order completes one round of federated training, and each sub-federation completes the training of its own sub-federation model using the transferred previous sub-federation model and the local training set. In other words, in step S2, the sub-federation models are sequentially transferred between the sub-federations according to the predetermined transfer order to perform multiple rounds of federated training, to obtain sub-federation models containing common knowledge. Each complete transfer process performed according to the predetermined transfer order corresponds to one round of federated training, and each sub-federation completes the training of its own sub-federation model using the knowledge of the transferred sub-federation model and the local training set in each round.

[0081] According to one embodiment of the present invention, in step S2, each round of federated training includes: starting with the top-ranked sub-federation, sequentially passing the sub-federation model trained in the current round by the previous sub-federation to the next adjacent sub-federation as an intermediate teacher model to complete the training of the sub-federation model of the next sub-federation in the current round, according to a predetermined transfer order. In the initial round, the sub-federation model of the top-ranked sub-federation passes the initialized sub-federation model to the next sub-federation. In subsequent rounds, the top-ranked sub-federation uses the sub-federation model passed by the last sub-federation in the previous round to complete the training of its own sub-federation model before passing it to the next sub-federation. Simply put, step S2 is a stage of public knowledge accumulation. First, the model passed by the previous sub-federation is obtained and used as the teacher model. The local sub-federation model serves as the student model. Using the local data of the sub-federation and under the guidance of the teacher model, the local student model is learned. After the student model completes learning, it passes the learned model to the next sub-federation end according to the predetermined transfer order. The model of the last sub-federation is passed to the first sub-federation in the order. This process is repeated multiple times to fully accumulate the public useful knowledge of different federations. According to one embodiment of the present invention, the predetermined transfer order may be specified by the implementer, agreed upon by multiple sub-federations, randomly assigned, or determined by the implementer in some other manner, and the present invention imposes no limitations thereon. Each sub-federation appears at least once in the transfer order. Assuming there are four sub-federations, and the predetermined transfer order is set to, for example, 1, 2, 3, and 4, then starting with the sub-federation of the first sub-federation, it first transfers the sub-federation model obtained from initial training in step S1 to the second sub-federation as its intermediate teacher model. The second sub-federation then completes the current round of federated training for that sub-federation based on the intermediate teacher model, and the resulting sub-federation model is transferred to the third sub-federation as its intermediate teacher model... and so on, until the fourth sub-federation transfers its sub-federation model obtained in this round to the first sub-federation, completing a round of federated training. It should be understood that if a sub-federation dataset is highly valuable for improving model accuracy (such as a dataset annotated by an authoritative organization), that sub-federation can appear multiple times in the transfer order to improve prediction accuracy. For example, the predetermined transfer order is set to 1, 2, 3, 2, and 4. Regardless of the delivery order, each sub-federation can deliver and accumulate knowledge based on the predetermined delivery order without a central server to complete federated learning and improve the accuracy of predictions.

[0082] According to one embodiment of the present invention, the sub-federation models used by all sub-federations are homogeneous; for example, all sub-federations use the LSTM model as their sub-federation model. In this case, in step S2, during each training round, the parameters of each sub-federation model are updated based on the total loss determined by the sub-federation model's classification loss and the distance loss between the sample features extracted by the feature extraction layer of the intermediate teacher model and the feature extraction layer of the sub-federation model.

[0083] According to one embodiment of the present invention, among the sub-federation models adopted by all sub-federations, some sub-federation models are homogeneous, while some sub-federation models are heterogeneous. For example, assuming there are four sub-federations, the first and second sub-federations both adopt the ResNet18 model, the third sub-federation adopts the VGG16 model, and the fourth sub-federation adopts the VGG19 model. In this case, in step S2, when the intermediate teacher model of a sub-federation and its sub-federation models are homogeneous models, the parameters of the sub-federation models are updated based on at least the total loss determined by the classification loss of the sub-federation model and the distance loss between the sample features extracted by the feature extraction layer of the teacher model and the feature extraction layer of the sub-federation model; or when the intermediate teacher model of a sub-federation and its sub-federation models are heterogeneous models, the parameters of the sub-federation models are updated based on at least the total loss determined by the classification loss of the sub-federation model and the distance loss between the output of the teacher model and the sub-federation model.

[0084] Preferably, before utilizing the knowledge of the intermediate teacher model, the specific knowledge transfer method can be determined based on the prediction accuracy of the sub-federated model on the local validation set, so that the sub-federated model can have better prediction accuracy.

[0085] According to one embodiment of the present invention, in step S2, when the accuracy of the sub-federated model on the local validation set is greater than a predetermined first threshold, the sub-federated model is trained in the following manner: when the intermediate teacher model and the sub-federated model are isomorphic, the parameters of the sub-federated model are updated based on a first total loss determined by a classification loss of the sub-federated model for the sample and a first distance loss between the sample features extracted by the feature extraction layer of the teacher model and the sample features extracted by the feature extraction layer of the sub-federated model; and when the intermediate teacher model and the sub-federated model are heterogeneous, the parameters of the sub-federated model are updated based on a second total loss determined by a classification loss of the sub-federated model for the sample and a second distance loss between the classification output by the classification prediction layer of the teacher model and the classification output by the classification prediction layer of the federated model. Because the prediction accuracy of the model for tasks in different scenarios may fluctuate significantly, the value of the first threshold can be set by the implementer as needed according to actual conditions, and the present invention does not impose any limitation on this.

[0086] According to one embodiment of the present invention, the first total loss is determined in the following manner:

[0087]

[0088] in, Indicates the The first total loss corresponding to the sub-federation is Indicates the The number of samples in the training set of each federation, Represents a sample The corresponding label ,Label Indicative sample The category to which it belongs, Indicates the The training set of the federation, Indicates the The classification loss of the sub-federation model for the sample, Indicates the A sub-federation model of sub-federations, represents the first distance loss, Indicates the The feature extraction layer of the teacher model of the federation, Indicates the The feature extraction layer of the sub-federation model of the sub-federation, Represents the weight pre-set for the first distance loss.

[0089] According to one embodiment of the present invention, the second total loss is determined in the following manner:

[0090]

[0091] in, Indicates the The second total loss corresponding to the sub-federation is Indicates the The number of samples in the training set of each federation, Represents a sample The corresponding label ,Label Indicative sample The category to which it belongs, Indicates the The training set of the federation, Indicates the The classification loss of the sub-federation model for the sample, represents the second distance loss, represents the weight pre-set for the second distance loss, Indicates the A sub-federation model of sub-federations, Indicates the The teacher model in the middle of the federation, Represents the preset temperature hyperparameter. The values ​​in the classification prediction layer of the intermediate teacher model and the sub-federated model before being processed by their softmax functions are divided by the preset temperature hyperparameter and used as the input of their softmax functions. The temperature hyperparameter is usually set to a value greater than 1, and is used to adjust the distribution of the output sample features to make the distribution smoother, so as to amplify the category similarity to adjust and improve the model's discriminative ability. Of course, the temperature hyperparameter is not required and can be set to 1, which is equivalent to no adjustment; or the set temperature hyperparameter can be directly canceled. The subsequent embodiments involving temperature hyperparameters are similar and will not be repeated here. When the temperature hyperparameter is set, taking the ResNet model as an example, in the middle teacher model, the output of the fully connected layer (1000-d fc) in the classification prediction layer of the middle teacher model is divided by the temperature hyperparameter, and the resulting quotient is input into the middle teacher model's softmax function to calculate the classification of the middle teacher model's output. In the child federated model, the output of the fully connected layer (1000-d fc) in the classification prediction layer of the child federated model is divided by the temperature hyperparameter, and the resulting quotient is input into the child federated model's softmax function to calculate the classification of the child federated model's output. The second distance loss is then calculated based on the classification of the two. In the case of the VGG model, the quotient of the output of the last fully connected layer (FC-1000) divided by the temperature hyperparameter is input into the softmax function. Other procedures are similar and are not detailed here. It should be understood that the temperature hyperparameter here and in the following sections is only used to calculate the loss during training. The temperature hyperparameter is not used in the trained sub-federated model (deployed for prediction). That is, taking the Resnet model as an example, when the trained sub-federated model makes predictions, the output of the fully connected layer (1000-d fc) in the classification prediction layer of the sub-federated model is directly input into the softmax function of the sub-federated model to calculate the classification output of the sub-federated model.

[0092] According to one embodiment of the present invention, in step S2, when the accuracy of the sub-federated model on the local validation set is less than or equal to a predetermined first threshold, the current sub-federated model is trained in the following manner: when the intermediate teacher model is isomorphic to the current sub-federated model, after replacing the parameters of the sub-federated model with the parameters of the intermediate teacher model, the replaced sub-federated model is trained using the training set of the sub-federated model and the parameters of the replaced sub-federated model are updated according to the classification loss (i.e., the sample is input into the replaced sub-federated model for iterative training, and during training, the parameters of the replaced sub-federated model are updated according to the classification loss calculated between the classification of the sample output by the replaced sub-federated model and the corresponding classification label); when the intermediate teacher model is heterogeneous to the current sub-federated model, the parameters of the sub-federated model are updated according to the third total loss determined by the classification loss of the sample by the sub-federated model and the second distance loss between the classification output by the classification prediction layer of the intermediate teacher model and the classification output by the classification prediction layer of the sub-federated model. When the models are isomorphic, if the accuracy of the sub-federated model on the local validation set is less than or equal to a predetermined first threshold, it indicates that the current sub-federated model is performing poorly. Therefore, directly replacing the parameters of the sub-federated model with those of the intermediate teacher model can maximize the knowledge of the intermediate teacher model. Of course, it should be understood that it is also possible to implement it without setting the first threshold. For example, the sub-federated model can be updated by calculating the first total loss without replacing the sub-federated model.

[0093] According to one embodiment of the present invention, the third total loss is determined in the following manner:

[0094]

[0095] in, Indicates the The third total loss corresponding to the sub-federation is Indicates the The number of samples in the training set of each federation, Represents a sample The corresponding label ,Label Indicative sample The category to which it belongs, Indicates the The training set of the federation, Indicates the The classification loss of the sub-federation model for the sample, represents the second distance loss, represents the weight pre-set for the second distance loss, represents the reinforcement factor used to make more use of the teacher model’s knowledge, , Indicates the A sub-federation model of sub-federations, Indicates the The teacher model in the middle of the federation, Represents the preset temperature hyperparameter. The values ​​in the classification prediction layer of the intermediate teacher model and the sub-federation model before being processed by the softmax function are divided by the preset temperature hyperparameter and used as the input of the softmax function. This allows the sub-federation model of the current sub-federation to learn as much knowledge as possible from the teacher model.

[0096] The following takes the ith sub-federation as an example to illustrate the specific training process of the sub-federation in step S2. Assume that the local sub-federation model of the ith federation end is According to the predetermined transmission order, the previous sub-federation of the i-th sub-federation is j, so after the j-th sub-federation completes the training model, it obtains the sub-federation model ,Bundle Passed to the i-th sub-federation, the i-th sub-federation currently has two models, the local sub-federation model (Student model) and the sub-federation model of the previous sub-federation (Teacher Model), now relies on Decide how much to keep knowledge, through Conduct training.

[0097] The current sub-federation model exist Evaluate on and get the corresponding accuracy . And according to The total loss is calculated by comparing it to a first threshold, where:

[0098] 1) If , If it is the first threshold set, then it is considered that the current sub-federation model has sufficient knowledge and we can try to discard some teacher models. This can be accomplished through knowledge distillation. To ensure sufficient generality, two cases are described:

[0099] 1.1) If the teacher model and the student model in the middle are isomorphic, that is, the models are the same, then feature distillation can be used to better preserve the teacher model. Correspondingly, the first total loss is used to update the parameters of the student model.

[0100] 1.2) If the teacher model and the student model are heterogeneous, that is, the model structures are different, then for If there are differences in the output, then feature distillation cannot be performed. In this case, considering that the output label space (classification space) is the same, knowledge distillation can be performed on the final result. Correspondingly, the second total loss is used to update the parameters of the student model.

[0101] 2) If , then it can be considered that the current sub-federation model Contains too little useful information and needs to make the most of the teacher model At this point, there are still two situations:

[0102] 2.1) If the teacher model and the student model are isomorphic, then we can directly use right Initialize and then use Retraining (fine-tuning) , correspondingly, the parameters of the replaced sub-federated model are updated using the following classification loss:

[0103] ;

[0104] in, Indicates the The number of samples in the training set of each federation, Represents a sample The corresponding label ,Label Indicative sample The category to which it belongs, Indicates the The training set of the federation, Indicates the The classification loss of samples by the sub-federation model of the sub-federation.

[0105] 2.2) If the intermediate teacher model and student model are heterogeneous, then direct initialization is no longer possible. In order to make the most of the teacher model , you can try to increase the intensity of learning, and correspondingly, use the third total loss to update the parameters of the student model.

[0106] Local sub-federation model After sufficient training (e.g., training to convergence or training the local training set for a predetermined number of times), the model can be Passed to the next sub-federation k, as the teacher model to guide the sub-federation model of sub-federation k In this way, after multiple iterations according to the predetermined transfer order, the model containing all sub-federation knowledge is finally obtained, that is, the sub-federation model of the last sub-federation in the transfer order, which serves as the final teacher model. .

[0107] In step S3, the sub-federation model containing public knowledge is passed to each sub-federation as the final teacher model. Each sub-federation uses the knowledge of the final teacher model and the local training set to train its sub-federation model to obtain the final sub-federation model of each sub-federation.

[0108] According to one embodiment of the present invention, in step S3, when the final teacher model of a sub-federation and its sub-federation model are isomorphic models, the parameters of the sub-federation model are updated at least based on the total loss determined by the classification loss of the sub-federation model and the distance loss between the sample features extracted by the feature extraction layer of the teacher model and the feature extraction layer of the sub-federation model; or when the final teacher model of a sub-federation and its sub-federation model are heterogeneous models, the parameters of the sub-federation model are updated at least based on the total loss determined by the classification loss of the sub-federation model and the distance loss between the output of the teacher model and the sub-federation model.

[0109] Before utilizing the knowledge of the final teacher model, a specific knowledge transfer method can be determined based on the relative size of the teacher model's accuracy on the current sub-federation's validation set and a predetermined second threshold (equivalent to an evaluation of the prediction level of the final teacher model), and based on the relative size of the final teacher model's accuracy on the current sub-federation's validation set and the prediction accuracy of the sub-federation model of the current sub-federation on the local validation set (equivalent to an evaluation of the accuracy of the final teacher model and student model), so that the sub-federation model can have better prediction accuracy. Among them:

[0110] According to one embodiment of the present invention, in step S3, when the accuracy of the final teacher model on the validation set of the current sub-federation is less than a predetermined second threshold and less than the accuracy of the sub-federation model of the current sub-federation on the validation set, the current sub-federation trains its sub-federation model in the following manner: only using the training set of the current sub-federation to train its sub-federation model once or multiple times to obtain the final sub-federation model of the sub-federation.

[0111] According to one embodiment of the present invention, in step S3, when the accuracy of the final teacher model on the validation set of the current sub-federation is greater than or equal to a predetermined second threshold and / or greater than or equal to the accuracy of the current sub-federation model on the validation set (in other words, when the accuracy of the final teacher model on the validation set of the current sub-federation is less than the predetermined second threshold and less than the accuracy of the sub-federation model of the current sub-federation on the validation set does not hold), the current sub-federation trains its sub-federation model in the following manner: when the final teacher model is isomorphic to the sub-federation model of the current sub-federation, the sub-federation model of the current sub-federation is trained according to the sub-federation model of the current sub-federation. The parameters of the sub-federation model of the current sub-federation are updated according to the fourth total loss determined by the first distance loss between the classification loss of the sample by the sub-federation model and the sample features extracted by the feature extraction layer of its teacher model and the sample features extracted by the feature extraction layer of the sub-federation model of the current sub-federation; when the final teacher model and the sub-federation model of the current sub-federation are heterogeneous, the parameters of the sub-federation model of the current sub-federation are updated according to the fifth total loss determined by the second distance loss between the classification loss of the sub-federation model of the current sub-federation and the classification output by the classification prediction layer of the teacher model and the classification output by the classification prediction layer of the sub-federation model of the current sub-federation.

[0112] According to one embodiment of the present invention, the fourth total loss is calculated as follows:

[0113]

[0114] in, Indicates the The fourth total loss corresponding to the sub-federation is Indicates the The number of samples in the training set of each federation, Represents a sample The corresponding label ,Label Indicative sample The category to which it belongs, Indicates the The training set of the federation, Indicates the The classification loss of the sub-federation model for the sample, Indicates the A sub-federation model of sub-federations, represents the first distance loss, represents the feature extraction layer of the final teacher model, Indicates the The feature extraction layer of the sub-federation model of the sub-federation, Represents the adaptive weight.

[0115] According to one embodiment of the present invention, the fifth total loss is calculated as follows:

[0116]

[0117] in, Indicates the The fifth total loss corresponding to the sub-federation, Indicates the The number of samples in the training set of each federation, Represents a sample The corresponding label ,Label Indicative sample The category to which it belongs, Indicates the The training set of the federation, Indicates the The classification loss of the sub-federation model for the sample, represents the second distance loss, represents the adaptive weight, represents the final teacher model, Indicates the A sub-federation model of sub-federations, Represents the preset temperature hyperparameter. The values ​​in the classification prediction layers of the final teacher model and the sub-federated model, before being processed by their softmax functions, are divided by the preset temperature hyperparameter and used as the input to their softmax functions. When the temperature hyperparameter is set, taking the ResNet model as an example, in the final teacher model, the quotient obtained by dividing the output of the fully connected layer (1000-d fc) in the classification prediction layer of the final teacher model by the temperature hyperparameter is input into the softmax function of the final teacher model to calculate the classification of the final teacher model output. In the sub-federated model, the quotient obtained by dividing the output of the fully connected layer (1000-d fc) in the classification prediction layer of the sub-federated model by the temperature hyperparameter is input into the softmax function of the sub-federated model to calculate the classification of the sub-federated model output. The second distance loss is then calculated based on the classification of the two. For the VGG model, the quotient obtained by dividing the output of the last fully connected layer (FC-1000) by the temperature hyperparameter is input into the softmax function. Other processes are similar and are not detailed here.

[0118] According to one embodiment of the present invention, in step S3, each sub-federation independently determines an adaptive weight for the first distance loss or the second distance loss in the sub-federation based on the accuracy of the final teacher model on the validation set of the sub-federation and the accuracy of its sub-federation model on the validation set. Preferably, the adaptive weight is determined as follows:

[0119]

[0120] in, represents the adaptive weight, represents the weight corresponding to the first distance loss or the second distance loss (i.e., the weight of the first distance loss or the second distance in step S2),), Indicates that the final teacher model is The accuracy on the validation set of the federation, Indicates the The accuracy of the sub-federation model of each sub-federation on its validation set. It should be understood that the specific calculation method of the adaptive weight can be adjusted to produce other specific implementations. For example, in the above formula, 10 can be changed to 8 or 6, and / or 5 can be changed to 6 or 4. The present invention does not impose any limitations on this.

[0121] According to an example of the present invention, step S3 is equivalent to the personalized training stage of the sub-federation model. First, the model containing all sub-federation knowledge is trained. Delivered to each sub-federation, in each sub-federation As a teacher model, it combines the data of the local sub-federation to conduct local sub-federation model Finally, a personalized model suitable for classifying samples of the sub-federation is obtained.

[0122] Assume that each sub-federation has acquired the public knowledge accumulation model , the following takes the ith sub-federation as an example to illustrate the specific training process of the sub-federation. Assume that the local sub-federation model of the ith sub-federation is , now relying on Decide how much to keep knowledge, through the current sub-federation Conduct training.

[0123] Sub-federation model exist Evaluate on and get the corresponding accuracy , public knowledge accumulation model (the final teacher model) exist Evaluate on and get the corresponding accuracy ,in:

[0124] 3) If ,and , is the second threshold set artificially, then it is considered that the final teacher model performs poorly in the current sub-federation and is worse than the student model, that is, the final teacher model If the current sub-federation has almost no effect, we can consider ignoring the knowledge of the final teacher model (using a degree of 0) and directly training its sub-federation model using the training set of the current sub-federation; correspondingly, only the classification loss of the sample is used to update the parameters of the sub-federation model:

[0125] ;

[0126] 4) If and / or , then the final teacher model It has a certain reference significance, so we try to retain some useful teacher knowledge through knowledge distillation. Considering that we want to get a personalized model, we expect to discard some of the final teacher model knowledge, so we try to change the corresponding hyperparameters in knowledge distillation and calculate according to the above adaptive weight formula. ,in, The larger the value, the greater the reliance of the personalization stage on the teacher model, and the more the teacher model is better than the student model, The larger the value, the greater the reference significance of the teacher model. The more the student model refers to the teacher. Considering the two cases of model homogeneity and heterogeneity, we still discuss them separately:

[0127] 4.1) When the final teacher model is isomorphic to the sub-federation model of the current sub-federation, knowledge transfer is still performed through feature distillation. Correspondingly, the fourth total loss is used to update the parameters of the sub-federation model of the current sub-federation.

[0128] 4.2) When the final teacher model is heterogeneous with the sub-federation model of the current sub-federation, knowledge distillation is performed through the output. Correspondingly, the fifth total loss is used to update the parameters of the sub-federation model of the current sub-federation.

[0129] After the training process of steps S1-S3, a model can be trained for each sub-federation end by combining the data information of all sub-federations. , so that the average effect on all sub-federations is the best to achieve the target loss function:

[0130] ;

[0131] Where N is the total number of sub-federations, represents the sub-federation model of the k-th sub-federation, represents the classification loss of the i-th sub-federated model on its local test set. The remaining parameters have been introduced in the second part (the training samples used) and are not repeated here.

[0132] In step S3, the present invention provides personalized training solutions for isomorphic and heterogeneous models based on different knowledge distillation methods (feature distillation and result distillation), respectively, making it possible for sub-federations to obtain personalized models with different structures. In addition, the present invention adopts corresponding model training methods based on the different performances of the final teacher model in the local validation set of the sub-federation to further improve the prediction accuracy of the sub-federation model.

[0133] IV. Application Scenarios

[0134] According to one embodiment of the present invention, a classification method is provided, which is applied to a client, wherein the client belongs to a sub-federation. The method includes: obtaining a sub-federation model of the sub-federation where the client is located as a client prediction model, wherein the sub-federation model is trained using the method described in the third section; and inputting a sample to be predicted into the client prediction model to obtain a classification result.

[0135] In order to further verify the effectiveness of the method proposed in this invention and to illustrate the method of this invention, the inventors also conducted a simulation experiment on a real data set. The following is an explanation of the experimental process:

[0136] 1. Dataset Introduction

[0137] Name: OrganSMNIST dataset

[0138] Literature: [Bilic, Patrick, et al. "The liver tumor segmentation benchmark(lits)." arXiv preprint arXiv:1901.04056 (2019).]

[0139] This dataset is a large MNIST-like biological image collection from the MedMNIST dataset [Yang, Jiancheng, Rui Shi, and Bingbing Ni. "MedMNIST classification decathlon: A lightweight autoML benchmark for medical image analysis." 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI). IEEE, 2021.]. It consists of 25,221 CT images from 11 categories. The dataset is unevenly divided into 40 parts, each corresponding to a client. Every two clients form a sub-federation, for a total of 20 sub-federations. In each sub-federation, 40% of the data is used for model training, 30% for validation, and 30% for testing. This means that the dataset is divided into training, validation, and testing sets in a 4:3:3 ratio.

[0140] 2. Model structure and hyperparameter settings

[0141] For convenience, only the model and data isomorphism are verified, because the image size is , so the LeNet5 model is used (the input adjustment of the model is set to ) were used for the experiments. Gradient descent (SGD) was used for optimization, and the learning rate was set to 0.01.

[0142] 3. Comparison method

[0143] To demonstrate the effectiveness of the present invention, a comparison is made with a baseline method and two state-of-the-art methods. Since there is no suitable algorithm without a central server, the following algorithms with a central server are used for comparison:

[0144] Method 1: FedAVG [McMahan, Brendan, et al. "Communication-efficient learning of deep networks from decentralized data." Artificial intelligence and statistics. PMLR, 2017.]: This method directly aggregates model parameters without considering personalization.

[0145] Method 2: FedProx[Li, Tian, ​​et al. "Federated optimization inheterogeneous networks." Proceedings of Machine Learning and Systems 2(2020): 429-450.]: This method allows the local model to be somewhat different from the model on the central server through an approximation term.

[0146] Method 3: FedBN [Li, Xiaoxiao, et al. "Fedbn: Federated learning on non-iid features via local batch normalization." arXiv preprint arXiv:2102.07623(2021).].

[0147] 4. Evaluation indicators

[0148] In existing methods, there is no concept of sub-federation, and the average accuracy is calculated using all the models trained;

[0149] The method of the present invention uses the model of each sub-federation to test on the test data set of each sub-federation, records the accuracy, and finally compares the average accuracy (average prediction accuracy, in %) of all sub-federations.

[0150] 5. Analysis of experimental results

[0151] Table 1 Experimental results

[0152]

[0153] Table 1 shows the experimental results. The non-iid alpha parameter setting represents the degree of imbalance in the datasets across each federation. A smaller value indicates a more pronounced imbalance. Table 1 shows that compared to Method 1 (FedAVG, the baseline method), Method 2 (FedProx) shows almost no improvement in prediction accuracy, while the personalized method (FedBN) shows some improvement. The proposed method achieves significant improvements. Compared to the baseline method, when the partition parameter is set to 0.01, the proposed method achieves an improvement of over 10%, and when the partition parameter is set to 0.1, the proposed method achieves an improvement of over 8%. These experimental results fully demonstrate the effectiveness of the proposed method.

[0154] In summary, this invention effectively addresses the challenges of difficult deployment, decentralized nature, difficulty in personalization, and low precision in federated learning for federation, obtaining high-precision sub-federated models for each federated client. Furthermore, by employing different parameter update methods for homogeneous and heterogeneous models, this invention can handle multiple scenarios, including homogeneous and heterogeneous models, and spatial inconsistencies in input data. This allows for the generation of high-precision sub-federated models for each federated client in a variety of complex real-world scenarios. Furthermore, the system is stable and, under real-world communication cost constraints, ensures relatively high accuracy and robustness for each client.

[0155] It should be noted that although the above describes the various steps in a specific order, it does not mean that the steps must be performed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order as long as the required functions can be achieved.

[0156] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0157] A computer-readable storage medium may be a tangible device that holds and stores instructions used by an instruction execution device. Computer-readable storage media may include, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove having instructions stored thereon, and any suitable combination thereof.

[0158] While various embodiments of the present invention have been described above, the above descriptions are intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A federated model training method, characterized in that: The method is used for training a sub-federation model for multiple sub-federations, where each sub-federation is composed of multiple clients of the same organization. All clients of the sub-federation aggregate collected samples on a device representing the sub-federation to form a dataset containing a training set and a validation set for the sub-federation. The method includes: S1. In each sub-federation, the sub-federation model is initialized and trained locally using the training set of the sub-federation. The classification space of the sub-federation models corresponding to different sub-federations is the same. S2. The sub-federation model of the previous sub-federation is transferred to the next adjacent sub-federation in a predetermined transfer order, and multiple rounds of federation training are performed on each sub-federation to obtain a sub-federation model containing public knowledge. Each round of federation training is completed by performing a complete transfer process in the predetermined transfer order. Each sub-federation uses the knowledge transferred from the previous sub-federation model and the local training set to complete the training of its own sub-federation model. S3. The sub-federation model containing public knowledge is passed to each sub-federation as the final teacher model. Each sub-federation uses the knowledge of the final teacher model and the local training set to train its sub-federation model to obtain the final sub-federation model of each sub-federation.

2. The method according to claim 1, characterized in that In step S2, each round of federation training includes: starting from the sub-federation ranked first, in accordance with a predetermined transfer order, the sub-federation model of the previous sub-federation after training in this round is sequentially transferred to the adjacent next sub-federation as an intermediate teacher model to complete the training of the sub-federation model of the next sub-federation in this round, wherein the sub-federation model of the sub-federation ranked first in the initial round transfers the initialized sub-federation model to the next sub-federation, and the sub-federation ranked first in other rounds uses the sub-federation model transferred from the last sub-federation in the previous round to complete the training of its own sub-federation model and then transfers it to the next sub-federation.

3. The method according to claim 2, characterized in that Each sub-federation model includes a feature extraction layer for extracting sample features and a classification prediction layer for classifying according to the sample features. Different sub-federations can adopt heterogeneous or homogeneous sub-federation models, where: When the teacher model of a sub-federation is isomorphic to its sub-federation model, updating the parameters of the sub-federation model based on at least the total loss determined by the classification loss of the sub-federation model and the distance loss between the sample features extracted by the feature extraction layer of the teacher model and the feature extraction layer of the sub-federation model; or When a teacher model of a sub-federation and its sub-federation models are heterogeneous models, the parameters of the sub-federation model are updated according to at least the total loss determined by the classification loss of the sub-federation model and the distance loss between the outputs of the teacher model and the sub-federation model.

4. The method according to claim 3, characterized in that In step S2, when the accuracy of the sub-federation model on the local validation set is greater than a predetermined first threshold, the sub-federation trains its sub-federation model in the following manner: When the intermediate teacher model is isomorphic to the sub-federated model, updating the parameters of the sub-federated model according to a first total loss determined by the classification loss of the sub-federated model for the sample and a first distance loss between the sample features extracted by the feature extraction layer of the teacher model and the sample features extracted by the feature extraction layer of the sub-federated model; When the intermediate teacher model is heterogeneous with the sub-federated model, the parameters of the sub-federated model are updated according to a second total loss determined by the classification loss of the sub-federated model for the sample and a second distance loss between the classification output by the classification prediction layer of the teacher model and the classification output by the classification prediction layer of the federated model.

5. The method according to claim 4, characterized in that The first total loss is determined as follows: in, Indicates the The first total loss corresponding to the sub-federation is Indicates the The number of samples in the training set of each federation, Represents a sample The corresponding label ,Label Indicative sample The category to which it belongs, Indicates the The training set of the federation, Indicates the The classification loss of the sub-federation model for the sample, Indicates the A sub-federation model of sub-federations, represents the first distance loss, Indicates the The feature extraction layer of the teacher model of the federation, Indicates the The feature extraction layer of the sub-federation model of the sub-federation, Represents the weight pre-set for the first distance loss.

6. The method according to claim 4, characterized in that The second total loss is determined as follows: in, Indicates the The second total loss corresponding to the sub-federation is Indicates the The number of samples in the training set of each federation, Represents a sample The corresponding label ,Label Indicative sample The category to which it belongs, Indicates the The training set of the federation, Indicates the The classification loss of the sub-federation model for the sample, represents the second distance loss, represents the weight pre-set for the second distance loss, Indicates the A sub-federation model of sub-federations, Indicates the The teacher model in the middle of the federation, Represents the preset temperature hyperparameter. The values ​​in the classification prediction layer of the intermediate teacher model and the sub-federated model before being processed by their softmax function are divided by the preset temperature hyperparameter and used as the input of their softmax function.

7. The method according to claim 4, characterized in that In step S2, when the accuracy of the sub-federation model on the local validation set is less than or equal to a predetermined first threshold, the current sub-federation trains its sub-federation model in the following manner: When the intermediate teacher model is isomorphic to the current sub-federated model, the parameters of the sub-federated model are replaced with the parameters of the intermediate teacher model, and the classification loss of the sample is used to update the parameters of the replaced sub-federated model; When the intermediate teacher model is heterogeneous with the current sub-federated model, the parameters of the sub-federated model are updated according to the third total loss determined by the classification loss of the sub-federated model on the sample and the second distance loss between the classification output by the classification prediction layer of the intermediate teacher model and the classification output by the classification prediction layer of the sub-federated model.

8. The method according to claim 7, characterized in that The third total loss is determined as follows: in, Indicates the The third total loss corresponding to the sub-federation is Indicates the The number of samples in the training set of each federation, Represents a sample The corresponding label ,Label Indicative sample The category to which it belongs, Indicates the The training set of the federation, Indicates the The classification loss of the sub-federation model for the sample, represents the second distance loss, represents the weight pre-set for the second distance loss, represents the reinforcement factor used to make more use of the teacher model’s knowledge, , Indicates the A sub-federation model of sub-federations, Indicates the The teacher model in the middle of the federation, Represents the preset temperature hyperparameter. The values ​​in the classification prediction layer of the intermediate teacher model and the sub-federated model before being processed by their softmax function are divided by the preset temperature hyperparameter and used as the input of their softmax function.

9. The method according to any one of claim 3, characterized in that In step S3, when the accuracy of the final teacher model on the validation set of the current sub-federation is less than a predetermined second threshold and less than the accuracy of the sub-federation model of the current sub-federation on the validation set, the current sub-federation trains its sub-federation model in the following manner: Only the training set of the current sub-federation is used to train its sub-federation model once or multiple times to obtain the final sub-federation model of the sub-federation.

10. The method according to any one of claims 3, characterized in that: In step S3, when the accuracy of the final teacher model on the validation set of the current sub-federation is greater than or equal to a predetermined second threshold and / or greater than or equal to the accuracy of the current sub-federation model on the validation set, the current sub-federation trains its sub-federation model in the following manner: When the final teacher model is isomorphic to the sub-federated model of the current sub-federated model, updating the parameters of the sub-federated model of the current sub-federated model according to a fourth total loss determined by the classification loss of the sample by the sub-federated model of the current sub-federated model and the first distance loss between the sample features extracted by the feature extraction layer of the teacher model and the sample features extracted by the feature extraction layer of the sub-federated model of the current sub-federated model; When the final teacher model is heterogeneous with the sub-federation model of the current sub-federation, the parameters of the sub-federation model of the current sub-federation are updated according to a fifth total loss determined by the classification loss of the sub-federation model of the current sub-federation for the sample and the second distance loss between the classification output by the classification prediction layer of the teacher model and the classification output by the classification prediction layer of the sub-federation model of the current sub-federation.

11. The method according to claim 10, characterized in that In step S3, each sub-federation determines the adaptive weight set for the first distance loss or the second distance loss in the sub-federation based on the accuracy of the final teacher model on the validation set of the sub-federation and the accuracy of its sub-federation model on the validation set.

12. The method according to claim 11, characterized in that The adaptive weight is determined as follows: in, represents the adaptive weight, Represents the weight corresponding to the first distance loss or the second distance loss, Indicates that the final teacher model is The accuracy on the validation set of the federation, Indicates the The accuracy of the sub-federation model of each sub-federation on its validation set.

13. The method according to claim 12, characterized in that The fourth total loss is calculated as follows: in, Indicates the The fourth total loss corresponding to the sub-federation is Indicates the The number of samples in the training set of each federation, Represents a sample The corresponding label ,Label Indicative sample The category to which it belongs, Indicates the The training set of the federation, Indicates the The classification loss of the sub-federation model for the sample, Indicates the A sub-federation model of sub-federations, represents the first distance loss, represents the feature extraction layer of the final teacher model, Indicates the The feature extraction layer of the sub-federation model of the sub-federation, Represents the adaptive weight.

14. The method according to claim 12, characterized in that The fifth total loss is calculated as follows: in, Indicates the The fifth total loss corresponding to the sub-federation, Indicates the The number of samples in the training set of each federation, Represents a sample The corresponding label ,Label Indicative sample The category to which it belongs, Indicates the The training set of the federation, Indicates the The classification loss of the sub-federation model for the sample, represents the second distance loss, represents the adaptive weight, represents the final teacher model, Indicates the A sub-federation model of sub-federations, Represents the preset temperature hyperparameter, where the values ​​in the classification prediction layer of the final teacher model and the sub-federated model before being processed by their softmax function are divided by the preset temperature hyperparameter and serve as the input of their softmax function.

15. The method according to any one of claims 1 to 8, characterized in that All sub-federation models are used in one of the following classification scenarios: Used to predict the user's behavior category based on the collected motion sensor data, including: lying down, sitting, standing, walking, running, cycling, Nordic walking, watching TV, working on a computer, driving a car, going upstairs, going downstairs, skipping rope, playing football, cleaning the house, folding clothes, ironing clothes, using a vacuum cleaner; or Used to predict the user's lung health category based on the collected lung images, including: healthy, viral pneumonia, bacterial pneumonia; or It is used to segment the patient's organs based on the collected CT images. The segmented categories include heart, left lung, right lung, liver, spleen, pancreas, left kidney, right kidney, bladder, left femoral head, and right femoral head.

16. A classification method, applied to a client, wherein the client belongs to a sub-federation, characterized in that: The method comprises: Obtaining a sub-federation model of the sub-federation where the client is located as a client prediction model, where the sub-federation model is trained using the method according to any one of claims 1 to 15; Input the sample to be predicted into the client prediction model to obtain the classification result.

Images