An adaptive federated learning method based on style transfer exploration

Abstract

The application discloses a kind of self-adapting federated learning method based on style transfer exploration, belong to federated learning field, including: establishing style sharing center;Selective style transfer;Using decoder converts the feature information after style transfer into image with the style of other participants;In the original sample data of each participant, style exploration is carried out, and new style information feature is obtained;The style information feature explored is selectively migrated into original sample data using AdaIN model;Distributed training generates local model;Using public uniform distribution dataset, the model uploaded by each participant is verified, the model score of each participant is calculated, and then the PDVFedMul aggregation algorithm is used to aggregate the local model, so that the global model parameters of the new round are obtained.The application improves the generalization ability of the model, enhances the robustness of the model, provides a more reasonable weight distribution scheme for model aggregation, and improves the model performance.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of federated learning, and particularly relates to a self-adaptive federated learning method based on style transfer exploration. BACKGROUND

[0002] An emerging artificial intelligence technology, the proposal of the federated learning idea brings enlightenment to solve the current widely existing "data island" problem. Its running mechanism can safely link each participant, without sharing and migrating own local data, directly training the model locally, then sending the trained model to the central aggregation server, and then the central aggregation server performs unified aggregation to form a global model. The transmission of parameters in the training process is replaced from the original remote data transmission to encrypted transmission, thereby ensuring the safety of the data.

[0003] However, the model trained by most current federated learning performs well when tested on the data of the participants, but if the model is applied to institutions other than the participants, the model performance will often decline. For example, in a medical scenario, the patient data of each medical institution needs to be used for federated learning. The medical image imaging devices used by different medical institutions are different, and the imaging results of the same patient may also differ when the patient goes to two different hospitals for medical image examination of the same part. The trained model will not only be used in the medical institutions participating in the training, but also in many other medical institutions. Therefore, in order to improve the robustness of the model, the problem of federated domain generalization needs to be considered.

[0004] In the aspect of federal domain generalization, in 2017, Motiian S et al. proposed a unified framework for deep model supervised domain adaptation and generalization, which mapped source domain samples to a shared embedding space so that samples with the same class label but different domains were semantically aligned in this space. The semantic alignment loss was introduced to minimize the distance between samples from the same class but different domains, ensuring that these samples were close in the embedding space. To avoid the overlap of samples of different classes in the embedding space, the class separation loss was further introduced to maximize the distribution distance of samples of different classes. The final objective function combined the classification loss, semantic alignment loss, and class separation loss, forming a contrastive semantic alignment loss. This joint loss ensured that both class alignment and class separation were achieved during the training process, while improving the classification performance (Motiian S, Piccirilli M, Adjeroh D A, et al. Unified deep supervised domain adaptation and generalization [C]. Proceedings of the IEEE International Conference on Computer Vision, 2017: 5715-5725.). In 2019, Li Y et al. addressed the problem of heterogeneous domain generalization, i.e., in the case of source and target domains with different label spaces, how to train a feature representation model to effectively identify new data and new classes in new target domains. A meta-learning based scenario training paradigm was proposed, which stimulated domain transfer by creating meta-training and meta-testing splits at each iteration. Meta-learning was used to learn auxiliary losses to guide the feature extractor to learn more general features (Li Y, Yang Y, Zhou W, et al. Feature-critic networks for heterogeneous domain generalization [C]. International Conference on Machine Learning, 2019: 3915-3924.). However, the above methods usually need to collect multiple domain data to learn, which does not meet the privacy protection requirements in the federated learning setting.There are other methods that do not require data centralization, such as data augmentation (Volpi R, Namkoong H, Sener O, et al. Generalizing to unseen domains via adversarial data augmentation [J]. Advances in Neural Information Processing Systems, 2018, 31. 109845), self-supervised signal (Wang S, Yu L, Li C, et al. Learning from extrinsic and intrinsic supervisions for domain generalization [C]. European Conference on Computer Vision, Cham: Springer International Publishing, 2020: 159-176.), and operating deep neural networks (Matsuura T, Harada T. Domain generalization using a mixture of multiple latent domains [C]. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(07): 11749-11756.) to perform domain generalization. These methods can effectively regularize the local training of single source domain data, but it is difficult to utilize the rich cross-domain data distribution. In 2021, Quande Liu et al. proposed a context learning method based on continuous frequency space for federated domain generalization to address the inter-domain difference problem of data distribution in medical image segmentation tasks (Quande Liu, Cheng Chen, Jing Qin, Qi Dou, and Pheng-Ann Heng. Feddg: Federated domain generalization on medical image segmentation via episodic learning in continuous frequency space. In CVPR, 2021. 1, 3, 6, 11). This method decomposes the image information of local data into an amplitude spectrum representing low-level distribution information and a phase spectrum representing high-level semantics. The phase spectrum is locally preserved to protect privacy, while a shared distribution library is established on the central aggregation server to save the amplitude spectrum of the local image signal of each participant, enabling the transfer of distribution information across clients.The local amplitude spectrum is continuously interpolated with the amplitude spectrum of other clients in the low-frequency region to generate images containing multi-source distributions. The original images are treated as a meta-training set, and the interpolated images are treated as a meta-testing set. A scenario learning framework is designed to simulate distribution transfer. This method uses continuous frequency space interpolation and boundary-oriented optimization to improve the performance of the model on unseen domains and can also be extended to other tasks with strong distribution heterogeneity. However, the authors did not directly address the fundamental problem of data scarcity and style. In 2022, Oliveira H et al. proposed a meta-learner-based generalization technique for medical image segmentation, aiming to address the challenges of labeled data scarcity and domain differences in medical image segmentation tasks. Meta-learning simulates the training process of few-shot tasks, enabling the model to quickly adapt to new tasks with limited labeled data, thereby reducing the dependence on large-scale labeled data and significantly improving the generalization ability of traditional supervised learning methods. This study introduces a systematic classification of meta-learning methods, dividing them into optimization, metric, and fusion paradigms, and explores how to extend these methods from few-shot classification to weakly supervised segmentation tasks. Furthermore, the study proposes a multi-task meta-learning framework that separates the support set and query set for training, explicitly optimizing the model's generalization performance in new domains and new tasks, providing a new solution for weakly supervised semantic segmentation (Oliveira H, Cesar R M, Gama P H T, et al. Domain generalization in medical image segmentation via meta-learners [C]. 2022 35th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI). IEEE, 2022, 1: 288-293.). However, different sparse labeling methods (such as contour labeling and region labeling) have different effects, and fine-tuning is required for each task. If the task distribution of the meta dataset is not extensive, the model may overfit.

[0005] In addition, the performance of the model trained by federated learning is largely influenced by the federated aggregation algorithm. Currently, there are many aggregation algorithms proposed. For example, in 2017, McMahan B et al. proposed the algorithm of federated learning, mainly used for efficient training of deep learning models on distributed data, and proposed a federated average aggregation algorithm (FedAvg), which takes the proportion of data volume of each participant as the weight when aggregating the model. This method performs well on non-IID data and can reduce the number of communications required to train deep networks on decentralized data by several orders of magnitude. However, only considering the data volume when aggregating the model may lead participants to use poor quality data to make their model have a larger weight proportion when aggregating (McMahan B, Moore E, Ramage D, et al. Communication-efficient learning of deep networks from decentralized data [C] / / Artificial intelligence and statistics. PMLR, 2017: 1273-1282.). In 2019, Li T et al. proposed two aggregation algorithms, q-FedAvg and q-FedSDG, to solve the fairness problem in federated learning by dynamically adjusting device weights and objective functions. In q-FedSDG, higher weights are given to high-loss devices to improve the uniformity of the model among devices, but this method may reduce the performance of some well-performing participant models, leading to a decline in overall model performance. In contrast, q-FedAvg reduces the global communication frequency by performing multiple rounds of local updates, making it more suitable for scenarios where device resources are large and communication resources are limited. It can also flexibly balance fairness and overall accuracy by adjusting the parameter q. However, multiple rounds of local updates may cause the model to be biased towards the data distribution of certain devices, and the step size and learning rate need to be carefully adjusted, which increases the complexity of model implementation (Li T, Sanjabi M, Beirami A, et al. Fair resource allocation in federated learning [J]. arXiv preprint arXiv:1905.10497, 2019.). In the same year, Xie C et al. proposed a new asynchronous federated optimization algorithm, FedAsync, to solve the problem that traditional federated learning synchronous methods (such as FedAvg) are easily affected by slow devices.FedAsync achieves fast global model updates with good convergence and tolerance to delays by using an asynchronous update strategy and adaptive mixing weights (Xie C, Koyejo S, Gupta I. Asynchronous federated optimization [J]. arXiv preprint arXiv:1903.03934, 2019.). However, the model used by devices in the asynchronous architecture can be outdated, and the direction of global model updates can deviate as the delay increases, affecting the convergence speed and final performance. In 2020, Wang H et al. proposed the federated matching average algorithm FedMA, designed for modern neural network architectures such as CNN and LSTM. FedMA constructs a shared global model in a hierarchical manner by matching and averaging parameters in neural networks layer by layer, such as the channels of convolutional layers, the hidden states of LSTMs, and the neurons of fully connected layers. After the central aggregation server receives the local models from each participant, it uses a layer-by-layer processing approach to find the best match for neurons or channels using the maximum matching algorithm, and then performs a weighted average based on the matching results to form the global model parameters, which are used as the initial training parameters for the next layer. For the last layer (such as the fully connected layer), FedMA uses a weighting method based on the class distribution of device data to aggregate parameters and generate the final global model. FedMA not only outperforms mainstream federated learning algorithms in deep CNN and LSTM training on real-world datasets, but also reduces the overall communication burden (Wang H, Yurochkin M, Sun Y, et al. Federated learning with matched averaging [EB / OL]. arXiv preprint arXiv:2002.06440, 2020.). However, FedMA has difficulty handling dynamic model updates (such as dynamically adding or removing network layers), and it requires high device requirements for local training by participants, as participants need to freeze some layer parameters and perform subsequent layer training, while also needing to perform complex matching operations. In 2021, So J et al. proposed an aggregation protocol called TurboAggregate to address the problem of excessive aggregation overhead in federated learning. Using a multi-group cyclic aggregation strategy, participants are randomly assigned to multiple groups, and each group of participants is responsible for passing the aggregated model from the previous group to the next group. This grouping and transmission structure can reduce the communication complexity of the system from O(n. 2) to O(nlogn). After each participant trains the model locally, it protects the local model using random noise and sends the encrypted model to the central aggregation server. During the aggregation process, the random noise cancels each other out, so that while ensuring privacy protection, the global model can still be obtained (So J, Guler B, Avestimehr AS. Turbo-aggregate: Breaking the quadratic aggregation barrier in secure federated learning [J]. IEEE Journal on Selected Areas in Information Theory, 2021, 2(1): 479-489.). This method is suitable for large-scale federated learning scenarios, not only reduces the communication time complexity, but also protects the security of each participant's model, and has a more extensive application prospect. However, the encoding and decoding process of this method is complex, and when the number of participants is particularly large, the random grouping strategy will also bring additional computational overhead.

[0006] In terms of federated domain generalization, traditional federated domain generalization methods need to be centralized, which cannot meet the privacy protection requirements of federated learning. Although single-source domain data enhancement, self-supervised signal regularization and other methods can improve the performance of local training, they are difficult to fully utilize cross-domain distribution information. Some methods may cause overfitting and implementation complexity.

[0007] In terms of federated aggregation algorithm, some algorithms do not consider data quality, which may lead to the problem of high proportion of low-quality data. Especially in the actual smart medical scene, the training data quality of each medical institution is uneven, and only considering the data volume will reduce the enthusiasm of providing high-quality medical institutions, and even some institutions may deliberately use low-quality data to reduce the performance of the model. And there are differences in medical equipment imaging, time and region and other factors of different medical institutions, only considering the data volume does not conform to the fairness of the federated aggregation mechanism. Some algorithms also have communication delay and efficiency problems.

[0008] Therefore, it is necessary to study a more reasonable weight distribution federated learning aggregation algorithm suitable for smart medical scenarios. SUMMARY

[0009] In order to solve the problems of performance decline of the trained model on the unseen domain and unreasonable weight distribution of the local model aggregation in the prior art, the present application provides a federated domain generalization method based on style transfer exploration. The present application generalizes the federated domain through style transfer exploration, and aggregates the global model by comprehensively considering two indexes of data volume and data quality, which can improve the performance of the model on the unseen domain, enhance the robustness of the model, and provide a more reasonable weight distribution method.

[0010] The technical solution adopted by the present application to solve the technical problems is as follows:

[0011] The federated domain generalization method based on style transfer exploration provided by the present application mainly includes the following steps:

[0012] Step S1: Each participant calculates the overall style information feature of the current data locally and uploads it to the central aggregation server; the central aggregation server stores the style information features of each participant and establishes a style sharing center;

[0013] Step S2: Selective style transfer;

[0014] Each participant selects a part of sample data in the original sample data set for style transfer through the Mini-Batch K-Means clustering algorithm; and uses the AdaIN model to transfer the style information features of other participants obtained from the central aggregation server to the part of sample data selected from the original sample data set;

[0015] Step S3: Using a decoder to convert the style transferred feature information into an image with the style of other participants;

[0016] Step S4: Style exploration is performed in the original sample data of each participant to obtain new style information features; the AdaIN model is used to selectively transfer the explored style information features to the original sample data;

[0017] Step S5: The central aggregation server randomly generates a key pair for each participant for encryption, distributes the global model parameters and public keys to each participant, and retains the private key for decryption;

[0018] Step S6: Distributed training;

[0019] Each participant uses the global model parameters distributed by the central aggregation server, uses the local original sample data and the images after style transfer and exploration to train the model, and generates a local model;

[0020] Step S7: Each participant encrypts the trained local model parameters using the public key and uploads them to the central aggregation server;

[0021] Step S8: The central aggregation server decrypts the local model parameters uploaded by each participant using the private key, verifies the model uploaded by each participant using the public uniform distribution dataset, calculates the model score of each participant, and aggregates the local model using the PDVFedMul aggregation algorithm to obtain the global model parameters of the next round;

[0022] Repeat steps S5 to S8 until the global model converges or reaches the specified number of iterations, and the current federated learning task ends.

[0023] Further, in step S1, the local dataset of each participant is S n , and the corresponding VGG encoded style information feature is The semantic content feature is According to formula (1) and formula (2), the average value μ n and the standard deviation δ n of the style feature channel of the nth participant are calculated.

[0024]

[0025] Wherein, μ represents the channel average of the style information feature of the ith image of the nth participant, σ n represents the standard deviation of the style information feature of the ith image of the nth participant,

[0026]

[0027] When the picture is an RGB image, C=3, and when the picture is a grayscale image, C=1; H is the height of the feature map, and W is the width of the feature map; the style information of each participant is represented by Φ n =[μ n , δ n ], each participant sends its style information Φ n to the central aggregation server, and the central aggregation server stores all the received style information of the participants to build a style sharing center.

[0028] Further, the Mini-Batch K-Means clustering algorithm quickly selects sample data of the clustering center of Batch1, and performs the following operations:

[0029] 1) Select sample data of Batch1 / 2 to retain the original style;

[0030] 2) The remaining Batch1 / 2 sample data is used for AdaIN model to perform style transfer.

[0031] Further, the calculation formula of the AdaIN model performing style transfer is:

[0032]

[0033] Wherein, δ(s') represents the standard deviation of the style information feature of the target image to be transferred, μ(s') represents the mean value of the style information feature of the target image to be transferred, μ(c) represents the mean value of the semantic content feature of the source image, δ(c) represents the standard deviation of the semantic content feature of the source image, c represents the semantic content feature of the source image, and s' represents the style information feature of the target image to be transferred.

[0034] Further, in step S3, the calculation formula for converting the feature information after style transfer into an image with the style of other participants using the decoder is:

[0035] I c←s ' = decoder(AdaIN(s', c)) #(4)

[0036] Wherein, c←s' represents that the style information feature s' of the target image to be transferred is migrated to the semantic content feature c of the source image, decoder represents the decoder, I c←s' represents the image finally generated through style transfer.

[0037] Further, in step S4, the Mini-Batch K-Means clustering algorithm is used to quickly select the sample data of the clustering center of Batch2, and the following operations are performed:

[0038] 1) Select Batch2 / 2 sample data to retain the original style;

[0039] 2) The remaining Batch2 / 2 sample data is used for style exploration.

[0040] Further, when performing style exploration, the calculation formula of the average value and the standard deviation of the style information feature channel of the new sample data is:

[0041]

[0042] Wherein, α is a hyperparameter, representing the exploration level, used to control the degree of deviation of the exploration style; the style information feature obtained by exploration is used to perform style transfer in the original sample data with Batch2 using the AdaIN model, and the decoder is used to convert the image after style exploration.

[0043] Further, the specific implementation process of step S6 is as follows:

[0044] Step S6.1: initialization configuration;

[0045] 1) Deploy a public open-source uniform distribution dataset on the central aggregation server;

[0046] 2) Determine the initial global model parameters;

[0047] 3) Determine the encryption algorithm of data transmission;

[0048] 4) Each participant needs to transmit the data volume participating in the model training to the central aggregation server;

[0049] Step S6.2: local training;

[0050] After the initialization configuration is completed, each participant starts local model training; in the process of each round of iteration, let the objective function to be optimized by each participant be f (φ) represents the average loss of each participant, φ ∈ R d represents the model parameters of d dimensions, ρ n represents the data volume owned by the nth participant, and the calculation formula of f (φ) is:

[0051]

[0052] wherein, represents the data sample of the nth participant the loss result obtained after prediction under the given parameter φ, represents the ith sample data of the nth participant, represents the corresponding labeled data;

[0053] For the nth participant, the average loss of the sample is:

[0054]

[0055] The gradient of the nth participant in the tth iteration is The learning rate is η, and the calculation formula of the new round of local update is as follows:

[0056]

[0057] wherein, represents the local model update parameter of the nth participant in the t+1th round, represents the local model parameter of the nth participant in the tth round, represents the gradient operator.

[0058] Further, the central aggregation server deploys a public open-source uniform distribution dataset, which is an existing open-source medical image dataset, to verify the performance of the local model uploaded by each participant.

[0059] Further, the specific implementation process of step S8 is as follows:

[0060] Suppose there are N participants, and the total amount of data participating in training is Denoted as ρ; then the data volume proportion of the nth participant is The calculation formula of is as follows:

[0061]

[0062] In the tth round, the nth participant uploads the trained local model parameters to the central aggregation server, and the central aggregation server verifies the public uniform distribution dataset stored by it using the uploaded local model, and the score is denoted as The total sum of scores of all participants in the tth round is Denoted as S t ; then the data quality proportion of the nth participant in the tth round is The calculation formula of is as follows:

[0063]

[0064] The calculation formula of the model parameter aggregation weight of the nth participant is as follows:

[0065]

[0066] Wherein, β represents a hyperparameter, used to adjust the size of the influence factor of data volume and data quality;

[0067] The global model parameters obtained by iterative updating are as follows:

[0068]

[0069] Wherein, φ t+1 represents the global model parameters obtained by tth round iterative updating;

[0070] The total loss function is as follows:

[0071]

[0072] The beneficial effects of the present application are as follows:

[0073] ​1. In federated learning, the data distribution of each participant is limited, and the trained model is prone to performance degradation when used on data outside the participant institution. To this end, the present application proposes a federated domain generalization method based on style transfer exploration, which can share the style information features of each participant, and through converting the local original data into images with the style of other participants for data augmentation, the data distribution of each participant is greatly enriched, and the generalization ability of the model is improved.

[0074] 2. Most of the current aggregation algorithms do not consider data quality when aggregating. In actual training scenarios, the training data quality of each participant is uneven, and if the data quality is not considered, the enthusiasm of participants who provide high-quality data will be reduced, and even some institutions may deliberately use low-quality data to reduce the performance of the model. To this end, the present application considers both data volume and data quality, and proposes a multi-index federated aggregation algorithm based on public data verification, which provides a more reasonable weight distribution scheme for model aggregation, thereby improving the performance of the model. BRIEF DESCRIPTION OF DRAWINGS

[0075] Figure 1 A flowchart of a federated domain generalization method based on style transfer exploration in the present application.

[0076] Figure 2 A flowchart of a multi-index federated aggregation algorithm based on public uniform distribution data verification in the present application.

[0077] Figure 3 A flowchart of an adaptive federated learning method based on style transfer exploration provided by the present application. DETAILED DESCRIPTION

[0078] The present application will be further described in detail below in conjunction with the accompanying drawings.

[0079] The adaptive federated learning method based on style transfer exploration of the present application is mainly realized through the cooperation of the federated domain generalization method based on style transfer exploration and the multi-index federated aggregation algorithm based on public uniform distribution data verification.

[0080] Currently, most of the models trained by federated learning perform well on the participants' data, but if the model is applied to institutions outside the participants, the model performance will often decline. Currently, there are relatively few studies on federated domain generalization, and many methods require centralized processing of data, which does not meet the premise of federated learning. To this end, the present application proposes a federated domain generalization method based on style transfer exploration (Style Transfer Exploration Federated Domain Generalization, STE FDG), which uses a pre-trained VGG-19 as an encoder to convert the content and style of an image to a feature space, and then normalizes the content image features to match the mean and variance of the style image features. Then use the decoder to map the processed features back to the image space to get the image after style transfer; STE FDG will establish a style sharing center on the central aggregation server to store the style information of each participant. Thus, each participant can use the content information of the local sample data and the style information of other participants obtained from the central aggregation server to perform style transfer, generate enhanced data with the style of other participants through the decoder, and thus enrich the data distribution of each participant.

[0081] In the present application, the AdaIN model (Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In ICCV, 2017. 3, 4, 6, 13) is applied to STE FDG to realize federated domain generalization. As shown in Figure 1 Before starting local training, each participant will calculate the style information of its local data and pass it to the central aggregation server. The central aggregation server will collect the style information of all participants to establish a style sharing center, and each participant can get the style information of other participants from the central aggregation server to enrich the local data domain. In the process of updating the local data domain, each participant selectively converts the style of a small batch of original sample data to a new style with the style of other participants through the adaptive instance normalization model (AdaIN) to improve the diversity of the domain. At the same time, in order to further extrapolate new unseen domains, a portion of the original sample data of each participant can be selected for style exploration.

[0082] At present, many aggregation algorithms have been proposed, Google proposes a federated average algorithm, increases the weight of the participants whose current model performance is poor from the consideration of federated fairness, and some ways based on meta-learning. However, these rarely consider the problem of data quality of each participant, which is not conducive to the improvement of model performance. In view of this, the present application proposes a multi-index federated aggregation algorithm based on public data verification (PDVFedMul), which considers the quality of data as an index on the basis of the federated average algorithm (FedAvg), so that the local model parameters uploaded by each participant have more reasonable weights. By comprehensively considering the data volume and quality, reasonable weights can be adaptively and dynamically allocated, thereby improving the global model performance.

[0083] The present application constructs a new type of federated learning collaboration model according to the federated domain generalization method based on style transfer exploration and the multi-index federated aggregation algorithm based on public data verification. Through the STEFDG federated domain generalization technology, the local data domain distribution is enriched, and style exploration is also performed, further extrapolating the model to unseen domains, and significantly improving the performance of the model in the unseen domain. When aggregating the local model, the present application uses the PDFFedMul aggregation algorithm. Compared with the federated average algorithm (FedAvg), the PDFFedMul aggregation algorithm used by the present application considers the quality of the local model as an index, and proposes a more reasonable model aggregation weight distribution scheme.

[0084] Reference Figures 1 to 3 It is explained that the adaptive federated learning method based on style transfer exploration of the present application specifically includes the following steps:

[0085] Step S1: Establish a style sharing center;

[0086] Each participant calculates the overall style information feature of the current data locally, and then uploads it to the central aggregation server. After receiving the style information features of each participant, the central aggregation server stores them, and then establishes a style sharing center. Through the establishment of the style sharing center in the central aggregation server, each participant can obtain the style information features of other participants, thereby greatly enriching the local data distribution; and it is impossible to restore the image only through the style information features, so there is no risk of privacy leakage.

[0087] Specifically, the local data set of each participant is S n The corresponding VGG encoded style information feature is The semantic content feature is The average value μ of the nth participant's style feature channel is calculated according to formula (1) and formula (2) n And the standard deviation δ n :

[0088]

[0089] Wherein, represents the channel average of the style information feature of the ith image of the nth participant, represents the standard deviation of the style information feature of the ith image of the nth participant, represents the VGG encoded style information feature of the ith image of the nth participant, and n represents the amount of data owned by the nth participant;

[0090]

[0091] When the picture is an RGB image, C = 3, and when the picture is a grayscale image, C = 1; H is the height of the feature map, and W is the width of the feature map.

[0092] The style information of each participant can be represented by n =[μ n ,δ n ], each participant sends its style information n to the central aggregation server, and then the central aggregation server stores all the received style information of the participants to establish a style sharing center.

[0093] Step S2: Selective style transfer;

[0094] Each participant selects a part of the sample data in the original sample data set for style transfer through the Mini-Batch K-Means clustering algorithm. The AdaIN model is used to transfer the style information features of other participants obtained from the central aggregation server to the selected part of the original sample data;

[0095] Specifically, each participant obtains the style information n of other participants from the central aggregation server to perform style transfer. In order to obtain new data with the style of other participants without losing the performance of the original data, the Mini-Batch K-Means clustering algorithm is used to quickly select sample data with a clustering center of Batch1, and the following operations are performed:

[0096] 1) Select Batch1 / 2 of sample data to retain the original style;

[0097] 2) The remaining Batch1 / 2 of sample data is subjected to style transfer through the AdaIN model;

[0098]

[0099] wherein, δ(s') represents the standard deviation of the style information feature of the target image to be migrated, μ(s') represents the mean value of the style information feature of the target image to be migrated, μ(c) represents the mean value of the semantic content feature of the source image, δ(c) represents the standard deviation of the semantic content feature of the source image, c represents the semantic content feature of the source image, and s' represents the style information feature of the target image to be migrated.

[0100] Step S3: decoding to obtain the image after style migration;

[0101] Using the decoder, the feature information after style migration by the AdaIN model can be converted into an image with the style of other participants, thereby greatly enriching the local data domain distribution; the specific calculation formula is as follows:

[0102] I c←s = decoder(AdaIN(s', c))#(4)

[0103] wherein, c←s' represents migrating the style information feature s' of the target image to be migrated to the semantic content feature c of the source image, decoder represents the decoder, Ic ←S' represents the image finally generated by style migration.

[0104] Step S4: style exploration;

[0105] In order to further improve the performance of the model on the unseen domain, style exploration is performed on the original sample data of each participant to obtain new style information features, and the AdaIN model is used to selectively migrate the explored style information features to the original sample data, thereby improving the generalization ability of the model on the unseen domain;

[0106] Specifically, in order to further extrapolate the model to new unseen domains, the Mini-Batch K-Means clustering algorithm can also be used on the original sample data of each participant to quickly select sample data with a clustering center of Batch2, and the following operations are performed:

[0107] 1) Selecting Batch2 / 2 sample data to retain the original style;

[0108] 2) The remaining Batch2 / 2 sample data is subjected to style exploration; the specific calculation formula of the new sample data style information feature channel average value and standard deviation is as follows:

[0109]

[0110] wherein a is a hyper parameter, representing the exploration level, used to control the offset degree of exploration style. The exploration obtained style information feature is used AdaIN model, in the original sample data of batch Batch2, style transfer is carried out, and the decoder is used to convert into the image after style exploration.

[0111] After the above steps S1-S4, the federal domain generalization process is completed. The local of each participant not only increases the image with the style of other participants, but also increases the image after style exploration based on the original sample data, thereby greatly enriching the sample data of each participant's local. In the subsequent local training process, each participant uses the original sample data and the data after style transfer and exploration for model training.

[0112] Step S5: the central aggregation server issues global model parameters;

[0113] The central aggregation server randomly generates a key pair for each participant to encrypt, issues the global model parameters and public key to each participant, and retains the private key for decryption;

[0114] Step S6: distributed training;

[0115] Each participant uses the global model parameters issued by the central aggregation server, uses the local original sample data and the image after style transfer and exploration to train the model, and generates a local model;

[0116] The specific implementation process is as follows:

[0117] Step S6.1: initialization configuration;

[0118] Before formal distributed training, some initialization configurations need to be done as follows:

[0119] 1) Deploy a public open source uniform distribution dataset in the central aggregation server;

[0120] In the central aggregation server, a public uniform distribution dataset needs to be stored. This dataset is an open source medical image dataset that already exists (does not involve the local data of each participant, so there is no privacy leakage). This dataset has a wide source and is uniformly distributed, and is used to verify the performance of the local model uploaded by each participant;

[0121] 2) Determine the initial global model parameters; the central aggregation server needs to issue the initial global model parameters and some basic settings to each participant;

[0122] 3) Determine the encryption algorithm for data transmission;

[0123] 4) Each participant needs to pass the data volume of this participation in model training to the central aggregation server, that is, the total number of data of each participant participating in training, which is used for subsequent PDFFedMul aggregation algorithm to calculate data volume index.

[0124] Step S6.2: Local training;

[0125] After the initialization configuration is completed, each participant can start local model training.

[0126] In the process of each iteration, let the objective function to be optimized by each participant be Where f (φ) represents the average loss of each participant, φ ∈ R d represents the model parameters of d dimensions, ρ n represents the data volume owned by the nth participant, and the calculation formula of f (φ) is as follows:

[0127]

[0128] Where, represents the data sample of the nth participant The loss result obtained after prediction under the given parameter φ, represents the ith sample data of the nth participant, represents corresponding labeled data;

[0129] For the nth participant, the average loss of the sample is:

[0130]

[0131] The gradient of the nth participant in the tth iteration is The learning rate is η, and the calculation formula of the new round of local update is as follows:

[0132]

[0133] Where, represents the local model update parameter of the nth participant in the t+1th iteration, represents the local model parameter of the nth participant in the tth iteration, represents the gradient operator.

[0134] Step S7: Upload local model parameters;

[0135] Each participant encrypts the trained local model parameters using the public key issued, and then uploads them to the central aggregation server.

[0136] Step S8: Aggregate local models;

[0137] The central aggregation server decrypts the local model parameters uploaded by each participant by using a private key, verifies the model uploaded by each participant by using a public uniform distribution dataset, calculates the model score of each participant, and then aggregates the local model by using a PDVFedMul aggregation algorithm, so as to obtain the global model parameters of the next round;

[0138] The specific process is as follows:

[0139] Step S8.1: data volume proportion calculation;

[0140] In the case that the data volume of each participant participating in training is determined, the data volume proportion remains unchanged in each round of aggregation, denoted as E1. Assuming that there are N participants in total, the total amount of data participating in data training is denoted as ρ, then the data volume proportion of the nth participant is The calculation formula of is as follows:

[0141]

[0142] Step S8.2: data quality proportion calculation;

[0143] In the tth round, after the nth participant uploads the trained local model parameters to the central aggregation server, the central aggregation server verifies the public uniform distribution dataset stored by using the uploaded local model, and the score is denoted as then denotes the total sum of scores of all participants in the tth round, denoted as S t ; then the data quality proportion of the nth participant in the tth round is The calculation formula of is as follows:

[0144]

[0145] Step S8.3: model parameter aggregation weight calculation;

[0146] In the tth round, the model parameter aggregation weight of the nth participant is The calculation formula of is as follows:

[0147]

[0148] wherein β represents a hyperparameter, used for adjusting the size of the influence factor of data volume and data quality, the greater β is, the greater the influence of data volume proportion is, and vice versa, the greater the influence of data quality proportion is.

[0149] Step S8.4: global model parameter calculation;

[0150] The global model parameters obtained by the tth round of iterative update are:

[0151]

[0152] wherein, φ t+1 denotes the global model parameters obtained by the tth round of iterative update.

[0153] Step S8.5: total loss function calculation;

[0154] The total loss function of the tth round is:

[0155]

[0156] Repeat the above steps S5 to S8 until the global model converges or reaches the specified number of iterations, and the current federated learning task ends.

[0157] The above only describes the preferred embodiments of the present application, and it should be noted that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, and these improvements and refinements should also be considered as the protection scope of the present application.

Claims

1. A federated domain generalization method based on style transfer exploration, characterized in that, Includes the following steps: Step S1: Each participant calculates the overall style information features of the current data locally and uploads them to the central aggregation server; the central aggregation server stores the style information features of each participant and establishes a style sharing center. Step S2: Selective style transfer; Each participant selects a portion of sample data from the original sample dataset for style transfer using the Mini-Batch K-Means clustering algorithm; the AdaIN model is used to transfer style information features from other participants obtained from the central aggregation server to a portion of sample data selected from the original sample dataset. Step S3: Use a decoder to convert the style-transferred feature information into an image with the styles of other participants; Step S4: Explore styles in the original sample data of each participant to obtain new style information features; use the AdaIN model to selectively transfer the explored style information features back to the original sample data; Step S5: The central aggregation server randomly generates key pairs for encryption for each participant, distributes the global model parameters and public key to each participant, and retains the private key for decryption; Step S6: Distributed training; Each participant uses the global model parameters distributed by the central aggregation server, along with local raw sample data and images after style transfer and exploration, to train the model and generate a local model. Step S7: Each participant uses its public key to encrypt the trained local model parameters and uploads them to the central aggregation server. Step S8: The central aggregation server uses the private key to decrypt the local model parameters uploaded by each participant, and uses the public uniformly distributed dataset to verify the models uploaded by each participant, calculates the model scores of each participant, and then uses the PDVFedMul aggregation algorithm to aggregate the local models to obtain a new round of global model parameters. Repeat steps S5 to S8 until the global model converges or the specified number of iterations is reached, at which point this federated learning task ends.

2. The federated domain generalization method based on style transfer exploration according to claim 1, characterized in that, In step S1, the local data set of each participant is S n The corresponding VGG encoded style information feature is The semantic content feature is The average value μ of the style feature channel of the nth participant is calculated according to formula (1) and formula (2) n And the standard deviation δ n : wherein, denotes the channel mean of the style information feature of the i-th image of the n-th participant, denotes the standard deviation of the style information feature of the i-th image of the n-th participant, denotes the VGG-encoded style information feature of the i-th image of the n-th participant, n denotes the amount of data owned by the n-th participant; C=3 when the picture is an RGB image, and C=1 when the picture is a grayscale image; H is the height of the feature map, and W is the width of the feature map; the style information of each participant is denoted by n =[μ n , δ n ] , each participant sends its style information Φ n to the central aggregation server, and the central aggregation server stores all the received style information of the participants to construct a style sharing center.

3. The federated domain generalization method based on style transfer exploration of claim 1, wherein, The Mini-Batch K-Means clustering algorithm quickly selects sample data with cluster centers in batch Batch 1 and performs the following operations: 1) Select sample data from Batch 1 / 2 to preserve the original style; 2) The remaining sample data from Batch1I2 will be used for style transfer in the AdaIN model.

4. The federated domain generalization method based on style transfer exploration according to claim 3, characterized in that, The formula for calculating style transfer using the AdaIN model is as follows: Where δ(s′) represents the standard deviation of the style information features of the target image to be transferred, μ(s′) represents the mean of the style information features of the target image to be transferred, μ(c) represents the mean of the semantic content features of the source image, δ(c) represents the standard deviation of the semantic content features of the source image, c represents the semantic content features of the source image, and s′ represents the style information features of the target image to be transferred.

5. The federated domain generalization method based on style transfer exploration according to claim 1, characterized in that, In step S3, the formula for converting the style-transferred feature information into an image with the styles of other participants using the decoder is as follows: I c←s′ = decoder(AdaIN(s', c))#(4) wherein c←s' indicates that the style information feature s' of the target image to be migrated is migrated to the semantic content feature c of the source image, decoder indicates a decoder, I c←s′ denotes the image finally generated by style migration.

6. The federated domain generalization method based on style transfer exploration according to claim 1, characterized in that, In step S4, the Mini-Batch K-Means clustering algorithm is used to quickly select sample data with cluster centers in batch Batch 2, and the following operations are performed: 1) Select Batch 2 / 2 sample data to preserve the original style; 2) Use the remaining Batch 2 / 2 sample data for style exploration.

7. A federated domain generalization method based on style transfer exploration according to claim 6, characterized in that, When conducting style exploration, the average value of style information feature channels in new sample data. and standard deviation The calculation formula is: Here, α is a hyperparameter representing the exploration level, used to control the degree of style shift during exploration; the style information features obtained from exploration are used in the AdaIN model to perform style transfer on the original sample data in batch 2, and then converted into images after style exploration using a decoder.

8. A federated domain generalization method based on style transfer exploration according to claim 1, characterized in that, The specific implementation process of step S6 is as follows: Step S6.1: Initialize configuration; 1) Deploy a publicly available, uniformly distributed dataset on a central aggregation server; 2) Determine the initial global model parameters; 3) Determine the encryption algorithm for data transmission; 4) Each participating party needs to transmit the amount of data used in this model training to the central aggregation server; Step S6.2: Local training; After initial configuration is complete, each participant begins local model training; in each iteration, let the objective function to be optimized by each participant be... f(φ) represents the average loss of each participant, φ∈R d ρ represents the d-dimensional model parameters. n Let f(φ) represent the amount of data possessed by the nth participant. The formula for calculating f(φ) is: in, This represents the data sample of the nth participant. The loss result obtained after prediction given the parameter φ. This represents the i-th sample data of the n-th participant. express Corresponding labeled data; For the nth participant, the average loss of its sample is: The gradient of the nth participant in the t-th iteration is If the learning rate is η, then the formula for calculating the new round of local updates is as follows: in, This represents the local model update parameters of the nth participant in round t+1. This represents the local model parameters of the nth participant in round t. This represents the gradient operator.

9. A federated domain generalization method based on style transfer exploration according to claim 8, characterized in that, A publicly available, uniformly distributed dataset is deployed on a central aggregation server. This dataset is an existing open-source medical image dataset used to validate the performance of local models uploaded by each participant.

10. A federated domain generalization method based on style transfer exploration according to claim 1, characterized in that, The specific implementation process of step S8 is as follows: Suppose there are N participants, and the total amount of data involved in training is... Let ρ be the percentage of the data volume of the nth participant. The calculation formula is: In round t, the nth participant uploads its trained local model parameters to the central aggregation server. The central aggregation server then uses the uploaded local model to validate its stored public uniformly distributed dataset, denoted as . The total score of all participants in round t is: Let S be the value of S. t Then the data quality percentage of the nth participant in round t. The calculation formula is: The aggregate weights of the model parameters of the nth participant The calculation formula is: Where β represents a hyperparameter, used to adjust the magnitude of the influence factors on data volume and data quality; the global model parameters obtained through iterative updates are: where φ t+1 denotes the global model parameters obtained in the tth iteration update; The overall loss function is:

Images