Self-adaptive federal learning method based on style migration exploration

Abstract

A self-adaptive federated learning method based on style migration exploration belongs to the field of federated learning, and comprises the following steps: establishing a style sharing center; selective style migration; using a decoder to convert the feature information after style migration into images with other participant styles; performing style exploration in the original sample data of each participant to obtain new style information features; selectively migrating the explored style information characteristics to original sample data by using an AdaIN model; performing distributed training to generate a local model; the model uploaded by each participant is verified by using a public uniform distribution data set, the score of each participant model is calculated, and then the local model is aggregated by using a PDVFedMul aggregation algorithm, so that a new round of global model parameters are obtained. According to the method, the generalization ability of the model is improved, the robustness of the model is enhanced, a more reasonable weight distribution scheme is provided for model aggregation, and the model performance is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of federated learning, and specifically relates to an adaptive federated learning method based on style transfer exploration. Background Art

[0002] Federated learning is an emerging artificial intelligence technology that provides inspiration for solving the widespread problem of "data islands". Its operating mechanism can securely connect all participants without having to share or migrate their local data. Model training can be performed locally, and then the trained model is sent to the central aggregation server, which then aggregates the model into a global model. The parameter transmission during the training process is also replaced by encrypted transmission instead of the original remote data transmission, thereby ensuring data security.

[0003] However, most of the models trained by federated learning currently perform well when tested on the data of the participants, but if the model is applied to institutions other than the participants, the model performance will often decline. For example, in medical scenarios, patient data from various medical institutions are needed for federated learning. The medical imaging equipment used by different medical institutions is different, and the same patient goes to two different hospitals for medical imaging examinations of the same part, and the imaging results obtained may also be different. The trained model will not only be used in the medical institutions participating in the training, but also in many other medical institutions. Therefore, in order to improve the robustness of the model, it is necessary to consider the problem of generalization of the federated domain.

[0004] In terms of federated domain generalization, in 2017, Motiian S et al. proposed a unified framework for supervised domain adaptation and generalization of deep models, mapping source domain samples to a shared embedding space so that samples with the same category label but different domains are semantically aligned in the space. The semantic alignment loss is introduced to minimize the distance between samples from the same category but different domains, ensuring that these samples are close in the embedding space. In order to avoid overlapping of samples of different categories in the embedding space, the category separation loss is further introduced to maximize the distribution distance of samples of different categories. The final objective function combines the classification loss, semantic alignment loss, and category separation loss to form a contrast semantic alignment loss. This joint loss ensures that both category alignment and category separation are achieved during the training process, while improving the classification performance (Motiian S, Piccirilli M, Adjeroh DA, et al. Unified deep supervised domain adaptation and generalization [C]. Proceedings of the IEEE International Conference on Computer Vision, 2017: 5715-5725.). In 2019, Li Y et al. studied how to train a feature representation model to effectively identify new data and new categories in a new target domain when the source domain and the target domain have different label spaces. A meta-learning-based scenario training paradigm is proposed to stimulate domain transfer by creating meta-training and meta-test splits at each iteration. Meta-learning is used to learn auxiliary losses to guide the feature extractor to learn more general features (Li Y, Yang Y, Zhou W, et al. Feature-critic networks for heterogeneous domain generalization [C]. International Conference on Machine Learning, 2019: 3915-3924.). However, the above methods usually require multi-domain data to be centralized for learning, which does not meet the privacy protection requirements in the federated learning setting.There are other methods that do not require data collection, such as data augmentation (Volpi R, Namkoong H, Sener O, et al. Generalizing to unseen domains via adversarial data augmentation [J]. Advances in Neural Information Processing Systems, 2018, 31.109845), self-supervisory signals (Wang S, Yu L, Li C, et al. Learning from extrinsic and intrinsic supervisions for domain generalization [C]. European Conference on Computer Vision, Cham: Springer International Publishing, 2020: 159-176.), and deep neural network operations (Matsuura T, Harada T. Domain generalization using a mixture of multiple latent domains [C]. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(07): 11749-11756.) and other methods for domain generalization. These methods can effectively serve as regularization for local training of single source domain data, but it is difficult to utilize rich cross-domain data distribution. In 2021, Quande Liu et al. proposed a contextual learning method based on continuous frequency space for federated domain generalization to address the problem of inter-domain differences in data distribution in medical image segmentation tasks (Quande Liu, Cheng Chen, Jing Qin, Qi Dou, and Pheng-Ann Heng. Feddg: Federated domain generalization on medical image segmentation via episodic learning in continuous frequency space. In CVPR, 2021. 1, 3, 6, 11). This method decomposes the image information of local data into an amplitude spectrum representing low-level distribution information and a phase spectrum representing high-level semantics. The phase spectrum is retained locally to protect privacy. At the same time, a shared distribution library is established on the central aggregation server to store the amplitude spectrum of the local image signal of each participant to achieve cross-client distribution information transmission.The local amplitude spectrum is continuously interpolated with the amplitude spectrum of other clients in the low-frequency region to generate an image containing multi-source distribution. The original image is regarded as a meta-training set, the interpolated image is used as a meta-test set, and a scenario learning framework is designed to simulate distribution transfer. This method uses continuous frequency space interpolation and boundary-guided optimization to improve the performance of the model on unseen domains, and can also be extended to other tasks with strong distribution heterogeneity. However, the author did not directly deal with the basic problems caused by lack of data and style. In 2022, Oliveira H et al. proposed a generalization technology for medical image segmentation based on meta-learners, aiming to address the challenges caused by the scarcity of labeled data and inter-domain differences in medical image segmentation tasks. Meta-learning simulates the training process of few-shot tasks, enabling the model to quickly adapt to new tasks with limited labeled data, thereby reducing the dependence on large-scale labeled data and significantly improving the generalization ability of traditional supervised learning methods. This study introduces a systematic classification of meta-learning methods, divides meta-learning techniques into three paradigms: optimization, metric, and fusion, and deeply explores how to extend these methods from few-shot classification to weakly supervised segmentation tasks. In addition, the study proposed a multi-task meta-learning framework, which explicitly optimizes the generalization performance of the model in new domains and new tasks by separate training of the support set and the query set, providing a new solution for weakly supervised semantic segmentation (Oliveira H, CesarR M, Gama PHT, et al. Domain generalization in medical image segmentation via meta-learners [C]. 202235th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI). IEEE, 2022, 1: 288-293.). However, the effects of different sparse annotation methods (such as contour annotation, region annotation, etc.) vary, and each task needs to be fine-tuned. At the same time, if the tasks of the metadata dataset are not widely distributed, the model may overfit.

[0005] In addition, when using federated learning technology, the effect of the trained model will be greatly affected by the federated aggregation algorithm. At present, many aggregation algorithms have been proposed. For example, in 2017, McMahan B et al. proposed the algorithm idea of ​​federated learning, which is mainly used to efficiently train deep learning models on distributed data. They proposed a federated average aggregation algorithm (FedAvg), which takes the data volume proportion of each participant as the weight when aggregating the model. This method performs well under non-IID data and can reduce the number of communications required to train deep networks on decentralized data by several orders of magnitude. However, considering only the data volume when aggregating the model may cause the participants to use poor quality data in order to make their models have a larger weight proportion when aggregating (McMahan B, Moore E, Ramage D, et al. Communication-efficient learning of deep networks from decentralized data [C] / / Artificial intelligence and statistics. PMLR, 2017: 1273-1282.). In 2019, Li T et al. proposed two aggregation algorithms, q-FedAvg and q-FedSDG, to solve the fairness problem in federated learning by dynamically adjusting device weights and objective functions. Among them, q-FedSDG improves the uniformity of the model among devices by giving higher weights to high-loss devices. However, this method may reduce the performance of some participants with better performance, resulting in a decline in the performance of the entire model. In contrast, q-FedAvg reduces the frequency of global communication by performing multiple rounds of updates locally. It is more suitable for scenarios with large device resources and limited communication resources, and can flexibly balance fairness and overall accuracy by adjusting the parameter q. However, multiple rounds of local updates may cause the model to be biased towards the data distribution of certain devices, and hyperparameters such as step size and learning rate need to be carefully adjusted, which increases the complexity of model implementation (Li T, Sanjabi M, Beirami A, et al. Fair resource allocation in federated learning[J]. arXiv preprint arXiv:1905.10497, 2019.). In the same year, Xie C et al. proposed a new asynchronous federated optimization algorithm FedAsync to solve the problem that synchronous methods (such as FedAvg) in traditional federated learning are easily affected by slow devices.FedAsync achieves fast updates of the global model through asynchronous update strategies and adaptive hybrid weights, while having good convergence and tolerance to delays (Xie C, Koyejo S, Gupta I. Asynchronous federated optimization [J]. arXiv preprint arXiv: 1903.03934, 2019.). However, the asynchronous architecture allows devices to use models that may be relatively old. As the delay increases, the direction of global model updates may deviate, affecting the convergence speed and final performance. In 2020, Wang H et al. proposed the Federated Matching Average Algorithm FedMA, which is designed for modern neural network architectures (such as CNN and LSTM). FedMA builds a shared global model in a hierarchical manner by matching and averaging the parameters in the neural network layer by layer (such as channels in the convolutional layer, hidden states in the LSTM, and neurons in the fully connected layer). After the central aggregation server receives the local models of each participant, it uses a layer-by-layer processing method to find the best match of neurons or channels using the maximum matching algorithm, and performs weighted averaging based on the matching results to form global model parameters as the initial training parameters for the next layer. For the last layer (such as the fully connected layer), FedMA aggregates the parameters using a weighted method based on the distribution of device data categories to generate the final global model. FedMA not only outperforms mainstream federated learning algorithms in deep CNN and LSTM training on real-world datasets, but also reduces the overall communication burden (Wang H, Yurochkin M, Sun Y, et al. Federated learning with matched averaging [EB / OL]. arXiv preprint arXiv: 2002.06440, 2020.). However, FedMA has difficulty handling dynamic model updates (such as dynamically adding or deleting network layers), and it has high requirements for the local training devices of each participant, because the participants need to freeze some layer parameters and perform subsequent layer training, while also performing complex matching operations. In 2021, So J et al. proposed an aggregation protocol called TurboAggregate to solve the problem of excessive aggregation overhead in federated learning. A multi-group cyclic aggregation strategy is adopted. Each participant is randomly assigned to multiple groups. Each group of participants is responsible for passing the aggregation model of the previous group to the next group. This grouping and transmission structure can reduce the communication complexity of the system from O(n) of traditional federated learning. 2) drops to O(nlogn). After each participant trains the model locally, it uses random noise to protect the local model and sends the encrypted model to the central aggregation server. During the aggregation process of the central aggregation server, the random noise cancels each other out, so that the global model can be obtained while ensuring privacy protection (So J, Güler B, Avestimehr A S. Turbo-aggregate: Breaking the quadratic aggregation barrier insecure federated learning [J]. IEEE Journal on Selected Areas in Information Theory, 2021, 2 (1): 479-489.). This method is suitable for large-scale federated learning scenarios. It not only reduces the complexity of communication time, but also protects the security of the models of each participant, and has a wider application prospect. However, the encoding and decoding process of this method is complicated, and when the number of participants is particularly large, the random grouping strategy will also bring additional computational overhead.

[0006] In terms of federated domain generalization, traditional federated domain generalization methods require data centralization and cannot meet the privacy protection requirements of federated learning. Although regularization methods such as single-source domain data enhancement and self-supervision signals can improve local training performance, it is difficult to fully utilize cross-domain distribution information. Some methods may also lead to overfitting and implementation complexity.

[0007] In terms of federated aggregation algorithms, some algorithms do not consider data quality, which may lead to the problem of too high a proportion of low-quality data. Especially in actual smart medical scenarios, the quality of training data of various medical institutions is uneven. Considering only the data volume will reduce the enthusiasm of providing high-quality medical institutions, and there may even be a phenomenon that some institutions deliberately use low-quality data to reduce model performance. In addition, there will be differences in imaging of medical equipment in different medical institutions, as well as the influence of factors such as time and region. Considering only the data volume does not meet the fairness of the federated aggregation mechanism. Some algorithms also have communication delays and efficiency issues.

[0008] In view of this, it is necessary to study a federated learning aggregation algorithm with more reasonable weight distribution suitable for smart medical scenarios. Summary of the invention

[0009] In order to solve the problem of the performance degradation of the trained model in the unseen domain and the unreasonable distribution of the aggregate weights of the local model in the prior art, the present invention provides a federal domain generalization method based on style transfer exploration. The present invention generalizes the federal domain through style transfer exploration, and aggregates and generates a global model by comprehensively considering the two indicators of data volume and data quality. On the one hand, it can improve the performance of the model in the unseen domain and enhance the robustness of the model. On the other hand, it provides a more reasonable weight distribution method.

[0010] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0011] The present invention provides a federal domain generalization method based on style transfer exploration, which mainly includes the following steps:

[0012] Step S1: Each participant calculates the overall style information features of the current data locally and uploads them to the central aggregation server; the central aggregation server stores the style information features of each participant and establishes a style sharing center;

[0013] Step S2: Selective style transfer;

[0014] Each participant selects a part of sample data from the original sample data set through the Mini-Batch K-Means clustering algorithm for style transfer; the style information features of other participants obtained from the central aggregation server are transferred to a part of sample data selected from the original sample data set using the AdaIN model;

[0015] Step S3: Use a decoder to convert the feature information after style transfer into an image with the style of other participants;

[0016] Step S4: Perform style exploration in the original sample data of each participant to obtain new style information features; use the AdaIN model to selectively migrate the explored style information features to the original sample data;

[0017] Step S5: The central aggregation server randomly generates a key pair for each participant to use for encryption, sends the global model parameters and public key to each participant, and retains the private key for decryption;

[0018] Step S6: distributed training;

[0019] Each participant uses the global model parameters issued by the central aggregation server, local original sample data and images after style transfer and exploration to train the model and generate a local model;

[0020] Step S7: Each participant uses the public key to encrypt the trained local model parameters and upload them to the central aggregation server;

[0021] Step S8: The central aggregation server uses the private key to decrypt the local model parameters uploaded by each participant, and uses the public uniformly distributed data set to verify the models uploaded by each participant, calculates the model scores of each participant, and then uses the PDVFedMul aggregation algorithm to aggregate the local models to obtain a new round of global model parameters;

[0022] Repeat steps S5 to S8 until the global model converges or reaches the specified number of iterations, and the federated learning task ends.

[0023] Furthermore, in step S1, the local data set of each participant is S n , and its corresponding VGG encoding style information feature is The semantic content features are According to formula (1) and formula (2), the average value μ of the style feature channel of the nth participant is calculated: n and standard deviation δ n :

[0024]

[0025] in, represents the channel mean of the style information features of the i-th image of the n-th participant, represents the standard deviation of the style information features of the i-th image of the n-th participant, represents the VGG coding style information feature of the i-th image of the n-th participant, ρ n Indicates the amount of data owned by the nth participant;

[0026]

[0027] When the image is an RGB image, C = 3, when the image is a grayscale image, C = 1; H is the height of the feature map, W is the width of the feature map; the style information of each participant is expressed by Φ n =[μ n ,δ n ] indicates that each participant sends its style information to the central aggregation server. n ,The central aggregation server stores the style information received from all ,participants and builds a style sharing center.

[0028] Furthermore, the Mini-Batch K-Means clustering algorithm quickly selects batches as Batch 1 To find the sample data of cluster centers, do the following:

[0029] 1) Select Batch 1 / 2 sample data retains the original style;

[0030] 2) The remaining batch 1 / 2 sample data is used for style transfer of AdaIN model.

[0031] Furthermore, the calculation formula for style transfer of the AdaIN model is:

[0032]

[0033] Among them, δ(s') represents the standard deviation of the style information features of the target image to be migrated, μ(s') represents the mean of the style information features of the target image to be migrated, μ(c) represents the mean of the semantic content features of the source image, δ(c) represents the standard deviation of the semantic content features of the source image, c represents the semantic content features of the source image, and s' represents the style information features of the target image to be migrated.

[0034] Furthermore, in step S3, the calculation formula for converting the feature information after style transfer into an image with the style of other participants using the decoder is:

[0035] I c←s '=decoder(AdaIN(s′,c)) #(4)

[0036] Where c←s' represents the migration of the style information feature s' of the target image to be migrated to the semantic content feature c of the source image, decoder represents the decoder, and I c←s' Represents the final image generated by style transfer.

[0037] Furthermore, in step S4, the Mini-Batch K-Means clustering algorithm is used to quickly select the batch size as Batch 2 To find the sample data of cluster centers, do the following:

[0038] 1) Select Batch 2 / 2 sample data retains the original style;

[0039] 2) The remaining batch 2 / 2 sample data for style exploration.

[0040] Furthermore, when performing style exploration, the average value of the feature channel of the new sample data style information and standard deviation The calculation formula is:

[0041]

[0042] Among them, α is a hyperparameter, which indicates the exploration level and is used to control the degree of deviation of the exploration style. The style information features obtained by exploration are used in the AdaIN model in batches of Batch. 2 The style is transferred from the original sample data to the original sample data, and the decoder is used to convert it into an image after style exploration.

[0043] Furthermore, the specific implementation process of step S6 is as follows:

[0044] Step S6.1: Initialize configuration;

[0045] 1) Deploy public open source uniformly distributed datasets on a central aggregation server;

[0046] 2) Determine the initial global model parameters;

[0047] 3) Determine the encryption algorithm for data transmission;

[0048] 4) Each participant needs to transmit the data volume involved in the model training to the central aggregation server;

[0049] Step S6.2: local training;

[0050] After the initial configuration is completed, each participant starts local model training. In each round of iteration, the objective function to be optimized by each participant is f(φ) represents the average loss of each participant, φ∈R d represents the model parameter of d dimension, ρ n represents the amount of data owned by the nth participant, then the calculation formula of f(φ) is:

[0051]

[0052] in, Represents the data sample of the nth participant The loss result obtained after prediction given the parameter φ is, represents the i-th sample data of the n-th participant, express Corresponding annotation data;

[0053] For the nth participant, the average loss of its samples is:

[0054]

[0055] The gradient of the nth participant in the tth iteration is The learning rate is η, and the calculation formula for the new round of local updates is as follows:

[0056]

[0057] in, represents the local model update parameters of the nth participant in the t+1th round, represents the local model parameters of the nth participant in the tth round, Represents the gradient operator.

[0058] Furthermore, a public open source evenly distributed dataset is deployed on the central aggregation server. The public open source evenly distributed dataset is an existing open source medical image dataset, which is used to verify the performance of the local model uploaded by each participant.

[0059] Furthermore, the specific implementation process of step S8 is as follows:

[0060] Assume there are N participants, and the total amount of data training is Recorded as ρ; then the data volume of the nth participant accounts for The calculation formula is:

[0061]

[0062] In the tth round, the nth participant uploads the trained local model parameters to the central aggregation server, and the central aggregation server verifies the public uniformly distributed dataset it stores using the uploaded local model. Let the score be The total score of all participants in round t is Set to S t ; then the data quality ratio of the nth participant in the tth round is The calculation formula is:

[0063]

[0064] Aggregate weight of model parameters for the nth participant The calculation formula is:

[0065]

[0066] Among them, β represents a hyperparameter, which is used to adjust the size of the influencing factors of data volume and data quality;

[0067] The global model parameters obtained by iterative update are:

[0068]

[0069] Among them, φ t+1 Represents the global model parameters obtained by the tth round of iterative update;

[0070] The overall loss function is:

[0071]

[0072] The beneficial effects of the present invention are:

[0073] 1. In federated learning, the data distribution of each participant is limited, and the trained model is prone to performance degradation when used for data outside the participating organization. In this regard, the present invention proposes a federated domain generalization method based on style transfer exploration, which can share the style information features of each participant and perform data enhancement by converting its own local original data into images with the style of other participants, thereby greatly enriching the data distribution of each participant and improving the generalization ability of the model.

[0074] 2. Currently, most aggregation algorithms do not consider data quality issues when performing aggregation. In actual training scenarios, the quality of training data of various participants is uneven. If data quality is not considered, the enthusiasm of participants to provide high-quality data will be reduced, and there may even be a phenomenon in which some institutions deliberately use low-quality data to reduce model performance. In this regard, the present invention comprehensively considers data volume and data quality, and proposes a multi-indicator federated aggregation algorithm based on public data verification, which provides a more reasonable weight allocation scheme for model aggregation, thereby improving model performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 This is a flow chart of a federal domain generalization method based on style transfer exploration in the present invention.

[0076] Figure 2 This is a flow chart of a multi-index federated aggregation algorithm based on public uniformly distributed data verification in the present invention.

[0077] Figure 3 A flowchart of an adaptive federated learning method based on style transfer exploration provided by the present invention. DETAILED DESCRIPTION

[0078] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0079] The adaptive federated learning method based on style transfer exploration of the present invention is mainly realized by the collaboration of a federated domain generalization method based on style transfer exploration and a multi-index federated aggregation algorithm based on public uniformly distributed data verification.

[0080] At present, most of the models trained by federated learning perform well when tested on the data of the participants, but if the model is applied to institutions outside the participants, the model performance will often decline. At present, there are relatively few studies on federated domain generalization, and many methods require centralized data processing, which does not conform to the premise of federated learning. In this regard, the present invention proposes a federated domain generalization method based on style transfer exploration (Style Transfer Exploration Federated Domain Generalization, STE FDG), which uses the pre-trained VGG-19 as an encoder to convert the content and style of the image into the feature space, and then normalizes the image features of the content so that its mean and variance match the mean and variance of the style image features; then uses the decoder to map the processed features back to the image space, thereby obtaining the image after style transfer; STE FDG will establish a style sharing center on the central aggregation server to store the style information of each participant. As a result, each participant can use the content information of the local sample data and the style information of other participants obtained from the central aggregation server for style transfer, and generate enhanced data with the style of other participants through the decoder, thereby enriching the data distribution of each participant.

[0081] In this invention, the AdaIN model (Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In ICCV, 2017.3,4,6,13) is applied to STE FDG to achieve federated domain generalization. Figure 1 As shown in the figure, before starting local training, each participant will calculate the style information of its local data and pass it to the central aggregation server. The central aggregation server will collect the style information of all participants and establish a style sharing center. Each participant can obtain the style information of other participants through the central aggregation server, thereby enriching the local data domain. In the process of updating the local data domain, each participant selectively converts the style of a small batch of original sample data into a new style with the style of other participants through the adaptive instance normalization model (AdaIN) to improve the diversity of the domain. At the same time, in order to further extrapolate new unseen domains, a part of the data can continue to be selected from the original sample data of each participant for style exploration.

[0082] At present, many aggregation algorithms have been proposed. Google proposed the federated average algorithm, increased the weights of participants with poor performance in the current model from the perspective of federated fairness, and some meta-learning-based methods. However, these rarely consider the data quality of each participant, which is not conducive to improving model performance. In this regard, the present invention proposes a multi-index federated aggregation algorithm based on public uniformly distributed data verification (Multi-index Federated Aggregation Algorithm Based on Public Data Verification, PDVFedMul). On the basis of the federated average algorithm (FedAvg), the quality of the data is also taken into account, so that the local model parameters uploaded by each participant have more reasonable weights. By comprehensively considering the volume and quality of data, reasonable weights can be adaptively and dynamically allocated, thereby improving the performance of the global model.

[0083] The present invention constructs a novel collaborative model of federated learning based on a federated domain generalization method based on style transfer exploration and a multi-index federated aggregation algorithm based on public uniformly distributed data verification. Through the STE FDG federated domain generalization technology, the local data domain distribution is enriched, and style exploration is also performed to further extrapolate the model to unseen domains, significantly improving the performance of the model on unseen domains. When aggregating local models, the present invention uses the PDFFedMul aggregation algorithm. Compared with the federated average algorithm (FedAvg), the PDFFedMul aggregation algorithm used in the present invention takes into account the indicator of local model quality and proposes a more reasonable model aggregation weight allocation scheme.

[0084] See also Figures 1 to 3 To illustrate, an adaptive federated learning method based on style transfer exploration of the present invention specifically includes the following steps:

[0085] Step S1: Establish a style sharing center;

[0086] Each participant calculates the overall style information features of the current data locally and then uploads it to the central aggregation server; after receiving the style information features from each participant, the central aggregation server will store them and then establish a style sharing center. By establishing a style sharing center on the central aggregation server, each participant can obtain the style information features of other participants, thereby greatly enriching the local data distribution; and there is no way to restore the image only through the style information features, so there is no risk of privacy leakage.

[0087] Specifically, the local data set of each participant is S n , and its corresponding VGG encoding style information feature is The semantic content features are According to formula (1) and formula (2), the average value μ of the style feature channel of the nth participant is calculated: n and standard deviation δ n :

[0088]

[0089] in, represents the channel mean of the style information features of the i-th image of the n-th participant, represents the standard deviation of the style information features of the i-th image of the n-th participant, represents the VGG coding style information feature of the i-th image of the n-th participant, ρ n Indicates the amount of data owned by the nth participant;

[0090]

[0091] When the image is an RGB image, C=3, and when the image is a grayscale image, C=1; H is the height of the feature map, and W is the width of the feature map.

[0092] The style information of each participant can be obtained by n =[μ n ,δ n ] indicates that each participant sends its style information to the central aggregation server. n ,Then the central aggregation server stores the style information received from all ,participants and establishes a style sharing center.

[0093] Step S2: Selective style transfer;

[0094] Each participant selects a portion of sample data from the original sample data set for style transfer using the Mini-Batch K-Means clustering algorithm. Using the AdaIN model, the style information features of other participants obtained from the central aggregation server are transferred to the selected portion of the original sample data;

[0095] Specifically, each participant obtains the style information Φ' of other participants from the central aggregation server n In order to obtain new data with the style of other participants without losing the performance of the original data, the Mini-Batch K-Means clustering algorithm is used to quickly select the batch size as Batch 1 To find the sample data of cluster centers, do the following:

[0096] 1) Select Batch 1 / 2 sample data retains the original style;

[0097] 2) The remaining batch1 / 2 sample data, style transfer is performed through the AdaIN model;

[0098]

[0099] Among them, δ(s') represents the standard deviation of the style information features of the target image to be migrated, μ(s') represents the mean of the style information features of the target image to be migrated, μ(c) represents the mean of the semantic content features of the source image, δ(c) represents the standard deviation of the semantic content features of the source image, c represents the semantic content features of the source image, and s' represents the style information features of the target image to be migrated.

[0100] Step S3: decoding to obtain the image after style transfer;

[0101] The decoder can be used to convert the feature information after style transfer through the AdaIN model into an image with the style of other participants, thereby greatly enriching the local data domain distribution; the specific calculation formula is as follows:

[0102] I c←s '=decoder(AdaIN(s′,c))#(4)

[0103] Where c←s' represents the migration of the style information feature s' of the target image to be migrated to the semantic content feature c of the source image, decoder represents the decoder, and Ic ←S' Represents the final image generated by style transfer.

[0104] Step S4: style exploration;

[0105] In order to further improve the performance of the model in unseen domains, style exploration is performed in the original sample data of each participant to obtain new style information features. The AdaIN model is then used to selectively transfer the explored style information features to the original sample data, thereby improving the generalization ability of the model in unseen domains.

[0106] Specifically, in order to further extrapolate the model to new unseen domains, the Mini-Batch K-Means clustering algorithm can be used on the original sample data of each participant to quickly select the batch size as Batch. 2 To find the sample data of cluster centers, do the following:

[0107] 1) Select Batch 2 / 2 sample data retains the original style;

[0108] 2) The remaining batch 2 / 2 sample data for style exploration; the average value of the feature channel of the new sample data style information and standard deviation The specific calculation formula is as follows:

[0109]

[0110] Among them, α is a hyperparameter, which indicates the exploration level and is used to control the degree of deviation of the exploration style. The style information features obtained by exploration are used in the AdaIN model in batches of Batch. 2 The style is transferred from the original sample data to the original sample data, and the decoder is used to convert it into an image after style exploration.

[0111] After the above steps S1 to S4, the generalization process of the federated domain is completed. Each participant's local data not only adds images with the style of other participants, but also adds images after style exploration based on the original sample data, thereby greatly enriching the local sample data of each participant. In the subsequent local training process, each participant uses the original sample data and the style transfer and exploration data for model training.

[0112] Step S5: The central aggregation server sends global model parameters;

[0113] The central aggregation server randomly generates a key pair for each participant to use for encryption, sends the global model parameters and public key to each participant, and retains the private key for decryption;

[0114] Step S6: distributed training;

[0115] Each participant uses the global model parameters issued by the central aggregation server, local original sample data and images after style transfer and exploration to train the model and generate a local model;

[0116] The specific implementation process is as follows:

[0117] Step S6.1: Initialize configuration;

[0118] Before officially conducting distributed training, you need to do some initial configuration as follows;

[0119] 1) Deploy public open source uniformly distributed datasets on a central aggregation server;

[0120] The central aggregation server needs to store a public evenly distributed dataset, which is an existing open source medical image dataset (it does not involve the local data of each participant, so there is no privacy leakage). This dataset has a wide range of sources and is evenly distributed. It is used to verify the performance of the local models uploaded by each participant.

[0121] 2) Determine the initial global model parameters; the central aggregation server needs to send the initial global model parameters and some basic settings to each participant;

[0122] 3) Determine the encryption algorithm for data transmission;

[0123] 4) Each participant needs to transmit the data volume involved in the model training to the central aggregation server, that is, the total number of data involved in the training of each participant, which will be used for the subsequent PDFFedMul aggregation algorithm to calculate the data volume index.

[0124] Step S6.2: local training;

[0125] After the initial configuration is completed, all participants can start local model training.

[0126] In each round of iteration, the objective function to be optimized by each participant is Among them, f(φ) represents the average loss of each participant, φ∈R d represents the model parameter of d dimension, ρ n represents the amount of data owned by the nth participant, then the calculation formula of f(φ) is as follows:

[0127]

[0128] in, Represents the data sample of the nth participant The loss result obtained after prediction given the parameter φ is, represents the i-th sample data of the n-th participant, express Corresponding annotation data;

[0129] For the nth participant, the average loss of its samples is:

[0130]

[0131] The gradient of the nth participant in the tth iteration is The learning rate is η, and the calculation formula for the new round of local updates is as follows:

[0132]

[0133] in, represents the local model update parameters of the nth participant in the t+1th round, represents the local model parameters of the nth participant in the tth round, Represents the gradient operator.

[0134] Step S7: Upload local model parameters;

[0135] Each participant uses the issued public key to encrypt the trained local model parameters and then upload them to the central aggregation server.

[0136] Step S8: Aggregate local models;

[0137] The central aggregation server uses the private key to decrypt the local model parameters uploaded by each participant, and uses the public uniformly distributed data set to verify the models uploaded by each participant, calculates the model scores of each participant, and then uses the PDVFedMul aggregation algorithm to aggregate the local models to obtain a new round of global model parameters;

[0138] The specific process is as follows:

[0139] Step S8.1: Calculation of data volume ratio;

[0140] When the amount of data for each participant to participate in the training is determined, the proportion of data volume remains unchanged during each round of aggregation, denoted as E 1 Assuming there are N participants, the total amount of data involved in training is Denoted as ρ, the data volume of the nth participant accounts for The calculation formula is as follows:

[0141]

[0142] Step S8.2: Calculation of data quality ratio;

[0143] In the tth round, after the nth participant uploads the trained local model parameters to the central aggregation server, the central aggregation server verifies the public uniformly distributed dataset it stores using the uploaded local model. Let the score be but It represents the total score of all participants in round t, set as S t ; then the data quality ratio of the nth participant in the tth round is The calculation formula is as follows:

[0144]

[0145] Step S8.3: Calculation of model parameter aggregation weights;

[0146] In round t, the model parameter aggregation weight of the nth participant The calculation formula is as follows:

[0147]

[0148] Among them, β represents a hyperparameter, which is used to adjust the size of the influencing factors of data volume and data quality. The larger β is, the greater the impact of the data volume ratio is, and vice versa, the greater the impact of the data quality ratio is.

[0149] Step S8.4: global model parameter calculation;

[0150] The global model parameters obtained by iterative update in the tth round are:

[0151]

[0152] Among them, φ t+1 Represents the global model parameters obtained by the tth round of iterative update.

[0153] Step S8.5: Calculation of overall loss function;

[0154] The overall loss function of the tth round is:

[0155]

[0156] Repeat steps S5 to S8 until the global model converges or reaches the specified number of iterations, and the federated learning task is completed.

[0157] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A federated domain generalization method based on style transfer exploration, characterized in that: The following steps are involved: Step S1: Each participant calculates the overall style information features of the current data locally and uploads them to the central aggregation server; the central aggregation server stores the style information features of each participant and establishes a style sharing center; Step S2: Selective style transfer; Each participant selects a part of sample data from the original sample data set through the Mini-Batch K-Means clustering algorithm for style transfer; the style information features of other participants obtained from the central aggregation server are transferred to a part of sample data selected from the original sample data set using the AdaIN model; Step S3: Use a decoder to convert the feature information after style transfer into an image with the style of other participants; Step S4: Perform style exploration in the original sample data of each participant to obtain new style information features; use the AdaIN model to selectively migrate the explored style information features to the original sample data; Step S5: The central aggregation server randomly generates a key pair for each participant to use for encryption, sends the global model parameters and public key to each participant, and retains the private key for decryption; Step S6: distributed training; Each participant uses the global model parameters issued by the central aggregation server, local original sample data and images after style transfer and exploration to train the model and generate a local model; Step S7: Each participant uses the public key to encrypt the trained local model parameters and upload them to the central aggregation server; Step S8: The central aggregation server uses the private key to decrypt the local model parameters uploaded by each participant, and uses the public uniformly distributed data set to verify the models uploaded by each participant, calculates the model scores of each participant, and then uses the PDVFedMul aggregation algorithm to aggregate the local models to obtain a new round of global model parameters; Repeat steps S5 to S8 until the global model converges or reaches the specified number of iterations, and the federated learning task ends.

2. The method for generalizing a federated domain based on style transfer exploration according to claim 1, characterized in that: In step S1, the local data set of each participant is S n , and its corresponding VGG encoding style information feature is The semantic content features are According to formula (1) and formula (2), the average value μ of the style feature channel of the nth participant is calculated: n and standard deviation δ n : in, represents the channel mean of the style information feature of the i-th image of the n-th participant, represents the standard deviation of the style information features of the i-th image of the n-th participant, represents the VGG coding style information feature of the i-th image of the n-th participant, ρ n Indicates the amount of data owned by the nth participant; When the image is an RGB image, C = 3, when the image is a grayscale image, C = 1; H is the height of the feature map, W is the width of the feature map; the style information of each participant is expressed by Φ n =[μ n , δ n ] indicates that each participant sends its style information to the central aggregation server. n ,The central aggregation server stores the style information received from all ,participants and builds a style sharing center.

3. The method for generalizing a federated domain based on style transfer exploration according to claim 1, characterized in that: The Mini-Batch K-Means clustering algorithm quickly selects sample data of cluster centers in batch 1 and performs the following operations: 1) Select the sample data of Batch1 / 2 to retain the original style; 2) The remaining sample data of Batch1I2 is used for style transfer of AdaIN model.

4. The method for generalizing a federated domain based on style transfer exploration according to claim 3, characterized in that: The calculation formula for style transfer of the AdaIN model is: Among them, δ(s′) represents the standard deviation of the style information features of the target image to be migrated, μ(s′) represents the mean of the style information features of the target image to be migrated, μ(c) represents the mean of the semantic content features of the source image, δ(c) represents the standard deviation of the semantic content features of the source image, c represents the semantic content features of the source image, and s′ represents the style information features of the target image to be migrated.

5. The method for generalizing a federated domain based on style transfer exploration according to claim 1, characterized in that: In step S3, the calculation formula for using the decoder to convert the feature information after style transfer into an image with the style of other participants is: I c←s′ =decoder(AdaIN(s′,c))#(4) Where c←s′ means migrating the style information feature s′ of the target image to be migrated to the semantic content feature c of the source image, decoder means the decoder, and I c←s′ Represents the final image generated by style transfer.

6. The method for generalizing a federated domain based on style transfer exploration according to claim 1, characterized in that: In step S4, the Mini-Batch K-Means clustering algorithm is used to quickly select the sample data of the cluster center of batch 2, and the following operations are performed: 1) Select Batch2 / 2 sample data to retain the original style; 2) The remaining Batch2 / 2 sample data is used for style exploration.

7. The method for generalizing a federated domain based on style transfer exploration according to claim 6, characterized in that: When performing style exploration, the average value of the feature channel of the new sample data style information and standard deviation The calculation formula is: Among them, α is a hyperparameter, which indicates the exploration level and is used to control the degree of deviation of the explored style. The style information features obtained by exploration are transferred to the original sample data with a batch size of Batch2 using the AdaIN model, and then converted into images after style exploration using the decoder.

8. The method for generalizing a federated domain based on style transfer exploration according to claim 1, characterized in that: The specific implementation process of step S6 is as follows: Step S6.1: Initialize configuration; 1) Deploy public open source uniformly distributed datasets on a central aggregation server; 2) Determine the initial global model parameters; 3) Determine the encryption algorithm for data transmission; 4) Each participant needs to transmit the data volume involved in the model training to the central aggregation server; Step S6.2: local training; After the initial configuration is completed, each participant starts local model training. In each round of iteration, the objective function to be optimized by each participant is f(φ) represents the average loss of each participant, φ∈R d represents the model parameter of d dimension, ρ n represents the amount of data owned by the nth participant, then the calculation formula of f(φ) is: in, Represents the data sample of the nth participant The loss result obtained after prediction given the parameter φ is, represents the i-th sample data of the n-th participant, express Corresponding annotation data; For the nth participant, the average loss of its samples is: The gradient of the nth participant in the tth iteration is The learning rate is η, and the calculation formula for the new round of local updates is as follows: in, represents the local model update parameters of the nth participant in the t+1th round, represents the local model parameters of the nth participant in the tth round, Represents the gradient operator.

9. The method for generalizing a federated domain based on style transfer exploration according to claim 8, characterized in that: A public open source evenly distributed dataset is deployed on a central aggregation server. The public open source evenly distributed dataset is an existing open source medical image dataset and is used to verify the performance of the local model uploaded by each participant.

10. The method for generalizing a federated domain based on style transfer exploration according to claim 1, characterized in that: The specific implementation process of step S8 is as follows: Assume there are N participants, and the total amount of data training is Recorded as ρ; then the data volume of the nth participant accounts for The calculation formula is: In the tth round, the nth participant uploads the trained local model parameters to the central aggregation server, and the central aggregation server verifies the public uniformly distributed dataset it stores using the uploaded local model. Let the score be The total score of all participants in round t is Set to S t ; then the data quality ratio of the nth participant in the tth round is The calculation formula is: Aggregate weight of model parameters for the nth participant The calculation formula is: Among them, β represents a hyperparameter, which is used to adjust the size of the influencing factors of data volume and data quality; the global model parameters obtained by iterative update are: Among them, φ t+1 Represents the global model parameters obtained by the t-th round of iterative update; The overall loss function is:

Images