Dynamic model aggregation method based on federated learning

Abstract

The invention discloses a dynamic model aggregation method based on federal learning. The method comprises the steps that a server builds an initial focus detection model and carries out pre-training; the client trains the initial focus detection model based on local data to obtain a local focus detection model; the server obtains weight parameters of the local focus detection model, and aggregates the parameters based on a dynamic model aggregation algorithm; the client iteratively trains a local focus detection model through federal learning based on the aggregated parameter weight until the local focus detection model converges; and testing the detection effect of the local focus detection model. Through the processing scheme disclosed by the invention, the shared global model can be cooperatively trained on the server without sharing the local training data, so that the privacy of the local data can be protected.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image detection, and in particular to a dynamic model aggregation method based on federated learning. Background Art

[0002] For data-driven deep learning models, large datasets play an important role in improving model performance. However, medical data from a single clinical site is often scarce and biased, and therefore cannot support the establishment of accurate and robust deep learning models for critical medical applications. The simplest way to solve this problem is to obtain a large dataset by aggregating medical data from multiple different clinical sites. However, in the healthcare field, hospitals hope to improve their clinical diagnosis systems without leaking patient records. For privacy and security reasons, sharing medical data across clinical sites will be limited.

[0003] However, since the data between clients is usually non-independent and identically distributed (non-iid), different devices have specific preferences for data generation or collection, resulting in feature distribution drift and label distribution skew, which causes the aggregate model to deviate from the optimal representation. Therefore, it is difficult to learn a single global model applicable to all clients.

[0004] It can be seen that how to create a new dynamic model aggregation method has become a goal that the industry urgently needs to improve. Summary of the invention

[0005] In view of this, an embodiment of the present disclosure provides a dynamic model aggregation method based on federated learning, which at least partially solves the problems existing in the prior art.

[0006] In a first aspect, an embodiment of the present disclosure provides a dynamic model aggregation method based on federated learning, the method comprising the following steps: The server builds the initial lesion detection model and performs pre-training; The client trains the initial lesion detection model based on local data to obtain a local lesion detection model; The server obtains weight parameters of the local lesion detection model and aggregates the parameters based on a dynamic model aggregation algorithm; The client iteratively trains the local lesion detection model through federated learning based on the aggregated parameter weights until the local lesion detection model converges; The detection effect of the local lesion detection model is tested.

[0007] According to a specific implementation of the embodiment of the present disclosure, the server builds an initial lesion detection model and performs pre-training, including: Faster R-CNN is used as the lesion detection model, and the lesion detection model is pre-trained based on the public dataset to obtain the initial lesion detection model.

[0008] According to a specific implementation of the embodiment of the present disclosure, the client trains the initial lesion detection model based on local data to obtain a local lesion detection model, including the following steps: Acquire the lesion image collected by the client, and preprocess the lesion image; wherein the preprocessing includes resizing, cropping, normalizing, color jittering and contrast enhancement of the lesion image; The client iteratively trains the local lesion detection model through the preprocessed lesion images to obtain the local model weights for the convergence of the local lesion detection model of the client.

[0009] According to a specific implementation of the embodiment of the present disclosure, the loss function of iterative training is calculated based on the following formula: ; in, is the client’s loss function; For images with lesions; is the number of images of the lesion; is the classification loss; is the bounding box regression loss.

[0010] According to a specific implementation of the embodiment of the present disclosure, the server obtains weight parameters of the local lesion detection model and aggregates the parameters based on a dynamic model aggregation algorithm, including: Dynamic model aggregation based on the following formula: ; in, is the global model parameter in the t+1th iteration; is the local parameter trained by client n in the t+1th round of iterations; N is the number of clients participating in model aggregation in each round of federated learning; h is the dynamic weight factor of client n.

[0011] According to a specific implementation of an embodiment of the present disclosure, the dynamic weight factor of the client n is calculated based on the following formula: ; ; in, The local parameters trained by client n in the t+1th iteration; The average weight of N clients is used as the standard for calculating the weight space distance; h 1 is the dynamic weight factor of client 1; h2 is the dynamic weight factor of client 2; H n is the cosine similarity between clients.

[0012] According to a specific implementation of the embodiment of the present disclosure, testing the detection effect of the local lesion detection model includes the following steps: Detecting the local lesion detection model based on average accuracy, sensitivity and visualization effect graph; Among them, the average precision is calculated based on the following formula: ; Among them, AP is the average precision; It is a curve consisting of recall rate as the horizontal axis and precision as the vertical axis; The sensitivity is calculated based on the following formula: ; in, is the number of positive samples correctly predicted as positive samples; is the number of positive samples that are incorrectly predicted as negative samples.

[0013] The dynamic model aggregation method based on federated learning in the disclosed embodiment allows multiple clients to learn together, and obtains a global model for image detection after aggregation iteration. The target detection model for lesions that is adapted to the federated learning framework effectively utilizes the data of the client under the federated learning framework. It allows multiple local clients to collaboratively train a shared global model on the server based on the principle of privacy protection without sharing local training data, providing an effective solution for local data privacy protection. On this basis, the present invention proposes a new dynamic model aggregation method to better optimize the weight configuration between clients. By measuring the degree of similarity distance between different local models under privacy protection constraints, the drift of the client's local representation learning can be better controlled, and the weight configuration between clients can be optimized and the global aggregation can be promoted dynamically, so that the network weight can be better biased towards the overall feature representation. The present invention uses federated learning to effectively protect the privacy security of local medical data, and controls the drift of the client's local representation learning through a dynamic model aggregation algorithm, so that the network weight can be better biased towards the overall feature representation, thereby enhancing the robustness of the lesion detection model. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0015] Figure 1 A schematic diagram of a flow chart of a dynamic model aggregation method based on federated learning provided in an embodiment of the present disclosure; Figure 2 A schematic diagram of a flow chart of a dynamic model aggregation method based on federated learning provided in an embodiment of the present disclosure; Figure 3 A schematic diagram of a Faster R-CNN model training provided in an embodiment of the present disclosure; Figure 4 A schematic diagram of a federated learning framework provided for an embodiment of the present disclosure; Figure 5 A schematic diagram of a dynamic aggregation algorithm structure provided in an embodiment of the present disclosure; Figure 6 A comparative experimental visualization diagram provided for an embodiment of the present disclosure. DETAILED DESCRIPTION

[0016] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0017] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0018] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement a device and / or practice a method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0019] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0020] The embodiment of the present invention provides a dynamic model aggregation method based on federated learning, which applies federated learning to the field of target detection and then allocates weights using a new dynamic aggregation algorithm.

[0021] Traditional federated learning algorithms usually adopt FedAG model aggregation. It weights and averages the local models participating in the aggregation according to the proportion of data each client has relative to the total training data. However, local models trained on clients in a federated learning setting usually perform poorly in the early stages of federated learning. If weights are assigned only based on the size of the client dataset, assigning a large amount of weight to unlabeled clients may have an adverse effect on the update of the global model, hindering its convergence and performance. In addition, due to the non-iid nature of data between clients, different devices show specific preferences in data generation or collection, resulting in feature distribution drift and label distribution skewness. This causes the aggregated model to deviate from the optimal representation, making it challenging to learn a single global model suitable for all clients.

[0022] Although existing federated learning methods can solve the privacy problem of medical datasets by aggregating the parameters of local models, there is a problem of feature offset caused by inconsistent client data distribution in the federated learning detection framework. The present invention proposes a new dynamic model aggregation method, which can better control the drift of client local representation learning by measuring the similarity distance degree of different local models under privacy protection constraints, and dynamically optimize the weight configuration between clients and promote global aggregation, so that the network weight can better bias towards the overall feature representation.

[0023] Federated learning is a decentralized distributed machine learning framework that allows multiple local clients to collaboratively train a shared global model on a server based on the principle of privacy protection without sharing local training data. It provides an effective solution for local data privacy protection by achieving decentralized training across distributed data sources, that is, data is privately accessed by each local client without aggregating local data on a cloud server. This local data decentralized learning framework brings many advantages in solving real-world challenges such as data privacy, security, and access rights. The application of advanced federated learning technology in medical imaging has attracted widespread attention. The present invention adopts the classic target detection model fasterR-CNN as the lesion detection model. The detection model consists of Backbone, Neck, and Head to extract and detect features of lesion images.

[0024] This paper proposes a new method for lesion detection based on federated learning, which allows multiple clients (hospitals) to learn together and obtain a global model for image detection after aggregation and iteration. The federated learning framework in the clinical environment can significantly improve learning performance without sharing data by collaborating with distributed clients.

[0025] Figure 1 A schematic diagram of the process of a dynamic model aggregation method based on federated learning provided in an embodiment of the present disclosure.

[0026] Figure 2 For Figure 1 Corresponding flow chart of the dynamic model aggregation method based on federated learning.

[0027] like Figure 1 As shown, in step S110, the server builds an initial lesion detection model and performs pre-training.

[0028] In an embodiment of the present invention, the server builds an initial lesion detection model and performs pre-training, including: using Faster R-CNN as a lesion detection model, and pre-training the lesion detection model based on a public data set to obtain an initial lesion detection model.

[0029] More specifically, for example, Figure 3 As shown in the figure, taking polyp detection as an example, the server uses the classic target detection model Faster R-CNN as the polyp lesion detection model. The detection model consists of Backbone, Neck, and Head. It extracts and detects features of polyp lesion images, and pre-trains the detection model on the public data set to obtain the initial polyp lesion detection model. Among them, the public polyp data set includes four independent polyp data sets Kvasir-SEG, ETIS-Larib polypDB, CVC-ClinicDB, and CVC-ColonDB.

[0030] The Kvasir-SEG dataset is derived from the previous Kvasir dataset and contains 1000 polyp images with corresponding annotations, and polyp-specific bounding box annotations are generated using segmentation masks. The ETIS-Larib Polyp DB consists of 196 frames of polyps extracted from colonoscopy videos and their corresponding masks. The CVC-ClinicDB dataset includes 612 images with associated polyp and background segmentation masks obtained from 31 polyp video sequences of 23 patients. CVC-ColonDB extracts 1200 frames from 15 short colonoscopy sequences and annotates 380 polyp images with the largest size and perspective variations.

[0031] More specifically, the process proceeds to step S120.

[0032] In step S120, the client trains the initial lesion detection model based on local data to obtain a local lesion detection model.

[0033] In an embodiment of the present invention, the client trains the initial lesion detection model based on local data to obtain a local lesion detection model, including the following steps: acquiring a lesion image collected by the client, and preprocessing the lesion image; wherein the preprocessing includes resizing, cropping, normalizing, color jittering and contrast enhancement of the lesion image; the client iteratively trains the local lesion detection model through the preprocessed lesion image to obtain the local model weights for the convergence of the local lesion detection model of the client.

[0034] In this embodiment of the present invention, the loss function of iterative training is calculated based on the following formula: ; in, is the client’s loss function; For images with lesions; is the number of images of the lesion; is the classification loss; is the bounding box regression loss.

[0035] More specifically, for example, client data acquisition and preprocessing include resizing, cropping, normalization, color jittering, and contrast enhancement for polyp lesion images.

[0036] like Figure 5 As shown, the initial detection model and parameters in the server are sent to each client. The model is the polyp lesion detection model required for client training. In the subsequent federated learning framework, the model is divided into a global model in the server and a local model trained only locally to learn global features and local features respectively. In the iterative training of federated learning, only the global model weights updated by aggregation on the server are sent to each client.

[0037] Then, the client iteratively trains the local model (or local detection model) through the local polyp lesion data to obtain the local model weights for the client local model convergence. The iterative loss function is specifically: …………Formula 1 Next, go to step S130.

[0038] In step S130, the server obtains weight parameters of the local lesion detection model and aggregates the parameters based on a dynamic model aggregation algorithm.

[0039] In an embodiment of the present invention, the server obtains weight parameters of the local lesion detection model and aggregates the parameters based on a dynamic model aggregation algorithm, including: Dynamic model aggregation based on the following formula: …………Formula 2 in, is the global model parameter in the t+1th iteration; is the local parameter trained by client n in the t+1th round of iterations; N is the number of clients participating in model aggregation in each round of federated learning; h is the dynamic weight factor of client n; In an embodiment of the present invention, the dynamic weight factor of the client n is calculated based on the following formula: …………Formula 3 …………Formula 4 in, The local parameters trained by client n in the t+1th round of iterations; The average weight of N clients is used as the standard for calculating the weight space distance; h 1 is the dynamic weight factor of client 1; h 2 is the dynamic weight factor of client 2; H n is the cosine similarity between clients.

[0040] Next, go to step S140.

[0041] In step S140, the client iteratively trains the local lesion detection model through federated learning based on the aggregated parameter weights until the local lesion detection model converges.

[0042] More specifically, for example, Figure 4 As shown, the client uploads the trained local model weight parameters to the server for the next aggregation operation. The server aggregates the client parameters through the dynamic model aggregation algorithm, and the parameters obtained after aggregation and update are the global parameters of the server.

[0043] The dynamic model aggregation update function is: in, is the global model parameter in the t+1th iteration, is the local parameter trained by client n in the t+1th round of iterations, N is the number of clients participating in the model aggregation in each round of federated learning, and h is the dynamic weight factor of client n, which is calculated as follows: in, is the local parameter trained by client n in the t+1th round of iterations, The average weight of N clients is used as the criterion for calculating the weight space distance.

[0044] Next, go to step S150.

[0045] In step S150, the detection effect of the local lesion detection model is tested.

[0046] In an embodiment of the present invention, testing the detection effect of the local lesion detection model includes the following steps: Detecting the local lesion detection model based on average accuracy, sensitivity and visualization effect graph; Among them, the average precision is calculated based on the following formula: …………Formula 5 Among them, AP is the average precision; It is a curve consisting of recall rate as the horizontal axis and precision as the vertical axis; The sensitivity is calculated based on the following formula: …………Formula 6 in, is the number of positive samples correctly predicted as positive samples; is the number of positive samples that are incorrectly predicted as negative samples.

[0047] More specifically, for example, in order to comprehensively evaluate the effectiveness and advancement of the method proposed in the present invention under the federated learning framework, the detection performance indicators include average precision AP, sensitivity, and visualization effect diagrams.

[0048] In order to more realistically simulate the actual situation of federated learning, the present invention distributes these four non-independent and identically distributed data sets to four clients, and the four clients contain 1000, 196, 612 and 380 polyp images respectively, with resolutions ranging from 384×288 to 1920×1072 pixels. In the embodiment of the present invention, the present invention randomly distributes the training set, validation set and test set of each client in a ratio of 7:1:2.

[0049] In the embodiment of the present invention, in order to comprehensively evaluate the effectiveness and advancement of the method proposed in the present invention in the detection task of polyp images in the federated semi-supervised learning scenario, the present invention uses AP (AP50) which is more common in medical lesion detection to indicate that the detection IOU threshold is 0.5, and also uses sensitivity to ensure that the real target is better detected while minimizing missed detection, which is particularly important for the detection of medical lesion areas. Sensitivity represents the ability of the model to detect lesion areas, and the specific formula is as follows:

[0050] (True Positives) represents the number of positive samples that are correctly predicted as positive samples, (False Negatives) represents the number of positive samples that were incorrectly predicted as negative samples.

[0051] AP is the most commonly used target detection indicator. It measures the prediction accuracy of the detection algorithm at different confidence levels. It comprehensively considers the recall rate (R) and precision (P) to objectively evaluate the performance of the target detection algorithm at different thresholds. The higher the AP value, the better the detection performance.

[0052] (False Positives) represents the number of positive samples that were incorrectly predicted as positive samples.

[0053] It represents a curve with recall (R) as the horizontal axis and precision (P) as the vertical axis.

[0054] In order to evaluate and highlight the performance of the federated learning polyp lesion detection method based on client feature alignment, the present invention selects three advanced federated learning frameworks, FedAvg, FedBN, and FedMatch, as the baselines for the polyp image detection task. Among them, the weight of each local client in FedAvg is proportional to its data volume, and a global model is generated by aggregating multiple local models on the server side. On this basis, FedBN alleviates the feature shift in the heterogeneity of federated learning data by adding a batch normalization layer (BN) to the local model. FedMatch aggregates each local model by designing an aggregation method that takes into account the reliability of the local model. The visualization diagram is as follows Figure 6 In the prior art, in the process of federated learning lesion detection, due to the feature offset caused by inconsistent client data distribution and the poor feature representation learning ability of unlabeled clients, satisfactory results cannot be achieved in polyp image detection under the federated learning framework.

[0055] The dynamic model aggregation method based on federated learning proposed in the present invention assigns appropriate weights to each model by measuring the similarity distance between different local models under privacy constraints. In the weight aggregation stage, the client weight allocation is dynamically optimized according to the similarity distance of the weights of each layer. This optimization promotes global aggregation and increases the proportion of client weights that contribute more to the performance of the global model. It also alleviates the problem of feature drift caused by inconsistent client data distribution. This means that during the training process, the present invention can better control the drift of local feature learning between clients, making the network weights more inclined to the overall feature representation. Therefore, the global model can obtain a more comprehensive feature representation than a single client model, thereby improving the overall detection accuracy and robustness.

[0056] The above is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present disclosure should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A dynamic model aggregation method based on federated learning, characterized in that: The method comprises the following steps: The server builds the initial lesion detection model and performs pre-training; The client trains the initial lesion detection model based on local data to obtain a local lesion detection model; The server obtains weight parameters of the local lesion detection model and aggregates the parameters based on a dynamic model aggregation algorithm; The client iteratively trains the local lesion detection model through federated learning based on the aggregated parameter weights until the local lesion detection model converges; The detection effect of the local lesion detection model is tested.

2. The dynamic model aggregation method based on federated learning according to claim 1, characterized in that: The server builds an initial lesion detection model and performs pre-training, including: Faster R-CNN is used as the lesion detection model, and the lesion detection model is pre-trained based on the public dataset to obtain the initial lesion detection model.

3. The dynamic model aggregation method based on federated learning according to claim 1, characterized in that: The client trains the initial lesion detection model based on local data to obtain a local lesion detection model, including the following steps: Acquire the lesion image collected by the client, and preprocess the lesion image; wherein the preprocessing includes resizing, cropping, normalizing, color jittering and contrast enhancement of the lesion image; The client iteratively trains the local lesion detection model through the preprocessed lesion images to obtain the local model weights for the convergence of the local lesion detection model of the client.

4. The dynamic model aggregation method based on federated learning according to claim 3 is characterized in that: The loss function for iterative training is calculated based on the following formula: ； in, is the client’s loss function; For images with lesions; is the number of images of the lesion; is the classification loss; is the bounding box regression loss.

5. The dynamic model aggregation method based on federated learning according to claim 1, characterized in that: The server obtains weight parameters of the local lesion detection model and aggregates the parameters based on a dynamic model aggregation algorithm, including: Dynamic model aggregation based on the following formula: ； in, is the global model parameter in the t+1th iteration; is the local parameter trained by client n in the t+1th round of iterations; N is the number of clients participating in model aggregation in each round of federated learning; h is the dynamic weight factor of client n.

6. The dynamic model aggregation method based on federated learning according to claim 5, characterized in that: The dynamic weight factor of client n is calculated based on the following formula: ； ； in, The local parameters trained by client n in the t+1th round of iterations; The average weight of N clients is used as the standard for calculating the weight space distance; h 1 is the dynamic weight factor of client 1; h 2 is the dynamic weight factor of client 2; H n is the cosine similarity between clients.

7. The dynamic model aggregation method based on federated learning according to claim 1, characterized in that: Testing the detection effect of the local lesion detection model includes the following steps: Detecting the local lesion detection model based on average accuracy, sensitivity and visualization effect graph; Among them, the average precision is calculated based on the following formula: ； Among them, AP is the average precision; It is a curve consisting of recall rate as the horizontal axis and precision as the vertical axis; The sensitivity is calculated based on the following formula: ； in, is the number of positive samples correctly predicted as positive samples; is the number of positive samples that are incorrectly predicted as negative samples.

Images