Image anatomical segmentation method and system based on federated client customized adapter

Abstract

The present invention provides an image anatomical segmentation method and system based on a federated client customized adapter, the method comprising: performing adapter learning on a basic medical model, inserting a global-local decoupler GLD in the adapter module to obtain a client adaptive adapter; performing a channel contribution quantification operation on the convolutional layer in the client adaptive adapter to obtain the contribution score of each channel to generate a score as a parameter of the basic medical model; setting a client customized federated learning algorithm, using a client customized federated update algorithm under a federated learning paradigm, performing a federated update operation on the client according to the contribution score of each channel to obtain a suitable personalized model; model training and testing. The present invention solves the technical problems that the heterogeneity of distributed medical data island clients is significant, resulting in an imbalance in the categories of medical data, the learned global compromise model deviates greatly from the individual optimality of each client, and has low stability and slow convergence.

Description

Technical Field

[0001] The present invention relates to the field of deep learning, and in particular to an image anatomical segmentation method and system based on a federated client customized adapter. Background Art

[0002] Anatomical segmentation of medical images plays a key role in various medical applications such as epicardial fat segmentation, brain and brain tumor segmentation, which can help more accurate diagnosis and reduce the burden of medical professionals. Recent advances in deep learning and large-scale base models have significantly promoted the development of this field. U-Net, as a pioneer using the encoder-decoder architecture, is one of the most important methods in medical image segmentation. Various variants based on different architectures have been designed to handle medical image segmentation tasks, including CNN (Convolutional Neural Network)-based variants and transformer-based variants. In recent years, large-scale medical base models with strong cross-domain knowledge understanding, logical reasoning, and language generation capabilities have also greatly promoted the development of medical image segmentation. The application of these methods has significantly improved the accuracy of computer-aided diagnosis and simplified clinical workflows.

[0003] Despite their great success, existing approaches mainly belong to a centralized learning paradigm, where medical image data from different sources are fully delivered to a central server to jointly learn a single optimal segmentation model. In real medical scenarios, there is only limited access to distributed data silos, where sharing of local medical data between different clients is prohibited. Due to factors such as strict privacy regulations and limited network bandwidth, only insensitive model weights are allowed to be shared. Therefore, existing centralized approaches are no longer applicable to distributed medical scenarios.

[0004] Distributed medical data islands often exhibit severe client heterogeneity, including class imbalance and distribution diversity. This is because different hospitals specialize in different diseases, different anatomical regions have different probabilities of being examined in medical imaging equipment, and long-tail data distribution, which makes class imbalance a common phenomenon in medical data. Distribution diversity is usually caused by different data collection conditions, such as equipment, personnel, and environmental factors. Experiments show that due to the heterogeneity of various client fields, the direct application of traditional federated learning (FL) in medical image anatomical segmentation has significant instability and slow convergence. For example, the existing invention patent application document "A medical image segmentation method based on active federated learning" with publication number CN118335295A includes: the server obtains medical image data and preprocesses it, and then divides it into known clients and unknown clients; the known clients perform a round of federated learning on the image segmentation model based on the local training data set; the server uses an active learning method to reconstruct the training data set of each known client and sends it to the known clients to perform a round of federated learning; the server performs several distillation processes based on the model parameters after the current federated learning, and sends the model parameters obtained each time to each unknown client for model testing, and determines the final test results of the model parameters each time based on the feedback test results; the optimal parameters of the image segmentation model are obtained based on the model parameters corresponding to the optimal final test results. And the existing invention patent application document "A Medical Image Segmentation System Based on Federated Learning" with publication number CN116563301A, the existing method includes: training a domain classifier for domain classification based on image features; combining batch normalization and instance normalization in series according to the characteristics of batch normalization and instance normalization to form a style normalization module; using a style normalization module at the shallow level of the encoder of the segmentation model according to the structural characteristics of the deep neural network to form a style normalization segmentation network; according to the image features of the unknown domain, the domain classifier is input into the model-level attention module, and then the weight parameters of each domain model are generated and weighted aggregation is performed to obtain a new global model generalized to the unknown domain. In addition, domain heterogeneity also causes the learned global compromise model to deviate greatly from the individual optimality of each client, which inhibits the advantage of using distributed data to promote learning.

[0005] The existing invention patent application document "Personalized Federated Learning Method Based on Decoupled Knowledge Distillation" with publication number CN117152480A is implemented by adding decoupled knowledge distillation to the local client. By adding a local private model locally, the local client cooperates to train the global model and the local client independently trains the personalized model, as well as the knowledge interaction of the two models. The logistic regression layer is divided into a weighted sum of the target class and the non-target class. This approach enhances the effectiveness and flexibility of non-target class distillation. However, the aforementioned prior art does not divide and group the parameters of the middle layer of the model, such as convolution kernels and neurons, nor does it decouple at a more detailed level, which restricts the flexibility and effectiveness of the model.

[0006] The existing invention patent application document "Medical Image Classification Method, System, Device and Medium for Personalized Federated Learning" with publication number CN118053033A, the existing method includes: several medical institutions use private data for training, and use inter-class separability loss as the loss function to supervise the training to obtain a local model; decouple each local model into a general component and a personalized component, wherein the general component includes general information for medical image classification, and the personalized component includes personalized information of each medical institution; aggregate and update all general components to obtain new general components; transmit the new general components back to each client, integrate them with the personalized components in each client, generate the final local model, and classify the medical image based on the final local model. However, the parameters of the model can not only learn its own unique and globally common information, but also learn some common information between clients. The above-mentioned prior art method of directly binarizing the model parameters is too absolute, and this method fails to capture and utilize the common information between some clients. The above-mentioned existing scheme affects the effect of model update.

[0007] In summary, the existing technology has the problems of significant heterogeneity of distributed medical data island clients, resulting in imbalanced categories of medical data, large deviation between the learned global compromise model and the individual optimality of each client, as well as low stability and slow convergence. Summary of the invention

[0008] The technical problem to be solved by the present invention is: how to solve the technical problems existing in the prior art that the heterogeneity of distributed medical data island clients is significant, resulting in an imbalance in the categories of medical data, a large deviation between the learned global compromise model and the individual optimality of each client, as well as low stability and slow convergence.

[0009] The present invention adopts the following technical solution to solve the above technical problem: the image anatomical segmentation method based on the federated client customized adapter includes:

[0010] S1. Perform adapter learning on the medical basic model to complete the parameter-efficient fine-tuning (PEFT) operation. Insert the global-local decoupler GLD into the adapter module of the medical basic model to obtain a client-side adaptive adapter to adapt to the federated learning paradigm.

[0011] S2. Perform channel contribution quantization operation on the convolution layer in the client adaptive adapter to obtain the contribution score of each channel to generate the score as the parameter of the medical basic model;

[0012] S3. Set a client-customized federated learning algorithm. Under the federated learning paradigm, use the client-customized federated update algorithm to perform a federated update operation on the client according to the contribution score of each channel to obtain an applicable personalized model.

[0013] S4. Train and test the applicable personalized model, and obtain the segmentation masks of each structure by processing the preset medical image using the applicable personalized model.

[0014] The present invention designs a client-side adaptive adapter for medical anatomical segmentation. The present invention takes the medical basic model as the backbone, improves the traditional adapter module, and designs a client-customized federated learning algorithm, which can find the optimal applicable personalized model for different clients under the federated learning paradigm, and can achieve stable and personalized decentralized segmentation without sharing sensitive local data. The client-side adaptive adapter and client-customized federated learning algorithm proposed by the present invention can be directly applied to different medical basic models.

[0015] The present invention can adaptively customize local models for different clients, tightly couple the efficient client adaptive adapter with the client customized federated learning algorithm, and effectively extract the common knowledge in the basic medical model to alleviate the instability in federated learning training. The present invention stabilizes the heterogeneous federated learning process, achieves optimal client personalization, and replaces the existing global compromise segmentation model.

[0016] In a more specific technical solution, in S1, the adapter module includes: a fully connected layer FC and a convolutional layer Conv, wherein a global-local decoupler GLD is added to each convolutional layer Conv and the fully connected layer FC to adaptively decouple the adapter module into a global unit and a local unit.

[0017] The present invention proposes and applies a new parameter decomposition strategy to adaptively decompose adapter parameters into global units and local units, thereby achieving optimal client-side personalized federated learning and further stabilizing the heterogeneous federated training process.

[0018] The present invention adds a global local decoupler (GLD) to each convolutional layer Conv and fully connected layer (FC) to adaptively decouple the adapter into global units and local units. The global local decoupler GLD is designed specifically for the differences between the convolutional layer and the fully connected layer, and can play a consistent function.

[0019] In a more specific technical solution, S2 includes:

[0020] S21, quantify the contribution score of each convolution channel unit unit ,in, u represents the convolution channel unit, k , j Indicates the serial number of the client;

[0021] S22, using preset logic to process input features;

[0022] S23, the contribution of each convolution channel unit to the client The outer product is made with the predicted domain client probability output by the global-local decoupler GLD to obtain the channel contribution score of each client. .

[0023] In a more specific technical solution, in S21, the following formula is used to determine k Convolutional channel units on the client u For j Contribution score of clients :

[0024] =

[0025] in, Indicates i The samples correspond to the convolutional channel units in the global-local decoupler GLD. u Some features of Represents the convolution channel unit used to judge the global-local decoupler GLD u For k The parameters of the client classification contribution, Indicates i The input features of the samples Belong to j The predicted domain client probability of a client, Indicates i The input features of the samples Predicted to belong to j In the case of a client, dDenotes the global-local decoupler GLD.

[0026] The present invention divides and groups the parameter convolution kernels and multi-layer perceptron (MLP) neurons in the middle layer of the model, decouples at a more detailed level, and optimizes the flexibility and effectiveness of the model.

[0027] In a more specific technical solution, in S22, the following logic is used to process each input feature: :

[0028]

[0029] in, U Indicates the number of units, K Indicates the number of clients, and the Broadcast function indicates the broadcast of shape matching elements. is the weight of the global-local decoupler GLD, ⊙ is the element-wise multiplication, Indicates the contribution of each convolution channel unit to the client. Represents the shape U×K The real number, i Indicates the input feature number.

[0030] In a more specific technical solution, in S23, the following logic is used to use the i The predicted domain client probability corresponding to the input features , the contribution of each convolution channel unit to the client Perform weighted processing to obtain the contribution score of each channel of each client :

[0031]

[0032] in, represents the outer product, ， The Broadcast function represents the broadcast of shape matching elements.

[0033] In a more specific technical solution, S3 includes:

[0034] S31, executing a binarization federal update algorithm BFU (Binarization Federal Update), wherein, according to the contribution score of each channel, each convolution channel unit and the generation parameter of the convolution channel unit are binarized and divided into global parameters and local parameters, and the convolution channel unit and its own generation parameters are used as channel parameter units, and local units and global units are obtained by processing them, and global units are used for global update;

[0035] S32. Execute the Smooth Federal Update (SFU) algorithm, using the contribution score of the current channel parameter unit to represent the score of each client learned by the convolutional channel unit. According to the score, weights are added during the update process of each local model parameter to obtain a suitable personalized model to update the client.

[0036] The present invention closely combines the parameter-efficient adapter with the federated learning algorithm to effectively extract the common knowledge in the basic medical model to stabilize heterogeneous federated learning. The present invention proposes a federated update algorithm that is compatible with the federated client customized adapter, which can decouple the parameters of the local model, perform differential updates based on the different information learned from the model parameters, and further stabilize the heterogeneous federated training process.

[0037] During the execution of the smooth federated update algorithm SFU of the present invention, the amount of information of each client carried by the calculation model parameters can be captured, and each client model can be updated in a weighted manner according to the amount of information of each client.

[0038] In a more specific technical solution, in S31, the following logic is used to determine the similarity of the contribution of each channel parameter unit to different clients: :

[0039]

[0040] in, Indicated in k Convolutional channel units on the client u For j The contribution score of each client, express K The sum of the classification contribution scores of the clients, represents the logarithm of the classification contribution with base 2, Represents the shape real number;

[0041] Determine the similarity of contribution , is it less than the threshold ;

[0042] If yes, the current channel parameter unit is used as the local unit, otherwise, the channel parameter unit is used as the global unit;

[0043] Generate a mask Mask for each convolution channel unit on each client to represent the local unit and the global unit:

[0044]

[0045] in, Represents the convolution channel unit on the kth client u Mask;

[0046] When performing a global update, the global unit is globally integrated, wherein the global unit is updated using the following logic:

[0047]

[0048] in, Represents the convolution channel unit u exist t+1 The corresponding weight parameter for the round W , Represents the convolution channel unit on the kth client u In the t The weight parameter for rounds W .

[0049] The present invention customizes the local model required by each client by decoupling the parameters of the local model and performing differentiated updates based on different information learned from the model parameters, thereby further stabilizing model training.

[0050] In a more specific technical solution, in S32, the convolution channel unit of the kth client is executed using the following logic: u Global integration operations:

[0051]

[0052] in, Indicated in k Convolutional channel units on the client u For j The contribution score of each client, Indicates j The convolutional channel unit of the client in round t+1 u The weight parameter W .

[0053] The present invention designs two different federated update strategies for customized federated learning, including binary federated update BFU and smooth federated update SFU. SFU is conceptually based on BFU and is a generalized version of BFU. Both strategies can effectively alleviate the problems of instability and slow convergence of training in heterogeneous federated learning, and the generalized strategy SFU achieves better performance.

[0054] In a more specific technical solution, the image anatomical segmentation system based on the federated client customized adapter includes:

[0055] The client-side adaptive adapter building module is used to perform adapter learning on the medical basic model to complete the parameter efficient fine-tuning PEFT operation. In the adapter module of the medical basic model, the global-local decoupler GLD is inserted to obtain the client-side adaptive adapter to adapt to the federated learning paradigm;

[0056] A contribution score acquisition module is used to perform channel contribution quantification operation on the convolution layer in the client adaptive adapter to obtain the contribution score of each channel to generate the score as the parameter of the medical basic model. The contribution score acquisition module is connected to the client adaptive adapter construction module;

[0057] The federated update module is used to set the client-customized federated learning algorithm. Under the federated learning paradigm, the client-customized federated update algorithm is used to perform a federated update operation on the client according to the contribution score of each channel to obtain a suitable personalized model. The federated update module is connected to the contribution score acquisition module.

[0058] The image segmentation module is used to train and test the applicable personalized model, and obtain the segmentation masks of each structure based on the preset medical image using the applicable personalized model.

[0059] Compared with the prior art, the present invention has the following advantages:

[0060] The present invention designs a client-side adaptive adapter for medical anatomical segmentation. The present invention takes the medical basic model as the backbone, improves the traditional adapter module, and designs a client-customized federated learning algorithm, which can find the optimal applicable personalized model for different clients under the federated learning paradigm, and can achieve stable and personalized decentralized segmentation without sharing sensitive local data. The client-side adaptive adapter and client-customized federated learning algorithm proposed by the present invention can be directly applied to different medical basic models.

[0061] The present invention can adaptively customize local models for different clients, tightly couple the efficient client adaptive adapter with the client customized federated learning algorithm, and effectively extract the common knowledge in the basic medical model to alleviate the instability in federated learning training. The present invention stabilizes the heterogeneous federated learning process, achieves optimal client personalization, and replaces the existing global compromise segmentation model.

[0062] The present invention proposes and applies a new parameter decomposition strategy to adaptively decompose adapter parameters into global units and local units, thereby achieving optimal client-side personalized federated learning and further stabilizing the heterogeneous federated training process.

[0063] The present invention adds a global-local decoupler (GLD) to each convolutional layer Conv and fully connected layer (FC) to adaptively decouple the adapter into global units and local units. The global-local decoupler GLD is specifically designed for the differences between the convolutional layer and the fully connected layer, and can play a consistent function.

[0064] The present invention divides and groups the parameter convolution kernels and multi-layer perceptron (MLP) neurons in the middle layer of the model, decouples at a more detailed level, and optimizes the flexibility and effectiveness of the model.

[0065] The present invention closely combines the parameter-efficient adapter with the federated learning algorithm to effectively extract the common knowledge in the basic medical model to stabilize heterogeneous federated learning. The present invention proposes a federated update algorithm that is compatible with the federated client customized adapter, which can decouple the parameters of the local model, perform differential updates based on the different information learned from the model parameters, and further stabilize the heterogeneous federated training process.

[0066] The present invention customizes the local model required by each client by decoupling the parameters of the local model and performing differentiated updates based on different information learned from the model parameters, thereby further stabilizing model training.

[0067] The present invention designs two different federated update strategies for customized federated learning, including binary federated update BFU and smooth federated update SFU. SFU is conceptually based on BFU and is a generalized version of BFU. Both strategies can effectively alleviate the problems of instability and slow convergence of training in heterogeneous federated learning, and the generalized strategy SFU achieves better performance.

[0068] The present invention solves the technical problems existing in the prior art that the heterogeneity of distributed medical data island clients is significant, resulting in imbalanced categories of medical data, a large deviation between the learned global compromise model and the individual optimality of each client, as well as low stability and slow convergence. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 This is a schematic diagram of basic steps of an image anatomical segmentation method based on a federated client customized adapter according to Embodiment 1 of the present invention;

[0070] Figure 2 This is a schematic diagram of the overall network framework adopted by the image anatomical segmentation method based on the federated client customized adapter according to Embodiment 1 of the present invention;

[0071] Figure 3This is a schematic diagram of specific steps for obtaining contribution scores of each channel in Example 1 of the present invention;

[0072] Figure 4 A schematic diagram of data flow processing of a customized federated learning algorithm for a client according to Embodiment 1 of the present invention;

[0073] Figure 5 This is a schematic diagram of specific steps for performing a federated update operation on a client according to Embodiment 1 of the present invention. DETAILED DESCRIPTION

[0074] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described in combination with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0075] Example 1

[0076] like Figure 1 As shown, the image anatomical segmentation method based on the federated client customized adapter provided by the present invention includes the following basic steps:

[0077] S1. Perform adapter learning on the medical basic model to complete the parameter-efficient fine-tuning (PEFT) operation. Insert the global-local decoupler GLD into the adapter module of the medical basic model to obtain a client-side adaptive adapter to adapt to the federated learning paradigm.

[0078] In this embodiment, the medical basic model is used as the backbone network backbone, which helps to improve the instability problem in federated updates. However, the medical basic model contains a large number of parameters, so the parameter efficient fine-tuning (PEFT) technology is required to reduce the number of parameters that need to be updated through adapter learning. Therefore, the present invention inserts a global-local decoupler into the traditional adapter module to make the adapter module more suitable for the federated learning paradigm.

[0079] See also Figure 2 The client-side adaptive adapter for medical image anatomical segmentation in the paper takes the medical basic model as the backbone, improves the traditional adapter module, and designs a client-customized federated learning algorithm to find the optimal personalized model for different clients under the federated learning paradigm.

[0080] The typical adapter of the medical basic model includes: a fully connected layer FC and a convolutional layer Conv. The present invention adds a global-local decoupler GLD to each convolutional layer Conv and fully connected layer FC to adaptively decouple the adapter into a global unit and a local unit. The global-local decoupler GLD is specifically designed for the difference between the convolutional layer Conv and the fully connected layer FC, so that they can play the same function.

[0081] S2. Perform channel contribution quantization operation on the convolution layer in the client adaptive adapter to obtain the contribution score of each channel to generate the score as the parameter of the medical basic model;

[0082] like Figure 3 As shown, in this embodiment, step S2 of obtaining the contribution score of each channel also includes the following specific steps:

[0083] S21, quantify the contribution score of each convolution channel unit unit ,in, u represents the convolution channel unit, k , j Indicates the serial number of the client;

[0084] In this embodiment, the convolutional channel units that contribute more to customer prediction contain more customer-specific information, while the convolutional channel units that contribute less contain more invariant global information. In this embodiment, each convolutional channel unit is called a unit, and the quantization k Convolutional channel units on the client u For j Contribution score of clients , used to indicate the degree to which the convolution channel unit unit contains the current client information.

[0085] In this embodiment, the following formula is used to determine k Convolutional channel units on the client u For j Contribution score of clients :

[0086] = ;

[0087] S22, using preset logic to process input features;

[0088] In this embodiment, the following formula is used to process each input feature :

[0089]

[0090] in, UIndicates the number of units, K Indicates the number of clients. The Broadcast function indicates the broadcast of shape matching elements for replication. is the weight of the global-local decoupler GLD, ⊙ is the element-wise multiplication;

[0091] S23, the contribution of each convolution channel unit to the client Predicted domain client probability with the output of the global-local decoupler GLD Do the outer product to get the contribution score of each channel of each client :

[0092] ;

[0093] S3. Set a client-customized federated learning algorithm. Under the federated learning paradigm, use the client-customized federated update algorithm to perform a federated update operation on the client based on the contribution score of each channel to obtain a suitable personalized model.

[0094] like Figure 4 and Figure 5 As shown, in this embodiment, step S3 of performing a federated update operation on the client further includes the following specific steps:

[0095] S31, executing the Binarization Federal Update (BFU) algorithm. Specifically, according to the contribution score of each channel, each convolution channel unit and the generation parameter of the convolution channel unit are binarized into global parameters and local parameters, and the convolution channel unit and its own generation parameters are used as channel parameter units to obtain local units and global units, and the global units are used for global update.

[0096] In this embodiment, for each channel contribution score generated for each client, each convolution channel unit and the parameters for generating the convolution channel unit can be binarized and divided into global parameters or local parameters. In this embodiment, such a convolution channel unit and the parameters generated by itself are called a channel parameter unit.

[0097] In this embodiment, the similarity of contribution of each channel parameter unit to different clients is determined:

[0098]

[0099] Determine the similarity of the contribution If the contribution degree is similar Less than threshold , it means that the channel parameter unit is a local unit, otherwise it is determined to be a global unit;

[0100] Generate a mask Mask for each convolution channel unit on each client to represent the local unit and the global unit:

[0101] ;

[0102] During global update, only global units are globally integrated, and local units do not participate in global integration. In this embodiment, the global units are updated as follows:

[0103]

[0104] S32. Execute the Smooth Federal Update (SFU) algorithm, using the contribution score of the current channel parameter unit to represent the score of each client learned by the convolutional channel unit. According to the score, weights are added during the update process of each local model parameter to obtain a suitable personalized model to update the client.

[0105] In this embodiment, the smooth federated update algorithm SFU is an improved algorithm of the binary federated update algorithm. Considering that different convolutional channel units may learn unique information of different clients at the same time, it may be more extreme to directly divide the channel parameter unit into local or global values. The contribution score of the channel parameter unit is used to represent the degree to which the convolutional channel unit learns each client, and according to the contribution score, each local model parameter is weighted during the update process. The following formula is used to express the first k Client's u Units are globally integrated:

[0106]

[0107] In this embodiment, in the smooth federated update algorithm SFU of this embodiment, the local model of each client will participate in the update of all clients.

[0108] S4. Train and test the applicable personalized model, and obtain the segmentation masks of each structure by processing the preset medical image using the applicable personalized model.

[0109] In this embodiment, the personalized model is applied to perform multiple iterations using the above method. Each iteration is first trained locally for no less than 2 local rounds, and then the federated update algorithm is used for update testing. Given a medical image, the personalized model is used to automatically process the image to obtain the segmentation masks of each structure.

[0110] The present invention is implemented using the PyTorch framework, applies deep learning technology to medical image segmentation, and adopts a new client-side adaptive adapter for medical image anatomical segmentation to achieve personalization of federated clients. Specifically, the present invention constructs a client-side adaptive adapter using parameter efficient fine-tuning technology to extract client-invariant general knowledge contained in the medical basic model, thereby stabilizing decentralized medical image segmentation. In addition, the present invention decomposes adapter parameters into common and separate components based on binary and probability. Constant components in the client undergo global federated updates, while individual components specific to the client are updated independently of the client.

[0111] In summary, the present invention designs a client-side adaptive adapter for medical anatomical segmentation. The present invention takes the basic medical model as the backbone, improves the traditional adapter module, and designs a client-customized federated learning algorithm, which can find the optimal applicable personalized model for different clients under the federated learning paradigm, and can achieve stable and personalized decentralized segmentation without sharing sensitive local data. The client-side adaptive adapter and client-customized federated learning algorithm proposed in the present invention can be directly applied to different basic medical models.

[0112] The present invention can adaptively customize local models for different clients, tightly couple the efficient client adaptive adapter with the client customized federated learning algorithm, and effectively extract the common knowledge in the basic medical model to alleviate the instability in federated learning training. The present invention stabilizes the heterogeneous federated learning process, achieves optimal client personalization, and replaces the existing global compromise segmentation model.

[0113] The present invention proposes and applies a new parameter decomposition strategy to adaptively decompose adapter parameters into global units and local units, thereby achieving optimal client-side personalized federated learning and further stabilizing the heterogeneous federated training process.

[0114] The present invention adds a global local decoupler (GLD) to each convolutional layer Conv and fully connected layer (FC) to adaptively decouple the adapter into global units and local units. The global local decoupler GLD is designed specifically for the differences between the convolutional layer and the fully connected layer, and can play a consistent function.

[0115] The present invention divides and groups the parameter convolution kernels and multi-layer perceptron (MLP) neurons in the middle layer of the model, decouples at a more detailed level, and optimizes the flexibility and effectiveness of the model.

[0116] The present invention closely combines the parameter-efficient adapter with the federated learning algorithm to effectively extract the common knowledge in the basic medical model to stabilize heterogeneous federated learning. The present invention proposes a federated update algorithm that is compatible with the federated client customized adapter, which can decouple the parameters of the local model, perform differential updates based on the different information learned from the model parameters, and further stabilize the heterogeneous federated training process.

[0117] The present invention customizes the local model required by each client by decoupling the parameters of the local model and performing differentiated updates based on different information learned from the model parameters, thereby further stabilizing model training.

[0118] The present invention designs two different federated update strategies for customized federated learning, including binary federated update BFU and smooth federated update SFU. SFU is conceptually based on BFU and is a generalized version of BFU. Both strategies can effectively alleviate the problems of instability and slow convergence of training in heterogeneous federated learning, and the generalized strategy SFU achieves better performance.

[0119] The present invention solves the technical problems existing in the prior art that the heterogeneity of distributed medical data island clients is significant, resulting in imbalanced categories of medical data, a large deviation between the learned global compromise model and the individual optimality of each client, as well as low stability and slow convergence.

[0120] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. Image anatomical segmentation method based on federated client custom adapter, It is characterized in that The method comprises: S1. Perform adapter learning on the medical basic model to complete the parameter efficient fine-tuning PEFT operation. Insert the global-local decoupler GLD into the adapter module of the medical basic model to obtain a client-side adaptive adapter to adapt to the federated learning paradigm. S2. Performing a channel contribution quantization operation on the convolution layer in the client adaptive adapter to obtain a contribution score of each channel as a parameter generation score of the medical basic model; Wherein, the S2 includes: S21, quantify the contribution score of each convolution channel unit unit ,in, u represents the convolution channel unit, k , j Represents the serial number of the client; wherein, the following formula is used to determine the k The convolution channel units on the client u For j The contribution score of the client : = In the formula, Indicates i The samples correspond to the convolution channel units in the global-local decoupler GLD. u Features, Indicates the convolution channel unit used in the global-local decoupler GLD to determine the u For k Parameters of the client classification contribution, Indicates the i The input features of samples Belong to j The predicted domain client probability of the client, Indicates the i The input features of samples Predicted to belong to j In the case of the client, d represents the global-local decoupler GLD; S22, using preset logic to process input features; wherein, using the following logic to process each of the input features : In the formula, U Indicates the number of units, K Indicates the number of clients, and the Broadcast function indicates the broadcast of shape matching elements. is the weight of the global-local decoupler GLD, d represents the global-local decoupler GLD, ⊙ is the element-wise multiplication, Indicates the contribution of each of the convolution channel units to the client, Represents the shape U×K The real number, i Indicates the input feature number; S23, the contribution of each convolution channel unit to the client The outer product is made with the predicted domain client probability output by the global-local decoupler GLD to obtain the channel contribution score of each client. ; Among them, using the following logic, using the first i The predicted domain client probability corresponding to the input features , the contribution of each of the convolutional channel units to the client Perform weighted processing to obtain the channel contribution score of each client : In the formula, represents the outer product, ， The Broadcast function represents the broadcast of shape matching elements. Indicates i The input features of samples Belong to j The predicted domain client probability of each of the clients; S3. Setting a client-customized federated learning algorithm, and using the client-customized federated update algorithm under the federated learning paradigm to perform a federated update operation on the client according to the contribution scores of each channel to obtain an applicable personalized model; S4. The applicable personalized model is trained and tested, and each structure segmentation mask is obtained by processing the preset medical image using the applicable personalized model.

2. The image anatomical segmentation method based on the federated client customized adapter according to claim 1, It is characterized in that In S1, the adapter module includes: a fully connected layer FC and a convolutional layer Conv, wherein the global-local decoupler GLD is added to each of the convolutional layer Conv and the fully connected layer FC to adaptively decouple the adapter module into a global unit and a local unit.

3. The image anatomical segmentation method based on the federated client customized adapter according to claim 1, It is characterized in that The S3 includes: S31, executing a binary federated update algorithm BFU, wherein, according to the contribution scores of each channel, each convolution channel unit and the generation parameters of the convolution channel unit are binarized and divided into global parameters and local parameters, and the convolution channel unit and its own generation parameters are used as channel parameter units, and local units and global units are obtained by processing them, and the global unit is used for global update; S32. Execute the smooth federated update algorithm SFU, use the contribution score of the current channel parameter unit to represent the score of each client learned by the convolution channel unit, and according to the score, weight the update process of each local model parameter to obtain the applicable personalized model to update the client.

4. The image anatomical segmentation method based on the federated client customized adapter according to claim 3, It is characterized in that In S31, the following logic is used to determine the similarity of the contribution of each channel parameter unit to different clients: : In the formula, Indicated in k The convolution channel units on the client u For j The contribution score of the client, express K The sum of the contribution scores of the clients, represents the logarithm of the number of clients with base 2, Represents the shape real number; Determine the similarity of the contribution , is it less than the threshold ; If yes, the current channel parameter unit is used as the local unit; otherwise, the channel parameter unit is used as the global unit; Generate a mask Mask for each of the convolution channel units on each of the clients to characterize the local unit and the global unit: in, Represents the convolution channel unit on the kth client u The mask Mask; When performing the global update, the global unit is globally integrated, wherein the global unit is updated using the following logic: In the formula, Represents the convolution channel unit u exist t+1 The corresponding weight parameter for the round W , Indicates k The convolution channel units on the client u In the t The weight parameters for rounds W .

5. The image anatomical segmentation method based on the federated client customized adapter according to claim 3, It is characterized in that In S32, the following logic is used to execute k The convolution channel unit of the client u Global integration operations: In the formula, Indicated in k The convolution channel units on the client u For j The contribution score of the client, Indicates j The client is t+1 In round 1, the convolution channel unit u The weight parameter W .

6. An image anatomical segmentation system based on a federated client customized adapter, used to execute the image anatomical segmentation method based on a federated client customized adapter according to any one of claims 1 to 5, It is characterized in that The system comprises: The client-side adaptive adapter building module is used to perform adapter learning on the medical basic model to complete the parameter efficient fine-tuning PEFT operation. In the adapter module of the medical basic model, the global-local decoupler GLD is inserted to obtain the client-side adaptive adapter to adapt to the federated learning paradigm; A contribution score acquisition module, used for performing a channel contribution quantization operation on the convolution layer in the client adaptive adapter to obtain the contribution score of each channel as a parameter generation score of the medical basic model, and the contribution score acquisition module is connected to the client adaptive adapter construction module; a federation update module, used for setting a client-customized federated learning algorithm, and using the client-customized federation update algorithm under the federated learning paradigm to perform a federation update operation on the client according to the contribution scores of each channel to obtain an applicable personalized model, wherein the federation update module is connected to the contribution score acquisition module; The image segmentation module is used to train and test the applicable personalized model, and obtain the segmentation masks of each structure by processing the preset medical image using the applicable personalized model.

Images