A multi-institutional joint image recognition model training method based on packet weighting

Abstract

The application discloses a kind of based on the image recognition model training method of multi-agency combination of grouping weighting, which can be applied to medical image recognition and other collaborative learning tasks with multiple participants.In order to alleviate the influence brought by medical image data distribution inconsistency, first, the application groups clients according to the similarity of client models and intra-group global models;For intra-group clients, the application calculates a personalized model for each client according to the similarity between the local models of intra-group clients.Compared with the prior art, the application can reduce the client computing overhead while providing better image recognition models for participants.

Description

Technical Field

[0001] This invention relates to a multi-institutional image recognition model training method based on group weighting, specifically a medical image recognition model training method based on group weighting for multiple medical institutions. This method can be used to jointly train medical image recognition models across multiple medical institutions. It leverages medical image data from different medical institutions with varying distributions to learn a personalized image recognition model for each participant, resulting in image recognition models that exhibit good performance on their local data distributions. Background Technology

[0002] Federated learning is a novel machine learning paradigm that enables multiple participants to collaboratively train machine learning models without uploading their local data. Federated learning reduces communication overhead and protects privacy. Furthermore, because participants train in parallel, it significantly improves training efficiency and alleviates the computational and storage pressure on central servers. Federated learning has already found widespread applications in fields requiring multi-party collaboration, such as mobile input prediction, cross-hospital medical image recognition, and cross-bank financial data prediction.

[0003] In tasks involving multiple medical institutions jointly training medical image recognition models, hospitals typically cannot upload their local medical image data to a centralized server for training due to patient privacy concerns. Therefore, federated learning is a viable solution in this scenario. However, using federated learning in this context still faces several challenges. Data heterogeneity is one of the most common. The distribution of patient image data across different hospitals is usually different, and this non-uniform data distribution can negatively impact federated learning training. Returning the same model to all hospitals may also lead to a mismatch between the model and its local data distribution. Therefore, many methods aim to return a personalized model more suitable for each participant's local data distribution. However, in some scenarios, the number of participants can be very large. An increase in the number of participants can exacerbate data inconsistency and significantly increase the overhead and difficulty of computing a personalized model for each participant. Summary of the Invention

[0004] Purpose of the Invention: Collaborative learning of medical image recognition models among multiple medical institutions is a typical application scenario of federated learning. Among various federated learning frameworks, personalized federated learning methods return a personalized model to each participant (medical institution), enabling it to better match local medical image data. Since the data distribution of different hospitals is usually different, personalized federated learning often achieves better results. However, existing personalized federated learning suffers from high computational and communication overhead for participants, and when the inconsistency in the distribution of participant data is significant, irrelevant data from other participants may negatively impact the computation of the personalized model. To address the problems and shortcomings of existing technologies, this invention provides a multi-institution collaborative image recognition model training method based on group weighting. First, this method dynamically assigns participants to groups similar to themselves to eliminate the influence of irrelevant data and reduce subsequent computational overhead. For clients within the same group, this method calculates personalized model weighting coefficients for them, thereby further improving the personalized effect.

[0005] Technical solution: A multi-agency joint image recognition model training method based on group weighting, involving client and server:

[0006] Clients: These are the participants in the federated learning process, i.e., the medical institutions. There are n clients participating in the federated learning. Each client possesses local medical image data and maintains a set of local models.

[0007] Server: The central coordinator of federated learning (high-performance server or computing cluster), responsible for aggregating models uploaded by clients; in the first phase, the server maintains K sets of global models, and in the second phase, the server computes a set of personalized models for each client during each round of communication; the server possesses a medical image-assisted dataset D for model similarity calculation. aux .

[0008] A training method for a multi-agency joint image recognition model based on group weighting includes the following stages:

[0009] Initialization phase: The server initializes K groups of global models; n clients are randomly assigned to one of the K groups;

[0010] During the client grouping phase, clients are dynamically grouped to eliminate the influence of irrelevant clients with different data distributions, thereby improving personalization effects and reducing the computational overhead of the personalization model.

[0011] 11) The server sends the global model of the client's group to the client;

[0012] 12) The client receives the global model, uses it as the local model, and performs E rounds of local model parameter updates using the local medical image dataset and stochastic gradient descent algorithm;

[0013] 13) After completing local training, the client uploads the updated local model to the server;

[0014] 14) After the server collects the local models of all clients, it updates the global model of each group by using a weighted average of the local models of the clients within the group based on the number of samples. The weighting factor is the proportion of samples owned by the client.

[0015] 15) The server uses n sets of client local models and K sets of global models to extract features on the medical image-assisted dataset, and uses CKA to calculate the similarity between the features extracted by each set of client local models and each set of global models;

[0016] 16) The server reassigns the client to the group corresponding to the global model that is most similar to it;

[0017] 17) Repeat steps 11)-16) until the specified number of communication rounds is reached.

[0018] In the intra-group personalization phase, the clients are grouped, and a personalization weighting coefficient is calculated for each client within the group. This is used to return a personalized model for each client, i.e., the client's medical image recognition model.

[0019] 21) The server sets the initial personalized model of each client to the global model of its group;

[0020] 22) The server sends its corresponding personalized model to each client;

[0021] 23) The client receives the personalized model, uses it as a local model, and performs E rounds of local model parameter updates using the local medical image dataset and stochastic gradient descent algorithm;

[0022] 24) After completing local training, the client uploads the updated local model to the server;

[0023] 25) After the server collects the local models of all n groups of clients, it uses the models of all clients to extract features on the medical image-assisted dataset, and uses CKA to calculate the pairwise similarity of the features extracted by the client models within each group.

[0024] 26) Normalize the pairwise similarities obtained in step 25), and use them as weighting coefficients to calculate a personalized weighted model for each client;

[0025] 27) Repeat steps 22)-26) until the specified number of communication rounds is reached; so that the client obtains a personalized medical image recognition model.

[0026] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the image recognition model training method based on group weighting and multi-mechanism collaboration as described above.

[0027] A computer-readable storage medium storing a computer program that performs the image recognition model training method based on group-weighted multi-agency joint processing as described above.

[0028] Beneficial Effects: Compared with existing federated learning methods applied to medical image recognition, the multi-institutional joint image recognition model training method based on group weighting proposed in this invention has better personalization performance and lower client-side computational overhead. To eliminate the influence of irrelevant clients and reduce the computational overhead of personalization weighting coefficients, this invention uses an iterative client grouping method. To further improve the model's personalization performance, this invention measures the pairwise similarity between client models and calculates weighting coefficients based on this similarity to compute a personalized model for each client. Furthermore, compared with other existing personalized federated learning techniques, the client in this method incurs less computational and communication overhead during the medical image recognition model training process, thereby reducing the computational and communication costs for all participants. Therefore, compared with existing technologies, this invention can return a high-performance personalized medical image recognition model for each participant without increasing the computational and communication overhead of the participants. Attached Figure Description

[0029] Figure 1 This is a flowchart illustrating the method of the present invention;

[0030] Figure 2 This is a flowchart illustrating the local model update phase on the client side in this invention. Detailed Implementation

[0031] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading the present invention, any modifications of the present invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0032] like Figure 1 , Figure 2 As shown, the training method for a multi-agency joint image recognition model based on group weighting includes the following steps (for simplicity, except for the step corresponding to model update, the superscript t corresponding to the current communication round number is omitted in the other steps):

[0033] (1) Server execution steps (e.g.) Figure 1 )

[0034] Step 100: For a federated learning task with n medical institutions participating as n clients, client i has a size of m i Local medical image dataset D i and maintain a set of local image recognition models w i Server initialization of K groups of global models {w (1) ,w (2) ,…,w (K)};

[0035] Step 101: The server sets the communication round number T1, the client locally updates the round number E, and initializes the current round number t = 0;

[0036] Step 102, the server randomly initializes the group to which the client belongs {c i ,i∈[n]}, that is, randomly assigning clients to a group; forming a total of K groups, each group containing a set of clients {C k ,k∈[K]}, group C k The corresponding server maintains the group's global model w (k) ;

[0037] Step 103: The server sends the corresponding global model within the group to the client based on the client's current group membership, i.e., it sends the global model to client i.

[0038] Step 104: The client receives the global model sent by the server, uses it as the local model, performs local training by executing steps 200-203, returns after completion, and executes step 105.

[0039] Step 105: The server receives all local models uploaded by clients. Update the group-wide global model for each group according to the following formula:

[0040]

[0041] in

[0042]

[0043] Step 106, the server then uses each client's local model w i In a medical image-assisted dataset D of size q aux Features are extracted to obtain the feature matrix Z. i The server uses the group-wide model w for each group. (k) In the medical image-assisted dataset D aux Features are extracted to obtain the feature matrix Z. (k) ;

[0044] Step 107: The server uses CKA to calculate the similarity between the features extracted by each client's local model and each group's global model. The calculation method is as follows:

[0045]

[0046]

[0047]

[0048]

[0049] in, L i Represents the characteristic matrix Z i The kernel matrix, L (k) Represents the characteristic matrix Z (k) The kernel matrix used in this method is a linear kernel;

[0050] Step 108: Using the similarity CKA(i,k) calculated in step 107, the client's group is reassigned, and the client is assigned to the group corresponding to the global model with the highest similarity.

[0051]

[0052] Update the current round number t = t + 1;

[0053] Step 109: If t > T1, proceed to step 110; otherwise, return to step 103.

[0054] Step 110: The server sets the communication round number T2 and initializes the current round number t = 0;

[0055] Step 111: The server uses CKA to calculate the pairwise similarity of the client's local model within each group, resulting in K similarity matrices {S}. k ,k∈[K]}, where S k (i,j) = CKA(i,j); CKA is calculated in the same way as step 107;

[0056] Step 112: The server calculates a personalized model for each client based on the intra-group similarity matrix of the client's group. The personalized model for client i is calculated as follows:

[0057]

[0058]

[0059] Step 113: The server sends the personalized model to the corresponding client;

[0060] Step 114: The client receives the personalized model sent by the server, uses it as the local model, performs local training in steps 200-203, returns after completion, and executes step 115.

[0061] Step 115: The server receives all local models uploaded by clients; updates the current round number t = t + 1.

[0062] Step 116: If t > T2, the algorithm ends, and the local model on the client is the final personalized medical image recognition model; otherwise, return to step 111.

[0063] (2) Client execution steps (e.g.) Figure 2 )

[0064] Step 200: The client receives the model sent by the server, and denots the model as w. rcv ;

[0065] Step 201, the client will transfer the local model w local Set as the received model: w local =w rcv ;

[0066] Step 202: The client uses the stochastic gradient descent algorithm to refine the local model w based on the local medical image dataset. local Perform E-round model parameter updates;

[0067] Step 203: The client uploads the updated model to the server.

[0068] Obviously, those skilled in the art should understand that the steps of the multi-agency joint image recognition model training method based on group weighting described in the above embodiments of the present invention can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by the computing device. Furthermore, in some cases, the steps shown or described can be performed in a different order than presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of the present invention are not limited to any particular hardware and software combination.

Claims

1. A method for training an image recognition model based on a multi-institutional joint with packet weighting, characterized in that, The method involves a client and a server in the implementation process; Client: A medical institution participating in federated learning is referred to as a client; each medical institution has local medical image data for training and testing; there are a total of 8 clients participating in federated learning; Server: central coordinator of federated learning, responsible for aggregating the models uploaded by clients; in the first phase, the server maintains a group global model, in the second phase, the server computes a set of personalized models for each client when communicating each round; the server has a medical image auxiliary dataset for model similarity calculation ​ The image recognition model training method comprises the following stages: Initialization phase: server initialization group global model; will randomly assign one client to one of the groups; The client grouping stage: 11) The server sends the global model of the group where the client is located to the client; 12) The client receives the global model, uses it as a local model, uses the local medical image dataset and a stochastic gradient descent algorithm to perform rounds of local model parameter updates; 13) After completing the local training, the client uploads the updated local model to the server; 14) After the server collects the local models of all clients, it uses the local models of the clients in each group to update the global model of the group by weighted average according to the number of samples; 15) server usage group client local model and group global model respectively extract features on the medical image auxiliary dataset, and use CKA to calculate the similarity between the features extracted by each group of client local models and the features extracted by each group of global models; 16) The server reassigns the client to the group corresponding to the global model most similar to the client; 17) Repeat steps 11) to 16) until the specified number of communication rounds is reached; The intra-group personalization stage: 21) The server sets the initial personalized model of each client to the global model of the group where the client is located; 22) The server sends the personalized model corresponding to each client to the client; 23) The client receives the personalized model, which is used as a new local model, using the local medical image dataset and a stochastic gradient descent algorithm for wheel local model parameter update; 24) After completing the local training, the client uploads the updated local model to the server; 25) The server collects all After the local models of the group clients are collected, the server extracts features on the medical image auxiliary dataset using the models of all clients, and calculates the pairwise similarity of the features extracted by each group of client models using CKA. 26) Normalize the similarity obtained in step 25) and use it as a weighting coefficient to calculate the personalized weighted model for each client; 27) Repeat steps 22) to 26) until the specified number of communication rounds is reached; so that the client obtains a personalized medical image recognition model. 2.The method of claim 1, wherein, In 14), the global model of each group is updated by using the local models of the clients in the group by weighted average according to the number of samples, and the weighting coefficient is the proportion of the samples owned by the client.

3. The method of claim 1 or 2, wherein, In step 15), the model uploaded using the CKA computing client on the medical image-assisted dataset is maintained by the server. The similarity of features extracted by the global model is calculated as follows: Step 106, the server uses the local model of each client again On a medical image auxiliary dataset of size The features are extracted on the medical image auxiliary dataset of size , and a feature matrix is obtained ; The server uses the in-group global model for each packet In medical image auxiliary data set The features are extracted to obtain a feature matrix ; In step 107, the server calculates the similarity between the features extracted by each client local model and each group global model using CKA, and the calculation method is as follows: wherein , denotes the kernel matrix of the feature matrix , denotes the kernel matrix of the feature matrix , the kernel used in the present method is a linear kernel.

4. The method of claim 3, wherein the method further comprises: In 16), the client is reassigned to a group, and the process is as follows: using the calculated similarity reassigning the group of the client, assigning the client to the group corresponding to the global model with which the client has the greatest similarity, i.e. 。 5. The method of claim 4, wherein the method further comprises: On the medical image auxiliary dataset, calculate the personalized weighting coefficient for each client based on CKA, and calculate the personalized weighted model, and the process is as follows: Step a, the server computes pairwise similarities of intra-group clients for each group using CKA, resulting in a similarity matrix where ; Step b, the server computes a personalized model for each client based on the similarity matrix within the group of the group the client is in, the client The personalized model is computed as follows: 。 6. A computer device, characterized by: The computer device comprises a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor executes the above computer program to realize the grouping weighted multi-institutional joint image recognition model training method according to any one of claims 1-5.

7. A computer-readable storage medium, characterized in that: The computer readable storage medium stores a computer program for executing the grouping weighted multi-institutional joint image recognition model training method according to any one of claims 1-5.

Images