Federal tumor classification method and device based on hierarchical aggregation and parameter personalization

Abstract

The invention discloses a federated tumor classification method and device based on hierarchical aggregation and parameter personalization, and the method comprises the steps: updating local models of N clients through employing an initialized global model when the first round of training begins; n user sides respectively utilize local data to train and send updated model parameters to the server; the server evaluates the contribution of each user side to the global model, and obtains an updated global model by adopting a hierarchical aggregation method; updating the local model of the user side according to the updated global model in the next round of training, and reserving parameters of a corresponding layer of which the parameter amplitude is greater than a gradient threshold value in the local model; and after the number of training rounds is set, training is stopped, and the personalized model of each user side is output. According to the personalized federal learning method for aggregating global knowledge, personalized parameters of different users are reserved, and more fair global model training is realized, so that the privacy disclosure problem and the data isomerism problem are solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of federated learning and artificial intelligence, and specifically relates to a federated tumor classification method and device based on hierarchical aggregation and parameter personalization. Background Art

[0002] In recent years, tumor diagnosis systems based on deep learning have been a hot topic in the field of medical image analysis. Due to the large amount of data required by deep learning models, it is difficult and time-consuming to train an effective tumor image diagnosis system from scratch. Collecting data directly from different data institutions can help increase data capacity and diversity, thereby improving model performance. However, using datasets from different user units to train deep learning models can cause problems such as user privacy leakage. Federated learning (FL) has emerged as a new distributed training framework that prioritizes data local storage and model communication, providing a new way to protect user privacy during the development of computer diagnostic systems. Traditional FL methods, such as FedAvg with data size-driven weighted aggregation, require a user end with large data capacity and cannot effectively solve the problem of data heterogeneity.

[0003] Disadvantages of existing technology: (1) Due to the particularity of medical images, professional doctors are often required to read the images and diagnose whether the tumor images are benign or malignant, which puts a heavy burden on doctors and reduces their efficiency.

[0004] (2) Traditional federated learning measures user contributions based on a fixed data ratio throughout the training process. This affects data fairness and makes it difficult to achieve personalized settings for each user, especially in highly heterogeneous data scenarios, such as the diagnosis of ultrasound, pathology, and other images of tumors such as breast cancer and colorectal cancer.

[0005] (3) Traditional federated learning uses fixed weights between all model layers, which makes it difficult to measure the functional connections between layers and their contributions to the global model, causing deviations in the global model and affecting the personalization process. Summary of the invention

[0006] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and to provide a federated tumor classification method based on hierarchical aggregation and parameter personalization. Through a personalized federated learning method that aggregates global knowledge, users can obtain customized and personalized models, thereby solving privacy leakage problems and data heterogeneity problems.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions: In one aspect of the present invention, a federated tumor classification method based on hierarchical aggregation and parameter personalization is provided, comprising the following steps: Set the global number of training rounds; At the beginning of the first round of training, the global model is initialized right N Update the local model of each user terminal; N Each client trains the local model using local data and sends the updated model parameters to the server; The server evaluates the contribution of each client to the global model, and obtains an updated global model based on the contribution by adopting a hierarchical aggregation method; At the beginning of the next round of training, the global model is updated N The local model of each user terminal is updated, wherein the parameters of the corresponding layer whose parameter amplitude in the local model is greater than the gradient threshold are retained; When the number of training rounds reaches the global number of training rounds, the training is stopped and the trained local model of each user end is output.

[0008] As a preferred technical solution, the N Each client uses local data to train the local model and sends the updated model parameters to the server. Specifically: Setting up in a personalized federated learning system N A user terminal, N A local model for setting gradient communication between clients; In the r +1 round of training begins, i The client uses the slave server r The global model obtained by round is expressed as ,in L Represents the total number of layers of the model; then use the i Local data of each client Training, local data It is expressed as: , ( i ∈ [1, … , N ]); Among them, m i For the i The sample size of each user terminal; After the training is completed i The model parameters that the client will update Send to the server; The training purpose of the personalized federated learning system is to select the most appropriate model parameters , the formula is as follows: ; in, For each user's local training target, the cross entropy loss function is used as the classification loss; represents the target of the loss function; Indicates i The model parameters of the user side; The server calculates the i The aggregation weight of each client.

[0009] As a preferred technical solution, the server evaluates the contribution of each user terminal to the global model, specifically: In the r +1 round of training, the server receives N The model parameters updated by the user end and the aggregation weights of the previous round are used for each layer of model parameters Calculate a consensus vector , the formula is as follows: ; in, l ∈ [1, … , L ] is the number of model layers, L Indicates the total number of layers of the model; The server calculates the consensus vector Except for i Vectors outside the client The cosine similarity between , the formula is as follows: ; in , indicating that except for the i Aggregation vector out of user-side gradients; Cosine similarity Standardize to get r +1 round i Client No. l Aggregate weights of layer model parameters , used to characterize the contribution of each user terminal to the global model, the formula is as follows: .

[0010] As a preferred technical solution, a hierarchical aggregation method is used based on the contribution to obtain an updated global model, specifically: Updated global model The model parameter update value of each layer of each user end and aggregation weight The calculation is as follows: .

[0011] As a preferred technical solution, the parameters of the corresponding layers whose parameter amplitude in the local model is greater than the gradient threshold are retained, specifically: The first i The first user l The updated value of the model parameters of the layer The threshold is preset as the gradient threshold T ; The model parameters are compared and if the amplitude of the model parameter is greater than T , the corresponding mask value is 1; In the next round of training, the update formula of the local model is as follows: ; in, For the l A vector composed of 0 and 1 with the same length as the layer parameters. When it is 1, it means that the local model parameters are used at the corresponding position, and when it is 0, it means that the global model parameters are used at the corresponding position.

[0012] As a preferred technical solution, the global model obtained through training and the local model obtained on each user end are used to perform verification and evaluation using the indicators of accuracy, F1 score and Matthews correlation coefficient in the verification set.

[0013] As a preferred technical solution, the local model of the user terminal is a neural network model for classifying and identifying tumor medical images.

[0014] Another aspect of the present invention further provides a federated tumor classification device based on hierarchical aggregation and parameter personalization, which is applied to the above-mentioned federated tumor classification method based on hierarchical aggregation and parameter personalization, and includes a user terminal and a server; The user terminal is used to: update the local model according to the global model provided by the server; train the local model using local data; and send the updated model parameters to the server; The server is used to: evaluate the contribution of each user terminal to the global model; obtain an updated global model based on the contribution by adopting a hierarchical aggregation method; and send the updated global model to the user terminal; In the process of updating the local model according to the global model provided by the server, the user end retains the parameters of the corresponding layers in the local model whose parameter amplitude is greater than the gradient threshold.

[0015] Another aspect of the present invention provides a computer program product, including computer instructions, which, when executed by a processor, implement the above-mentioned federated tumor classification method based on hierarchical aggregation and parameter personalization.

[0016] Another aspect of the present invention provides a storage medium storing a computer program, which, when executed by a processor, implements the above-mentioned federated tumor classification method based on hierarchical aggregation and parameter personalization.

[0017] Compared with the prior art, the present invention has the following advantages and beneficial effects: (1) The present invention uses Shapley value to evaluate and aggregate the contribution of each user end to the global model in a hierarchical manner, ensuring a fair contribution to the global model, thereby benefiting all users.

[0018] (2) The present invention retains the important parameters of the local model by comparing the parameter amplitude in the local model with the size of the gradient threshold, thereby solving the problem that the server obtains the consensus vector by using the contribution and gradient of the previous round of users, which affects the accuracy of the prediction. It improves the performance of the local model and realizes the personalization of federated learning.

[0019] (3) The hierarchical and parameter aggregation federated learning method provided by the present invention can be used to distinguish benign and malignant tumors, or be applied to multiple application scenarios such as pathological images, imaging images, and ultrasound images. It reduces the burden of professional doctors reading and diagnosing due to the particularity of medical images, and greatly improves efficiency.

[0020] (4) Extensive results from three user-side experiments demonstrate that the proposed hierarchical and parameter aggregation federated learning approach provides excellent algorithmic performance compared to other state-of-the-art federated learning methods and has potential for application in the field of medical image processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a flow chart of a federated tumor classification method based on hierarchical aggregation and parameter personalization according to an embodiment of the present invention; Figure 2 It is a structural diagram of a federated tumor classification device based on hierarchical aggregation and parameter personalization according to an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0023] Embodiment 1: like Figure 1As shown, this embodiment provides a federated tumor classification method based on hierarchical aggregation and parameter personalization, including the following steps: S1. Build a personalized federated learning system.

[0024] The personalized federated learning system is divided into global hierarchical aggregation and local training, and training is achieved through continuous communication between the client and the server. Therefore, in each global communication round, each client receives the global model aggregated by the server, and then performs local training. After local training, the model update value (gradient) is sent to the server for aggregation, and so on. The final output is a proprietary model for each client, which is the personalized model.

[0025] S1.1. Setting up in a personalized federated learning system N A user terminal, N Set up a local model for gradient communication between clients and set the number of global training rounds R .

[0026] S1.2. The objective function (optimization goal, training purpose) of the personalized federated learning system of the present invention is to select the most appropriate model parameters. , the formula model parameters are as follows: ; in, For each user's local training target, the cross entropy loss function is used as the classification loss; represents the target of the loss function; Indicates i The model parameters of the user side; The server calculates the i The aggregation weight of each client.

[0027] S2. Local training.

[0028] S2.1, in r +1 round of training starts, using the previous round (i.e. r The global model of the wheel For i The local model of each client is updated, that is, ,in L Indicates the total number of layers in the model (especially at the beginning of the first round of training, the initialized global model is used right N local model of each client).

[0029] S2.2, N Each user terminal uses local data to train the local model.i Local data of each client It is expressed as: ,(i ∈ [1, … , N ]); in: m i For the i The sample size of each user.

[0030] S2.3. After local training is completed i The model parameters that the client will update Send to the server.

[0031] S3, global hierarchical aggregation.

[0032] The server receives the updated model parameters of each user terminal and evaluates the contribution of each user terminal to the global model based on the idea of ​​Shapley value when participating in training or not, so as to determine the contribution of the user terminal to the global model, and then obtains the updated global model based on the contribution by adopting the hierarchical aggregation method.

[0033] In particular, Shapley value is a widely used user value evaluation theory, defined as: ; Assume that there is N client, represented by {1,…, i ,..., N}; S Indicates that different users form different alliances. N A subset of clients; U (⋅) represents a utility function, and in this embodiment, cosine similarity is used.

[0034] S3.1. The server evaluates the contribution of each client to the global model, specifically: Since it is difficult to enumerate all possible combinations of clients, it is necessary to calculate the client i Precision SV i Therefore, the hierarchical aggregation method is used for effective approximation. r +1 round of training, the server receives N The model parameters updated by the user end and the historical aggregation weights (i.e., the previous round, the first round, the second round, the third round, the fourth round, the fifth ... r Aggregate weight of round) Calculate a consensus vector , the formula is as follows: ; in,l ∈ [1, … , L ] is the number of model layers, L Indicates the total number of layers of the model; The server then measures each user i The deviation between the update direction of the vector and the consensus vector determines the deviation of each user i The contribution of the current round. This is calculated by calculating the consensus vector Except for i Vectors outside the client The cosine similarity between them is obtained. The calculation of embodies the idea of ​​Shapley value. and The greater the similarity, the i The contribution of each client to the model aggregation is smaller. Therefore, the difference from 1 is used as the contribution calculation. The calculation formula is as follows: ; in , indicating that except for the i Aggregation vector out of user-side gradients; In order to ensure that the total weight of the user side is 1, the cosine similarity needs to be Standardize and get r +1 round i Client No. l Aggregate weights of layer model parameters , used to characterize the contribution of each user terminal to the global model, the formula is as follows: .

[0035] S3.2. Based on the contribution, a hierarchical aggregation method is used to obtain an updated global model, specifically: Updated global model The model parameter update value of each layer of each user end (i.e. gradient) and the aggregation weight calculated based on the idea of ​​Shaley value The calculation is as follows: .

[0036] S4, second round of updates.

[0037] At the beginning of the next round of training, the client downloads the updated global model from the server ,Continue the subsequent federated learning training until the global training is completed. In this process, the server obtains the consensus vector by using the contribution and gradient of the previous round of users, which may affect the accuracy of the prediction.

[0038] In order to solve this problem, the present application further proposes a multi-parameter personalization method, which highly retains important local parameters during the local model training process.

[0039] Specifically, the i The first user l The updated value of the model parameters of the layer (i.e., the gradient) The threshold is preset as the gradient threshold T Then, the model parameters are compared and if its amplitude is greater than the gradient threshold T , the corresponding mask value is 1.

[0040] In the new round of user-side training data, the update calculation formula of the local model is as follows: ; Since the l-th layer parameter is a vector, It is a vector of 0 and 1 of the same length. When it is 1, it indicates that the local model parameters are used at the corresponding position. When it is 0, it indicates that the global model parameters are used at the corresponding position.

[0041] As a preferred technical solution, this embodiment dynamically sets the gradient threshold T As a value parameter for each layer personalization of 25%, that is, the mask ratio is 0.25.

[0042] S5. When the number of training rounds reaches the global number of training rounds, stop training and output the trained local model of each user end, that is, the personalized model.

[0043] S6. Model effect evaluation.

[0044] In order to compare the indicators of different methods, this application uses the accuracy under the best verification accuracy (ACC), F1 score and Matthews correlation coefficient (MCC) to evaluate the performance and effectiveness of different methods.

[0045] In particular, the local model of the user terminal described in this embodiment is a neural network model for classifying and identifying tumor medical images, which can be used to distinguish benign and malignant tumors, or applied to multiple application scenarios such as pathological images, imaging images, and ultrasound images.

[0046] Embodiment 2: This embodiment provides the federated tumor classification method based on hierarchical aggregation and parameter personalization described in the instantiation of Example 1, so that those skilled in the art can better understand the scheme of the present application and the advantages and beneficial effects achieved.

[0047] The steps to instantiate the solution are as follows: 1. This embodiment uses breast ultrasound image datasets from three centers, including: BUSI dataset (open source dataset, containing 674 images), GDPH dataset (from Guangdong Provincial People's Hospital, containing 846 images), SYSUCC dataset (from the Affiliated Cancer Hospital of Sun Yat-sen University, containing 1559 images), and regards each dataset as a single user end. Benign and malignant images of breast cancer tumors are selected respectively, and their images are divided into 160*160 image blocks. The distribution similarity of image blocks in each user end is low.

[0048] 2. In this embodiment, under the unified standard of image block pixel size of 32, ResNet-34 is used as the backbone network for deep learning, and Adam optimizer is used with a learning rate of 1e -4 , 50 rounds of global training and 1 round of local training are performed. The federated tumor classification method based on hierarchical aggregation and parameter personalization (hereinafter referred to as FedLP) of the present invention is compared with the FedAvg method and FedProx method in the prior art, as well as the most advanced FedPer method (arXiv 2019), FedALA (AAAI2023), FedPAC (ICLR 2023), and FedGH (MM 2023) baselines.

[0049] 3. In order to evaluate the indicators of different methods, this embodiment records the accuracy (ACC), F1 score and Matthews correlation coefficient (MCC) of the test set at the best verification accuracy. The training set, validation set and test set are divided in a ratio of 7:1:2. All results are cross-validated five times with different 7:1:2 partitions to ensure the fairness and consistency of the results. This embodiment is implemented in the environment of Pytorch 2.2.2 using NVIDIA RTX 3090 workstation. The results are shown in Table 1.

[0050] ; Table 1. Quantitative comparison results of this application with different methods, averaged by five-fold cross validation.

[0051] 4. In the comparison of all methods, FedLP of the present application achieved the highest overall performance accuracy of 0.8224 and MCC of 0.6015. It is better than FedPer, FedALA and FedPAC. FedPer and FedPAC split the model into feature representation and personalized classification heads. FedALA adopts a parameter personalization method similar to ours. However, it is affected by learning efficiency, and for users with a high data ratio, it is greatly affected by data distribution. On the contrary, from the user side, the method FedLP of the present application achieves excellent fair performance and personalized performance among user sides.

[0052] 5. This embodiment further studies the effect of mask ratio on personalization. In FedLP, personalization is achieved by manually keeping part of the locally updated parameters unchanged. The ratio of parameters that are not changed, i.e., mask ratio, is used to explore the effect of personalization performance. This embodiment sets the mask ratio to {0.25, 0.5, 0.75}. As shown in Table 2, the accuracy ranges from 0.8224 to 0.8399, which means that the higher the mask ratio, the better the personalization performance improvement.

[0053] ; Table 2. Impact of mask ratio on personalization performance. Results are averaged across all clients, averaged through 5-fold cross validation.

[0054] 6. This invention outperformed other methods in multi-center federated learning-based breast cancer ultrasound image diagnosis, demonstrating its great potential in medical image diagnosis.

[0055] Embodiment 3: In this embodiment, a federated tumor classification device based on hierarchical aggregation and parameter personalization is provided, the device comprising a user terminal and a server; The user terminal is used to: update the local model according to the global model provided by the server; train the local model using local data; and send the updated model parameters to the server; The server is used to: evaluate the contribution of each user terminal to the global model; obtain an updated global model based on the contribution by adopting a hierarchical aggregation method; and send the updated global model to the user terminal; In the process of updating the local model according to the global model provided by the server, the user end retains the parameters of the corresponding layers in the local model whose parameter amplitude is greater than the gradient threshold.

[0056] It should be noted here that the device provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure can be divided into different functional modules to complete all or part of the functions described above. The device can be applied to a federal tumor classification method based on hierarchical aggregation and parameter personalization in the above embodiment.

[0057] Embodiment 4: In this embodiment, a computer program product is provided, including computer instructions, which, when executed by a processor, implement a federated tumor classification method based on hierarchical aggregation and parameter personalization according to the above embodiment.

[0058] Embodiment 5: In this embodiment, a storage medium is provided, which stores a program. When the program is executed by a processor, a federated tumor classification method based on hierarchical aggregation and parameter personalization of the above embodiment is implemented.

[0059] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0060] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A federated tumor classification method based on hierarchical aggregation and parameter personalization, characterized in that: The steps include: Set the global number of training rounds; At the beginning of the first round of training, the global model is initialized right N Update the local model of each user terminal; N Each client trains the local model using local data and sends the updated model parameters to the server; The server evaluates the contribution of each client to the global model, and obtains an updated global model based on the contribution by adopting a hierarchical aggregation method; At the beginning of the next round of training, the global model is updated N The local model of each user terminal is updated, wherein the parameters of the corresponding layer whose parameter amplitude in the local model is greater than the gradient threshold are retained; When the number of training rounds reaches the global number of training rounds, the training is stopped and the trained local model of each user end is output.

2. The method for federated tumor classification based on hierarchical aggregation and parameter personalization according to claim 1, characterized in that: Said N Each client uses local data to train the local model and sends the updated model parameters to the server. Specifically: Setting up in a personalized federated learning system N A user terminal, N A local model for setting gradient communication between clients; In the r +1 round of training begins, i The client uses the slave server r The global model obtained by round is expressed as ,in L Represents the total number of layers of the model; then use the i Local data of each client Training, local data It is expressed as: ，( i ∈ [1, … , N ])； Among them, m i For the i The sample size of each user terminal; After the training is completed i The model parameters that the client will update Send to the server; The training purpose of the personalized federated learning system is to select the most appropriate model parameters , the formula is as follows: ； in, For each user's local training target, the cross entropy loss function is used as the classification loss; represents the target of the loss function; Indicates i The model parameters of the user side; The server calculates the i The aggregation weight of each client.

3. The federated tumor classification method based on hierarchical aggregation and parameter personalization according to claim 1, characterized in that: The server evaluates the contribution of each client to the global model, specifically: In the r +1 round of training, the server receives N The model parameters updated by the user end and the aggregation weights of the previous round are used for each layer of model parameters Calculate a consensus vector , the formula is as follows: ； in, l ∈ [1, … , L ] is the number of model layers, L Indicates the total number of layers of the model; The server calculates the consensus vector Except for i Vectors outside the client The cosine similarity between , the formula is as follows: ； in , indicating that except for the i Aggregation vector out of user-side gradients; Cosine similarity Standardize to get r +1 round i Client No. l Aggregate weights of layer model parameters , used to characterize the contribution of each user terminal to the global model, the formula is as follows: 。 4. The method for federated tumor classification based on hierarchical aggregation and parameter personalization according to claim 3, characterized in that: Based on the contribution, a hierarchical aggregation method is used to obtain an updated global model, specifically: Updated global model The model parameter update value of each layer of each user end and aggregation weight The calculation is as follows: 。 5. The federated tumor classification method based on hierarchical aggregation and parameter personalization according to claim 1, characterized in that: The parameters of the corresponding layer whose parameter amplitude in the local model is greater than the gradient threshold are specifically: The first i The first user l The updated value of the model parameters of the layer The threshold is preset as the gradient threshold T ; The model parameters are compared and if the amplitude of the model parameter is greater than T , the corresponding mask value is 1; In the next round of training, the update formula of the local model is as follows: ； in, For the l A vector composed of 0 and 1 with the same length as the layer parameters. When it is 1, it means that the local model parameters are used at the corresponding position, and when it is 0, it means that the global model parameters are used at the corresponding position.

6. The method for federated tumor classification based on hierarchical aggregation and parameter personalization according to claim 1, characterized in that: The global model obtained through training and the local model obtained on each user end are used for verification and evaluation using the accuracy, F1 score and Matthews correlation coefficient indicators in the validation set.

7. The method for federated tumor classification based on hierarchical aggregation and parameter personalization according to claim 1, characterized in that: The local model of the user terminal is a neural network model used to classify and identify tumor medical images.

8. A federated tumor classification device based on hierarchical aggregation and parameter personalization, characterized in that: A federated tumor classification method based on hierarchical aggregation and parameter personalization applied to any one of claims 1-7, comprising a user terminal and a server; The user terminal is used to: update the local model according to the global model provided by the server; train the local model using local data; and send the updated model parameters to the server; The server is used to: evaluate the contribution of each user terminal to the global model; obtain an updated global model based on the contribution by adopting a hierarchical aggregation method; and send the updated global model to the user terminal; In the process of updating the local model according to the global model provided by the server, the user end retains the parameters of the corresponding layers in the local model whose parameter amplitude is greater than the gradient threshold.

9. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the federated tumor classification method based on hierarchical aggregation and parameter personalization as described in any one of claims 1 to 7 is implemented.

10. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the federated tumor classification method based on hierarchical aggregation and parameter personalization according to any one of claims 1 to 7 is implemented.

Images