A medical image-oriented federated learning fairness improvement method and system

Abstract

This invention discloses a method for improving fairness in federated learning for medical images, comprising: the k-th client among K clients acquiring a PET-CT image pair consisting of a positron emission tomography (PET) image and its corresponding computed tomography (CT) image; preprocessing the PET-CT image pair to obtain a preprocessed PET-CT image pair; and inputting the PET-CT image pair into a multimodal neural image tumor segmentation model W. k The model is trained in the process, and a multimodal neural image tumor segmentation model W is obtained by calculating the loss function. k gradient g k Using gradient g k Multimodal neural image tumor segmentation model W k Parameters are updated to obtain the preliminarily trained multimodal neural image tumor segmentation model W for the k-th client. k This invention addresses the technical problem in the existing qFedAvg aggregation method where quantization affects the accuracy of model parameters, leading to greater performance differences between clients and thus reducing the fairness of federated learning.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer science and distributed learning, and more particularly, relates to a method and system for improving fairness of federated learning based on neural image tumor segmentation task. BACKGROUND

[0002] In the medical field, due to the privacy sensitivity of medical data and the difference in multi-center data distribution, there are some difficulties and challenges in the traditional centralized data training method. Therefore, federated learning (FL) technology emerges as the times require, which provides a new training paradigm that can share models between hospitals without sacrificing user privacy.

[0003] However, due to the different characteristics and distribution of medical data of different modalities, there are problems of data sparsity and distribution difference, which will lead to unfairness and difference in model performance during the federated learning process, so that the data of some modalities cannot be fully utilized, thereby affecting the accuracy and stability of the model; in addition, due to unreliable cooperation and fairness, the difficulty of fairness trade-off is also a challenge; again, in federated learning, each participant may have different data characteristics and quality, resulting in differences in data availability and reliability, which will affect the training and performance fairness of the model, making it very difficult to achieve cooperation and performance fairness among different modalities of medical data.

[0004] In order to solve the above problems, the fairness aggregation mechanism based on federated learning has been widely applied, and the existing fairness aggregation methods under federated learning mainly include two kinds, the first one is the aggregation method of federated averaging (FedAvg), which averages the model gradients of the clients and aggregates them to form a global model; the second one is the aggregation method of quantization-based federated averaging (qFedAvg), which quantizes the parameters before uploading the model gradients, converts the floating-point number parameters into a smaller number of bits, and then dequantizes the received quantized parameters on the server side to convert them back to floating-point number parameters, and then averages and aggregates the parameters to obtain the global model.

[0005] However, the above two federated learning aggregation methods have some defects that cannot be ignored:

[0006] Firstly, the FedAvg aggregation method directly averages the model gradients of each client without considering the uneven distribution of data between clients and the non-independent and identically distributed nature of data, which on the one hand leads to insufficient contribution of some participants to affect the global model, and on the other hand leads to overfitting to the training data of some clients, challenging the generalization ability of the model.

[0007] Secondly, the qFedAvg aggregation method quantizes the model gradient of the client and then uploads it to the server. Since quantization affects the accuracy of model parameters, the performance of some participants may decrease significantly during the updating process, leading to a larger performance difference between clients and reducing the fairness of federated learning. SUMMARY

[0008] In view of the above defects or improvement needs of the prior art, the present application provides a federated learning fairness improvement method and system for medical images, which aims to solve the technical problems that the existing FedAvg aggregation method directly averages the model gradients of each client without considering the uneven distribution of data between clients and the non-independent and identically distributed nature of data, which on the one hand leads to insufficient contribution of some participants to affect the global model, and on the other hand leads to overfitting to the training data of some clients, challenging the generalization ability of the model, and the existing qFedAvg aggregation method quantizes the model gradient of the client and then uploads it to the server. Since quantization affects the accuracy of model parameters, the performance of some participants may decrease significantly during the updating process, leading to a larger performance difference between clients and reducing the fairness of federated learning.

[0009] A federated learning fairness improvement method for medical images aims to solve the technical problems that the existing FedAvg aggregation method directly averages the model gradients of each client without considering the uneven distribution of data between clients and the non-independent and identically distributed nature of data, which on the one hand leads to insufficient contribution of some participants to affect the global model, and on the other hand leads to overfitting to the training data of some clients, challenging the generalization ability of the model, and the existing qFedAvg aggregation method quantizes the model gradient of the client and then uploads it to the server. Since quantization affects the accuracy of model parameters, the performance of some participants may decrease significantly during the updating process, leading to a larger performance difference between clients and reducing the fairness of federated learning.

[0010] To achieve the above object, according to one aspect of the present application, a medical image-oriented federated learning fairness improvement method is provided, comprising the following steps:

[0011] (1) The kth client in the K clients acquires a PET-CT image pair composed of a positron emission computed tomography (PET) image and a corresponding computed tomography (CT) image, and pre-processes the PET-CT image pair to obtain a pre-processed PET-CT image pair, wherein k [1, K];

[0012] (2) The kth client in the K clients inputs the pre-processed PET-CT image pair in step (1) into a multi-modal neural image tumor segmentation model W k for training, obtains the gradient g k of the multi-modal neural image tumor segmentation model W k by calculating a loss function, updates the parameters of the multi-modal neural image tumor segmentation model W k using the gradient g k , obtains a preliminarily trained multi-modal neural image tumor segmentation model W k of the kth client, and uploads the gradient g k of the preliminarily trained multi-modal neural image tumor segmentation model W k to a server.

[0013] (3) The server performs adversarial noise perturbation on the gradient g k from the kth client to obtain an optimized gradient g′ k of the kth client;

[0014] (4) The server performs fair aggregation on the optimized gradient g′ k of the kth client in step (3) to obtain an aggregated global model gradient g global , and distributes the aggregated global model gradient g global to the kth client.

[0015] (5) The kth client replaces the gradient g k of the preliminarily trained multi-modal neural image tumor segmentation model W k in step (2) with the global model gradient g global obtained in step (4);

[0016] (6) The kth client repeats the above steps (3) to (5) for m times, and the process ends.

[0017] Preferably, the PET image and the CT image acquired in step (1) are both in digital imaging and communications in medicine (DICOM) format.

[0018] Step (1) is first to obtain a PET-CT image pair composed of a PET image in DICOM format and its corresponding CT image, and then to convert the PET-CT image pair in DICOM format into a PET-CT image pair in Neuroimaging Informatics Technology Initiative (NIfTI) format using the SimpleITK library. Thereafter, only the data containing the PET image and the data containing the CT image in the PET-CT image pair in NIfTI format are removed. Then, all PET-CT slices in the PET-CT image pair are extracted, i.e. the PET-CT slices not containing a lesion label are deleted and the PET-CT slices containing a lesion label are retained. Finally, the extracted PET-CT slices are converted into a 512*512 png format PET-CT image pair as the preprocessed PET-CT image pair.

[0019] Preferably, the multi-modal neural image tumor segmentation model is a U-Net network model with a local self-attention mechanism, which includes an encoder module composed of two parallel first and second encoders, and an attention component part and a decoder sequentially connected with the encoder module.

[0020] The input of the first encoder is the PET image in the preprocessed PET-CT image pair. First, it is input into the first double convolution module to obtain a 32-channel feature map. Then, the 32-channel feature map obtained by the first double convolution module is input into the maximum pooling layer to halve the height and width of the feature map while keeping the number of channels unchanged, so as to realize spatial down-sampling and further obtain a 32-channel down-sampled feature map. Subsequently, the 32-channel down-sampled feature map is input into the second double convolution module to obtain a feature map with an output channel number of 64. Then, the feature map is input into the maximum pooling layer to further reduce the size of the feature map, and further obtain a 64-channel down-sampled feature map. Then, the 64-channel down-sampled feature map is input into the third double convolution module to obtain a feature map with an output channel number of 128. Subsequently, the feature map is input into the maximum pooling layer to halve the size of the feature map, and further obtain a 128-channel down-sampled feature map. Subsequently, the 128-channel feature map is input into the fourth double convolution module to obtain a 256-channel feature map. Finally, the feature map is input into the maximum pooling layer to halve the size of the feature map, and further obtain the final intermediate feature map wherein H, W and C represent the height, width and dimension of the intermediate feature map F pet of the PET image.

[0021] The structure of the second encoder is exactly the same as that of the first encoder, which will not be described here again. The input of the second encoder is the CT image corresponding to the PET image input into the first encoder, and the output of the second encoder is the intermediate feature map wherein H, W, C represent the intermediate feature maps F of the CT image respectively ct height, width and dimension.

[0022] The input of the attention component part is the intermediate feature map F of the PET image of the first encoder output pet and the intermediate feature map F of the CT image of the second encoder output ct which firstly performs the concatenation on the channel dimension on the two input intermediate feature maps to obtain a 512-channel feature map, and then inputs the feature map into a double convolution module with the input channel and the output channel being 512 to obtain a 512-channel PET-CT fusion intermediate feature map F pet After that, the PET-CT fusion intermediate feature map F pet-ct is fused in the channel domain, that is, average pooling and maximum value pooling are respectively performed on it to obtain two channel information, then the two channel information are input into a double convolution module to obtain two 1x1xC channel attention maps, after that, the two channel attention maps are superimposed and input into a Sigmoid activation function to obtain the final channel attention map Mc, and then the feature map fused in the channel domain is calculated according to the channel attention map Mc:

[0023]

[0024] wherein represents the matrix element corresponding multiplication operation; subsequently, the obtained feature map F' is respectively subjected to maximum value pooling and average pooling operation in the channel dimension to obtain two maximum pooling feature maps and average pooling feature maps After that, the obtained maximum pooling feature map and average pooling feature map are concatenated in the channel dimension, then a new convolution layer with the size of 7x7 and the output channel number of 1 is used to extract the two-dimensional feature map after the concatenation to generate a spatial attention map with the size same as the two-dimensional feature map after the concatenation and the channel number of 1; subsequently, the Sigmoid activation function is applied to the spatial attention map to map the value of the spatial attention map to 0 to 1 to obtain the final spatial attention map Ms, and then the feature map F'' fused in the spatial domain and the spatio-temporal domain is calculated according to the following formula:

[0025]

[0026] The input of the decoder is the feature map F" output by the attention component part after spatial domain fusion and space-time domain fusion. The feature map F" is first input into the fifth double convolution module to obtain a feature map with 512 output channels. Then the 512-channel feature map is input into a transpose convolution module with a size of 2*2 and a stride of 2. The spatial size of the feature map is restored to twice that of the feature map F", and the number of channels is reduced from 512 to 256, thereby obtaining a feature map with a higher resolution and 256 channels. Subsequently, the feature map is spliced with the 256-channel feature map output by the fourth double convolution module of the first encoder in the channel dimension to obtain a new 512-channel feature map F" 512 . Subsequently, the new feature map F" 512 is input into the sixth double convolution module to obtain a feature map with 256 output channels. Then the feature map is input into a transpose convolution module to restore the spatial size of the feature map to twice that of P" 512 , reduce the number of feature map channels from 256 to 128, and splice the 128-channel feature map with the 128-channel feature map output by the third double convolution module in the first encoder in the channel dimension to obtain a new feature map F" with 256 channels. 256 Then the new feature map F" 256 is input into the seventh double convolution module to obtain a feature map with 128 output channels. The feature map is input into a transpose convolution module to restore the spatial size of the feature map to twice that of F" 256 , and reduce the number of feature map channels from 128 to 64. Subsequently, the 64-channel feature map is spliced with the 64-channel feature map output by the second double convolution module of the first encoder in the channel dimension to obtain a new feature map F" with 128 channels. 128 Subsequently, the new feature map F" 128 is input into the eighth double convolution module to obtain a feature map with 64 output channels. The feature map is input into a transpose convolution module to restore the spatial size of the feature map to twice that of F" 128 , and reduce the number of feature map channels from 64 to 32. Subsequently, the 32-channel feature map is spliced with the 32-channel feature map output by the first double convolution module of the first encoder in the channel dimension to obtain a new feature map F" with 64 channels. 64 Subsequently, the new feature map F" 64 is input into the ninth double convolution module to obtain a feature map F" with 32 output channels. 32 The feature map F" 32The input is to the output convolutional layer, to reduce the number of channels of the feature map to a single channel through a 1x1 convolutional layer, and then to obtain a probability distribution matrix of the tumor position; then, the probability distribution matrix is mapped to the interval [0, 1] through a softmax function, the element values in the probability matrix greater than 0.5 are set to 1, and the element values in the probability matrix less than 0.5 are set to 0 (i.e. considering that the position is background, i.e. no lesion), thereby obtaining a binary matrix containing only 0 and 1; finally, the binary matrix is saved as a binary picture, which is the final lesion segmentation prediction result.

[0027] Preferably, step (2) comprises the following sub-steps:

[0028] (2-1) The kth client in the K clients divides the preprocessed PET-CT image pairs obtained in step (1) into a training set and a test set in a ratio of 8:2, where K is a natural number greater than or equal to 2, and k∈[1, K].

[0029] (2-2) The kth client in the K clients initializes the total number of iterations of the training process and the learning rate η of the multi-modal neural image tumor segmentation model;

[0030] (2-3) The kth client in the K clients inputs the PET image in each PET-CT image pair in the training set obtained in step (2-1) into the first encoder to obtain the intermediate feature map F of the PET image, and simultaneously inputs the CT image in each PET-CT image pair in the training set obtained in step (2-1) into the second encoder to obtain the intermediate feature map F of the CT image, where H, W and C represent the height, width and dimension of the intermediate feature map F pet of the PET image;

[0031] (2-4) The kth client in the K clients inputs the intermediate feature map F pet of the PET image and the intermediate feature map F ct of the CT image obtained in step (2-3) into the attention component part to obtain the feature map F'' fused in the channel domain and the spatial domain;

[0032] (2-5) The kth client in the K clients inputs the feature map F'' fused in the channel domain and the spatial domain obtained in step (2-4) into the decoder to obtain the final lesion segmentation prediction result F pred ;

[0033] (2-6) The kth client in the K clients obtains the lesion segmentation prediction result F predcalculating a loss function, using the loss function to calculate a gradient g k , using the calculated gradient g k updating parameters of the multi-modal neural image tumor segmentation model W k to obtain the kth client's preliminary trained multi-modal neural image tumor segmentation model W k ,

[0034] wherein the loss function in this step is a Dice Loss function, and its calculation formula is: wherein ∈ is 1*10 -8 , used to prevent the denominator from being 0. gt and F pred represent the true result and the predicted result respectively, and |F gt |+|F pred | represents the total sum of voxels in the true result and the predicted result.

[0035] Preferably, step (3) comprises the following sub-steps:

[0036] (3-1) the kth client in the K clients uploads the gradient g k of the multi-modal neural image tumor segmentation model W k preliminarily trained in step (2) to the server.

[0037] (3-2) the server calculates the perturbed gradient of the kth client according to the gradient uploaded by the kth client.

[0038] wherein the perturbed gradient is calculated using the following formula:

[0039]

[0040] and

[0041] wherein S t is a set composed of all clients, denotes the normalized gradient of the kth client, and ε denotes a scaling adversarial noise constant, which has a value equal to 1*10 -5 , denotes a sign function applied to the normalized gradient of the kth client.

[0042] (3-3) the server performs target optimization on the perturbed gradient of the kth client obtained in (3-2) to obtain the optimized gradient g′ k of the kth client.

[0043] The minimization objective of this step is: in This represents the original loss function for the k-th client. Let g represent the loss function after adding adversarial noise perturbation for the k-th client, and g represent the gradient.

[0044] Preferably, step (4) includes the following sub-steps:

[0045] (4-1) Set the counter t = 1;

[0046] (4-2) Determine whether t is equal to the preset federated learning rounds threshold. If it is, the process ends; otherwise, proceed to step (4-3).

[0047] (4-3) The server calculates the gradient g′ of the kth client obtained in step (3). k Calculate the dependency space fraction between the gradient of the k-th client and the global average gradient. Where g k，t This represents the gradient of the k-th client in the t-th round of federated learning training. This represents the average gradient of the K clients during the t-th round of federated learning training;

[0048] This step involves calculating the dependency space score using the following formula.

[0049]

[0050] Where var represents variance and cov represents covariance.

[0051] (4-5) The dependency space score between the gradient of the k-th client obtained in step (4-3) and the global average gradient. Perform a range transformation to obtain the transformed dependency space score w for the k-th client. k，Dep w k，Dep This represents the dependency space score after transformation for the k-th client;

[0052] This step involves using the following formula to obtain the dependency space score w of the kth client after transformation. k，Dep :

[0053]

[0054] Where exp represents the exponential function.

[0055] (4-5) The server calculates the optimized gradient g' of the k-th client obtained in step (3-3). k Calculate the orientation space score of the k-th client for the global model.

[0056] This step involves calculating the spatial fraction using the following formula.

[0057]

[0058] Here, cos represents cosine similarity.

[0059] (4-6) The server uses an exponential function to evaluate the orientation space score of the k-th client for the global model obtained in step (4-5). Normalization is performed to obtain the normalized orientation space score w for the k-th client. k，Dir ;

[0060] This step involves calculating the normalized orientation space fraction w using the following formula. k，Dir :

[0061]

[0062] (4-7) The server calculates the dependency space score w of the kth client obtained in step (4-4). k，Dep The normalized orientation space score w of the kth client obtained in steps (4-6) k，Dir Calculate the weight w′ of the k-th client in terms of collaboration fairness. k，Col .

[0063] This step involves calculating the weight w′ of the k-th client in terms of collaboration fairness using the following formula. k，Col :

[0064]

[0065] (4-8) The server calculates the gradient g′ of the kth client obtained in step (3). k Obtain the L2 norm scaling factor |g of the k-th client gradient. k，t |2.

[0066] This step involves calculating the L2 norm scaling factor |g| of the k-th client gradient using the following formula. k，t |2:

[0067]

[0068] Where γ is the regularization coefficient, (g′ k ) i It is the gradient g′ k As the i-th component of the vector.

[0069] (4-9) The server sums the L2 norm scaling factors of the gradients of all clients obtained in step (4-8) to obtain the sum of the L2 norm scaling factors of the gradients of all clients

[0070] (4-10) The server calculates the distance space weight coefficient w of the kth client according to the L2 norm scaling factor |g k，t of the kth client gradient obtained in step (4-8) and the sum of the L2 norm scaling factors of the gradients of all clients obtained in step (4-9) k，dis ;

[0071] wherein the distance space weight coefficient w of the kth client is calculated by the following formula k，dis :

[0072]

[0073] (4-11) The server calculates the loss space weight coefficient w of the kth client according to the loss function value l k.t of the kth client and the maximum loss function value max k∈[s k，t ]l k.t , among the K clients k，loss , wherein l k.t represents the loss function value of the kth client in the tth round of federated learning training, s k，t represents the set of all participating k clients in the tth round of federated learning training

[0074] wherein the loss space weight coefficient w of the kth client is calculated by the following formula k , loss:

[0075]

[0076] (4-12) The server calculates the weight w' of the kth client in performance fairness according to the distance space weight coefficient w k of the kth client obtained in step (4-10) and the loss space weight coefficient w k，loss of the kth client obtained in step (4-11) k，Per ;

[0077] wherein the weight w' of the kth client in performance fairness is calculated by the following formula k，Per :

[0078] w′ k，Per = w k，dis *w k，loss ​

[0079] (4-13) The server calculates the weight w′ of the k-th client in terms of cooperation fairness based on step (4-7). k，Col The weight w′ of the k-th client in terms of performance fairness obtained in step (4-12) k，Per Calculate the fair weight w for the k-th client. k ;

[0080] Specifically, this step involves calculating the fair weight w of the k-th client using the following formula. k :

[0081] w k =λ×w′ k，Per +(1-λ)×w′ k，Col

[0082] Wherein, λ represents the factor that balances the weights of collaboration and performance fairness, and its value ranges from 0 to 1;

[0083] (4-14) The server assigns the fair weight w to the k-th client obtained in step (4-13). k Perform weight balancing to obtain a fair weight for the k-th client.

[0084] (4-15) The server calculates the fair weight after balancing the k-th client based on the result obtained in step (4-14). Obtain the final normalized weights for the k-th client.

[0085] Specifically, this step uses the following formula to calculate the final normalized weight of the k-th client.

[0086]

[0087] (4-16) The server obtains the final normalized weight of the k-th client based on step (4-15). The gradient g′ of the k-th client after optimization in step (3-3) k Calculate the gradient g of the aggregated global model global ;

[0088] Specifically, this step involves calculating the aggregated global model gradient g using the following formula. global :

[0089]

[0090] (4-17) The server will aggregate the global model gradient g obtained in step (4-16). globalSend it to the kth client and set t = t + 1.

[0091] Preferably, step (4-14) includes the following sub-steps:

[0092] (4-14-1) The server assigns the fair weight w to the k-th client obtained in step (4-13). k Calculate the minimum weight w min w min The minimum fair weight for all K clients;

[0093] (4-14-2) The server obtains the minimum weight w based on step (4-14-1). min Calculate the equilibrium weight w equ ;

[0094] This step involves calculating the equilibrium weight w using the following formula. eau :

[0095]

[0096] Where T represents the pre-set minimum weight threshold, which ranges from 0 to 1, preferably 0.1;

[0097] (4-14-3) The server uses the equilibrium weight w obtained in step (4-14-2) equ and the fair weight w of the kth client obtained in step (4-13) k Calculate the weights after balancing the k-th client.

[0098] This step involves calculating the balanced weight of the k-th client using the following formula.

[0099]

[0100] Here, max(a, b) represents taking the maximum value of the two numbers a and b.

[0101] According to another aspect of the present invention, a federated learning fairness improvement system for medical images is provided, comprising:

[0102] The first module, which is set in the kth client among the K clients, is used to acquire PET-CT image pairs consisting of positron emission tomography (PET) images and their corresponding computed tomography (CT) images, and to preprocess the PET-CT image pairs to obtain preprocessed PET-CT image pairs.

[0103] The second module, located on the kth client out of the K clients, is used by the kth client to input the PET-CT image pairs preprocessed by the first module into the multimodal neural image tumor segmentation model W. k The model is trained in the process, and a multimodal neural image tumor segmentation model W is obtained by calculating the loss function. k gradient g k Using gradient g k Multimodal neural image tumor segmentation model W k Parameters are updated to obtain the preliminarily trained multimodal neural image tumor segmentation model W for the k-th client. k The pre-trained multimodal neural image tumor segmentation model W k gradient g k Upload to the server, where k∈[1,K].

[0104] The third module, located on the server, is used to process the gradient g from the k-th client. k Perform anti-noise perturbation to obtain the optimized gradient g′ of the k-th client. k ;

[0105] The fourth module, located on the server, is used to calculate the gradient g′ for the k-th client after optimization by the third module. k Perform fair aggregation to obtain the aggregated global model gradient g. global And aggregate the global model gradient g global It is sent to the kth client.

[0106] The fifth module, located on the kth of the K clients, is used to apply the global model gradient g obtained from the fourth module. global Replace the pre-trained multimodal neural image tumor segmentation model W in the second module. k gradient g k ;

[0107] The sixth module, located on the kth client out of the K clients, is used to repeat the above modules three through five m times, after which the process ends.

[0108] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects:

[0109] 1. The application adopts step (3), which adds designed micro adversarial noise to the gradient uploaded by the client before polymerization, flattens the loss space, and standardizes the gradient size, thereby accelerating the convergence speed of the model, improving the accuracy of the model, and reducing the performance decline caused by uneven data distribution. The generalization ability of the model on different data distributions is enhanced. Therefore, the technical problem that the model of some participants may have significant performance decline during the updating process, the performance difference between clients becomes larger, and the fairness of federated learning is reduced caused by the quantization of parameters in the existing qFedAvg method is solved.

[0110] 2. The application adopts step (4), which introduces a method for cross-modal medical data collaboration and performance fairness, provides a hierarchical correction method for solving the collaboration fairness and performance fairness problem in distributed learning, improves the model performance while ensuring the fairness of federated learning. Therefore, the technical problem that the existing FedAvg method cannot improve the performance fairness and collaboration fairness of federated learning, and causes overfitting to the training data of some clients, which challenges the generalization ability of the model is solved. BRIEF DESCRIPTION OF DRAWINGS

[0111] Figure 1 is the overall flowchart of the federated learning fairness improvement method for medical images of the application;

[0112] Figure 2 is the detailed flowchart of step (3) in the method of the application.

[0113] Figure 3 is the detailed flowchart of step (4) in the method of the application. DETAILED DESCRIPTION

[0114] In order to make the purpose, technical scheme and advantages of the application clearer, the application will be further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the application and do not limit the application. In addition, the technical features involved in each embodiment of the application described below can be combined with each other as long as they do not conflict with each other.

[0115] The basic method of the application is to use two main measures to improve the collaboration fairness, performance fairness and fairness trade-off of the federated learning under the multi-modal neural image tumor segmentation task.

[0116] The first aspect is to design an adversarial noise disturbance method. In order to solve the problem of model construction under high heterogeneous data in distributed method, the method adds designed micro adversarial noise to the gradient uploaded by the user before aggregation, reduces the gap between the difference of user gradient difference, realizes the enhancement of generalization, and also improves the accuracy of the model.

[0117] The second aspect is a method for designing optimization of cross-modal medical data collaboration and performance fairness under federated learning, realizing reliable collaboration and performance fairness enhancement, optimizing fairness trade-off, and improving training of federated models with cross-modal medical data. The collaboration and performance fairness are realized through the design of hierarchical contribution correction, which corrects the contribution of clients in collaboration and performance in a hierarchical manner based on different correlation spaces from cross-modal data. In addition, the adaptive balance mechanism of fairness is combined to balance the collaboration and performance fairness by adjusting the adjusted weight and incorporating the fairness balance mechanism, realizing the priority ranking based on real demand, and ensuring that the weight of all clients remains above the minimum fairness threshold.

[0118] The purpose of the present application is to improve the training of federated models with cross-modal medical data, and realize collaboration and performance fairness through the design of hierarchical contribution correction.

[0119] The present application has the advantages of solving the problem of model construction of high-heterogeneous data under multi-modal medical data fusion and distributed method, and realizing collaboration fairness promotion, performance fairness enhancement and adaptive fairness balance in the federated learning scene.

[0120] In order to make the purpose, technical method and advantages of the present application more clear and explicit, the present application will be further described in detail below in combination with the drawings.

[0121] As shown in Figure 1 The present application provides a federated learning fairness improvement method and system for medical images, including the following steps:

[0122] (1) The kth client in K clients acquires a PET-CT image pair composed of a positron emission computed tomography (PET) image and its corresponding computed tomography (CT) image, and pre-processes the PET-CT image pair to obtain a pre-processed PET-CT image pair, wherein K is any natural number, and k∈[1,K];

[0123] Specifically, the PET image and the CT image obtained in this step are both in the Digital Imaging and Communications in Medicine (DICOM) format, this step first obtains a PET-CT image pair composed of a PET image in the DICOM format and a corresponding CT image, then converts the PET-CT image pair in the DICOM format into a PET-CT image pair in the Neuroimaging Informatics Technology Initiative (NIfTI) format using the SimpleITK library, and then removes the data containing only the PET image and the data containing only the CT image in the PET-CT image pair in the NIfTI format. Then, all PET-CT slices in the PET-CT image pair are extracted, that is, the PET-CT slices not containing the lesion label are deleted, and the PET-CT slices containing the lesion label are retained, and finally, the extracted PET-CT slices are converted into a png format PET-CT image pair of 512*512, as the preprocessed PET-CT image pair.

[0124] (2) The kth client in the K clients inputs the preprocessed PET-CT image pair in step (1) into the multi-modal neural image tumor segmentation model W k for training, obtains the gradient g k of the multi-modal neural image tumor segmentation model W k by calculating a loss function, updates the parameters of the multi-modal neural image tumor segmentation model W k using the gradient g k to obtain the kth client's preliminarily trained multi-modal neural image tumor segmentation model W k , and uploads the gradient g k of the preliminarily trained multi-modal neural image tumor segmentation model W k to the server.

[0125] The multi-modal neural image tumor segmentation model of the application is a U-Net network model with a local self-attention mechanism, which includes an encoder module composed of two parallel first encoders and second encoders, and an attention component part and a decoder connected in sequence with the encoder module.

[0126] The input of the first encoder is the PET image in the pre-processed PET-CT image pair, which is first input into the first double convolution module to obtain a 32-channel feature map (the number of channels of the input feature map of the double convolution module is 3, and the number of channels of the output feature map is 32, and the structure of the double convolution module is as follows: first, a 3x3 convolution kernel is used to fill 1 pixel around the image, and the input feature map is convolved, the convolution kernel has in_ch input channels and out_ch output channels, then batch normalization is performed, and then the ReLU function is activated, then another 3x3 convolution kernel is used to fill 1 pixel around the image to perform convolution operation, and the output channel number is converted from out_ch to the same out_ch, and then batch normalization and ReLU function activation are performed again, and the double convolution module is constructed); then, the 32-channel feature map obtained by the first double convolution module is input into the maximum pooling layer (the pooling window size is 2x2, and the step is 2), the height and width of the feature map are halved, and the channel number remains unchanged, spatial down-sampling is realized, and then a 32-channel down-sampled feature map is obtained; then, the 32-channel down-sampled feature map is input into the second double convolution module to obtain a feature map with an output channel number of 64, and then the feature map is input into the maximum pooling layer to further reduce the size of the feature map, and then a 64-channel down-sampled feature map is obtained; then, the 64-channel down-sampled feature map is input into the third double convolution module to obtain a feature map with an output channel number of 128; then, the feature map is input into the maximum pooling layer to halve the size of the feature map, and then a 128-channel down-sampled feature map is obtained; then, the 128-channel feature map is input into the fourth double convolution module to obtain a 256-channel feature map; finally, the feature map is input into the maximum pooling layer to halve the size of the feature map, and then the final intermediate feature map is obtained wherein H, W, C represent the height, width and dimension of the intermediate feature map Fpet of the PET image, respectively.

[0127] The structure of the second encoder is exactly the same as that of the first encoder, and will not be described here again, and the input of the second encoder is the CT image corresponding to the PET image input into the first encoder, and the output of the second encoder is the intermediate feature map Fct of the CT image wherein H, W, C represent the height, width and dimension of the intermediate feature map Fct of the CT image, respectively. ct

[0128] The input of the attention component part is the intermediate feature map Fpet of the PET image output by the first encoder and the intermediate feature map Fct of the CT image output by the second encoder pet ct ​​which firstly concatenates the two input intermediate feature maps in the channel dimension to obtain a 512-channel feature map, and then inputs the feature map into a double convolution module with an input channel and an output channel of 512 to obtain a 512-channel PET-CT fusion intermediate feature map F pet- After that, the PET-CT fusion intermediate feature map F pet-ct is fused in the channel domain, that is, average pooling and maximum value pooling are respectively performed on it to obtain two channel information, and then the two channel information are input into a double convolution module (firstly, a 1x1 convolution layer is used to reduce the input channel number, and then a ReLU activation function is used, followed by another 1x1 convolution layer to restore the channel number to the original value, and finally the convolution output is input into a Sigmoid activation function, which maps the convolution output value to 0 to 1), to obtain two 1x1xC channel attention maps, after that, the two channel attention maps are superimposed and input into a Sigmoid activation function to obtain the final channel attention map Mc, and then the feature map fused in the channel domain is calculated according to the channel attention map Mc:

[0129]

[0130] wherein represents a matrix element corresponding multiplication operation; subsequently, the obtained feature map F′ is respectively subjected to maximum value pooling and average pooling operations in the channel dimension to obtain two maximum pooling feature maps and average pooling feature maps After that, the obtained maximum pooling feature map and average pooling feature map are concatenated in the channel dimension, and then a new convolution layer with a size of 7x7 and an output channel number of 1 is used to extract the two-dimensional feature map after concatenation to generate a spatial attention map with a channel number of 1 and the same size as the two-dimensional feature map after concatenation; subsequently, a Sigmoid activation function is applied to the spatial attention map to map the value of the spatial attention map to 0 to 1 to obtain a final spatial attention map Ms, and then the feature map F″ fused in the spatial domain and the spatio-temporal domain is calculated according to the following formula:

[0131]

[0132] The input of the decoder is the feature map F" output by the attention component part after spatial domain fusion and space-time domain fusion. First, the feature map F" is input into the fifth double convolution module (the structure of the fifth double convolution module is completely consistent with that of the double convolution module in the encoder), to obtain a feature map with 512 output channels. Then, the 512-channel feature map is input into a transpose convolution module with a size of 2*2 and a stride of 2, to restore the spatial size of the feature map to twice that of the feature map F" and reduce the number of channels from 512 to 256, thereby obtaining a feature map with a higher resolution and 256 channels. Subsequently, the feature map is spliced with the 256-channel feature map output by the fourth double convolution module of the first encoder in the channel dimension, to obtain a new feature map F" with 512 channels 512 . Subsequently, the new feature map F" 512 is input into the sixth double convolution module, to obtain a feature map with 256 output channels. Then, the feature map is input into a transpose convolution module, to restore the spatial size of the feature map to twice that of F" 512 , reduce the number of channels of the feature map from 256 to 128, and splice the 128-channel feature map with the 128-channel feature map output by the third double convolution module in the first encoder in the channel dimension, to obtain a new feature map F" with 256 channels 256 . Then, the new feature map F" 256 is input into the seventh double convolution module, to obtain a feature map with 128 output channels. The feature map is input into a transpose convolution module, to restore the spatial size of the feature map to twice that of F" 256 , reduce the number of channels of the feature map from 128 to 64, and subsequently splice the 64-channel feature map with the 64-channel feature map output by the second double convolution module of the first encoder in the channel dimension, to obtain a new feature map F" with 128 channels 128 . Subsequently, the new feature map F" 128 is input into the eighth double convolution module, to obtain a feature map with 64 output channels. The feature map is input into a transpose convolution module, to restore the spatial size of the feature map to twice that of F" 128 , reduce the number of channels of the feature map from 64 to 32, and subsequently splice the 32-channel feature map with the 32-channel feature map output by the first double convolution module of the first encoder in the channel dimension, to obtain a new feature map F" with 64 channels 64 . Subsequently, the new feature map F" 64 is input into the ninth double convolution module, to obtain a feature map F" with 32 output channels 32 . The feature map F" 32The input is input to the output convolutional layer to reduce the number of channels of the feature map to a single channel through a 1x1 convolutional layer, thereby obtaining a probability distribution matrix of the tumor position; then, the probability distribution matrix is mapped to the [0, 1] interval through a softmax function, the element values in the probability matrix greater than 0.5 are set to 1 (i.e. considering that the position is foreground, i.e. lesion), and the element values in the probability matrix less than 0.5 are set to 0 (i.e. considering that the position is background, i.e. no lesion), thereby obtaining a binary matrix containing only 0 and 1; finally, the binary matrix is saved as a binary picture, which is the final lesion segmentation prediction result.

[0133] Specifically, step (2) comprises the following sub-steps:

[0134] (2-1) The kth client in the K clients divides the preprocessed PET-CT image pairs obtained in step (1) into a training set and a test set according to a ratio of 8:2, wherein K is a natural number greater than or equal to 2, and k ∈ [1, K].

[0135] The process of dividing the data into a training set and a test set in this step is as follows: first, all the preprocessed PET-CT image pairs are randomly shuffled, and then all the shuffled PET-CT image pairs are evenly divided into 5 parts, and any 1 part is taken as the test set and the remaining 4 parts are taken as the training set.

[0136] (2-2) The kth client in the K clients initializes the total number of iterations of the training process and the learning rate η of the multi-modal neural image tumor segmentation model;

[0137] Specifically, the total number of iterations in this step is in the range of 100 to 300, and is preferably 200.

[0138] The value of the learning rate η in this step is in the range of 0 to 1, and is preferably 0.001.

[0139] (2-3) The kth client in the K clients inputs the PET image in each PET-CT image pair in the training set obtained in step (2-1) into the first encoder to obtain the intermediate feature map F Meanwhile, the CT image in each PET-CT image pair in the training set obtained in step (2-1) is input into the second encoder to obtain the intermediate feature map F Wherein H, W and C represent the height, width and dimension of the intermediate feature map F pet of the PET image, respectively.

[0140] Specifically, the PET image in the PET-CT image pair is first input into a first double convolution module to obtain a 32-channel feature map (the number of channels of the input feature map of the double convolution module is 3, and the number of channels of the output feature map is 32. The structure of the double convolution module is as follows: first, a 3x3 convolution kernel is used to perform convolution calculation on the input feature map, the convolution kernel has in_ch input channels and out_ch output channels, then batch normalization is performed, and then the ReLU function is activated, and then another 3x3 convolution kernel is used to perform convolution operation, the output channel number is converted from out_ch to the same out_ch, and then batch normalization and ReLU function activation are performed again, and thus the double convolution module is constructed); then, the 32-channel feature map obtained by the first double convolution module is input into a maximum pooling layer (the pooling window size is 2x2, and the stride is 2), the height and width of the feature map are halved, the number of channels remains unchanged, spatial down-sampling is realized, and a 32-channel down-sampled feature map is obtained; then, the 32-channel down-sampled feature map is input into a second double convolution module to obtain a feature map with an output channel number of 64, and then the feature map is input into a maximum pooling layer to further reduce the size of the feature map, and a 64-channel down-sampled feature map is obtained; then, the 64-channel down-sampled feature map is input into a third double convolution module to obtain a feature map with an output channel number of 128; then, the feature map is input into a maximum pooling layer to halve the size of the feature map, and a 128-channel down-sampled feature map is obtained; then, the 128-channel feature map is input into a fourth double convolution module to obtain a 256-channel feature map; finally, the feature map is input into a maximum pooling layer to halve the size of the feature map, and a final intermediate feature map is obtained

[0141] In this step, the same operation as the PET image is performed on the CT image to obtain the intermediate feature map of the CT image and

[0142] (2-4) The kth client in the K clients inputs the intermediate feature map F of the PET image obtained in step (2-3) pet and the intermediate feature map F of the CT image ct into the attention component part to obtain a feature map F" fused in the spatial domain and the spatio-temporal domain;

[0143] Specifically, this step splices the two input intermediate feature maps in the channel dimension to obtain a 512-channel feature map, and then inputs the feature map into a double convolution module with an input channel and an output channel of 512 to obtain a 512-channel PET-CT fusion intermediate feature map F pet-c, and then the PET-CT fused intermediate feature map F pet-ct is fused in the channel domain, that is, average pooling and maximum pooling are respectively performed on it to obtain two channel information, and then the two channel information is input into a double convolution module (firstly, a 1x1 convolution layer is used to reduce the number of input channels, and then a ReLU activation function is used, followed by another 1x1 convolution layer to restore the number of channels to the original value, and finally the convolution output is input into a Sigmoid activation function, which maps the numerical value of the convolution output to 0 to 1), to obtain two 1x1xC channel attention maps, and then the two channel attention maps are superimposed and input into a Sigmoid activation function to obtain the final channel attention map Mc, and then the feature map fused in the channel domain is calculated according to the channel attention map Mc:

[0144]

[0145] wherein represents a matrix element corresponding multiplication operation; subsequently, the obtained feature map F' is respectively subjected to maximum pooling and average pooling operations in the channel dimension to obtain two maximum pooling feature maps and average pooling feature maps , respectively. Then, the obtained maximum pooling feature map and average pooling feature map are spliced in the channel dimension, and then a new convolution layer with a size of 7x7 and an output channel number of 1 is used to extract the spliced two-dimensional feature map to generate a spatial attention map with a channel number of 1 and a size same as that of the spliced two-dimensional feature map; subsequently, a Sigmoid activation function is applied to the spatial attention map to map the value of the spatial attention map to 0 to 1 to obtain a final spatial attention map Ms, and then the feature map F'' fused in the spatial domain and the spatio-temporal domain is calculated according to the following formula:

[0146]

[0147] The advantages of the above steps (2-3) and (2-4) are that the local self-attention mechanism method combines the fusion of the channel domain and the spatial domain, and can extract and utilize the key information of the multi-modal data. Learning the feature importance can enhance the attention and utilization of key features by the model, and improve the performance of the model.

[0148] (2-5) The kth client in the K clients inputs the feature map F'' fused in the channel domain and the spatial domain obtained in step (2-4) into the decoder to obtain the final lesion segmentation prediction result F pred .

[0149] Specifically, the feature map F" is first input into the fifth double convolution module (the structure of the fifth double convolution module is exactly the same as that of the double convolution module in the encoder), to obtain a feature map with an output channel number of 512, and then the 512-channel feature map is input into a transpose convolution module with a size of 2*2 and a stride of 2, to restore the spatial size of the feature map to twice that of the feature map F", and reduce the channel number from 512 to 256, thereby obtaining a feature map with a higher resolution and a channel number of 256; subsequently, the feature map is spliced with the 256-channel feature map output by the fourth double convolution module of the first encoder in the channel dimension, to obtain a new 512-channel feature map F" 512 , and then the new feature map F" 512 is input into the sixth double convolution module, to obtain a feature map with an output channel number of 256, and then the feature map is input into a transpose convolution module, to restore the spatial size of the feature map to twice that of F" 512 , reduce the channel number of the feature map from 256 to 128, and splice the 128-channel feature map with the 128-channel feature map output by the third double convolution module in the first encoder in the channel dimension, to obtain a new feature map F" with a channel number of 256 256 , and then the new feature map F" 256 is input into the seventh double convolution module, to obtain a feature map with an output channel number of 128, and then the feature map is input into a transpose convolution module, to restore the spatial size of the feature map to twice that of F" 256 , reduce the channel number of the feature map from 128 to 64, and subsequently, splice the 64-channel feature map with the 64-channel feature map output by the second double convolution module of the first encoder in the channel dimension, to obtain a new feature map F" with a channel number of 128 128 ; subsequently, the new feature map F" 128 is input into the eighth double convolution module, to obtain a feature map with an output channel number of 64, and then the feature map is input into a transpose convolution module, to restore the spatial size of the feature map to twice that of F" 128 , reduce the channel number of the feature map from 64 to 32, and subsequently, splice the 32-channel feature map with the 32-channel feature map output by the first double convolution module of the first encoder in the channel dimension, to obtain a new feature map F" with a channel number of 64 64 ; subsequently, the new feature map F" 64 is input into the ninth double convolution module, to obtain a feature map F" with an output channel number of 32 32 , and the feature map F" 32The input is input to the output convolutional layer to reduce the number of channels of the feature map to a single channel through a 1x1 convolutional layer, and then a probability distribution matrix of the tumor position is obtained; then, the probability distribution matrix is mapped to the interval [0, 1] through a sofimax function, the element values in the probability matrix greater than 0.5 are set to 1 (i.e. considering that the position is foreground, i.e. lesion), and the element values in the probability matrix less than 0.5 are set to 0 (i.e. considering that the position is background, i.e. no lesion), thereby obtaining a binary matrix containing only 0 and 1; finally, the binary matrix is saved as a binary picture, which is the final lesion segmentation prediction result.

[0150] (2-6) The kth client in the K clients obtains the lesion segmentation prediction result F according to step (2-5) pred The loss function is calculated, and the gradient g is calculated using the loss function k The calculated gradient g is used k The parameters of the multi-modal neural image tumor segmentation model W are updated, thereby obtaining the multi-modal neural image tumor segmentation model W k preliminarily trained by the kth client k ,

[0151] Specifically, the loss function in this step is the DiceLoss function, and the calculation formula is:

[0152] Wherein, ∈ is 1*10 -8 , used to prevent the denominator from being 0. F gt and F pred represent the true result and the prediction result respectively, and |F gt |+|F pred | represents the sum of the voxels in the true result and the prediction result.

[0153] (3) The server performs adversarial noise disturbance on the gradient g k from the kth client to obtain the optimized gradient g' of the kth client k ;

[0154] As shown in Figure 2 , specifically, step (3) includes the following sub-steps:

[0155] (3-1) The kth client in the K clients uploads the gradient g k of the multi-modal neural image tumor segmentation model W k preliminarily trained by step (2) to the server.

[0156] (3-2) The server calculates the disturbance gradient g of the kth client according to the gradient uploaded by the kth client.

[0157] Specifically, the step is to calculate the perturbation gradient using the following formula

[0158]

[0159] And

[0160] Where S t is a set of all clients, denotes the normalized gradient of the kth client, and ε denotes the scaling adversarial noise constant, which is equal to 1*10 -5 , denotes the sign function applied to the normalized gradient of the kth client.

[0161] (3-3) The server optimizes the perturbation gradient of the kth client obtained in (3-2) to obtain the optimized gradient g′ k of the kth client.

[0162] Specifically, the minimization target of this step is: Where denotes the original loss function of the kth client, denotes the loss function after adding adversarial noise perturbation to the kth client, and g denotes the gradient.

[0163] The advantage of this step (3) is that by adding a small adversarial noise designed to the gradient uploaded by the client before aggregation, the loss space is flattened, and the gradient size is standardized, which realizes the acceleration of the convergence speed of the model, improves the accuracy of the model, and reduces the performance degradation caused by uneven data distribution. Enhance the generalization ability of the model on different data distributions.

[0164] (4) The server performs fair aggregation on the gradient g′ k of the kth client optimized in step (3) to obtain the aggregated global model gradient g global , and delivers the aggregated global model gradient g global to the kth client.

[0165] As Figure 3 shown, specifically, this step includes the following sub-steps:

[0166] (4-1) Set the counter t = 1;

[0167] (4-2) judge whether t is equal to the preset federated learning round threshold value, if yes, the process ends, otherwise go to step (4-3);

[0168] Specifically, the preset federated learning round threshold value in this step is in the range of 100 to 300, preferably 200.

[0169] (4-3) the server calculates the gradient g of the kth client according to the kth client obtained in step (3) k Calculate the dependence space score between the gradient of the kth client and the global average gradient Where g k，t represents the gradient of the kth client in the tth round of federated learning training, represents the average gradient of the K clients in the tth round of federated learning training.

[0170] Specifically, the dependence space score is calculated by the following formula in this step

[0171]

[0172] Where var represents variance, and cov represents covariance.

[0173] (4-5) the server converts the dependence space score between the gradient of the kth client obtained in step (4-3) and the global average gradient To get the converted dependence space score w of the kth client k，Dep , w k，Dep represents the converted dependence space score of the kth client.

[0174] Specifically, the converted dependence space score w of the kth client is obtained by the following formula in this step k，Dep :

[0175]

[0176] Where exp represents the exponential function.

[0177] (4-5) the server calculates the direction space score of the kth client to the global model according to the optimized gradient g of the kth client obtained in step (3-3) k

[0178] Specifically, the direction space score is calculated by the following formula in this step

[0179]

[0180] Where cos represents the cosine similarity.​

[0181] (4-6) The server calculates the directional space score of the kth client for the global model using an exponential function on the directional space score of the kth client obtained in step (4-5) k，Dir ;

[0182] Specifically, the normalized directional space score w is calculated using the following formula k，Dir :

[0183]

[0184] (4-7) The server calculates the weight w' of the kth client in terms of cooperation fairness according to the dependency space score w of the kth client obtained in step (4-4) k，Dep and the normalized directional space score w of the kth client obtained in step (4-6) k，Dir k，Col .

[0185] Specifically, the weight w' of the kth client in terms of cooperation fairness is calculated using the following formula k，Col :

[0186]

[0187] (4-8) The server obtains the L2 norm scaling factor |g k |2 of the gradient of the kth client according to the gradient g' of the kth client obtained in step (3) k，t .

[0188] Specifically, the L2 norm scaling factor |g k，t |2 of the gradient of the kth client is calculated using the following formula

[0189]

[0190] where γ is a regularization coefficient, (g' is the i-th component of the gradient g' as a vector. k i k

[0191] (4-9) The server sums the L2 norm scaling factors of the gradients of all clients obtained in step (4-8) to obtain the sum of the L2 norm scaling factors of the gradients of all clients

[0192]

[0193] ​​​​​(4-10) The server calculates the distance space weight coefficient w of the kth client according to the L2 norm scaling factor |g k，t of the gradient of the kth client obtained in step (4-8) The server calculates the sum of the L2 norm scaling factors of the gradients of all clients obtained in step (4-9) k，dis ;

[0194] Specifically, the distance space weight coefficient w of the kth client is calculated by the following formula k，dis :

[0195]

[0196] (4-11) The server calculates the loss space weight coefficient w of the kth client according to the loss function value l k.t of the kth client and the maximum loss function value max k∈[s k，t ]l k.t of the K clients k，loss , wherein l k.t represents the loss function value of the kth client in the tth round of federated learning training, s k，t represents the set of all participating k clients in the tth round of federated learning training

[0197] Specifically, the loss space weight coefficient w of the kth client is calculated by the following formula k , loss:

[0198]

[0199] (4-12) The server calculates the weight w' of the kth client in performance fairness according to the distance space weight coefficient w k of the kth client obtained in step (4-10) and the loss space weight coefficient w k，loss of the kth client obtained in step (4-11) k，Per ;

[0200] Specifically, the weight w' of the kth client in performance fairness is calculated by the following formula k，Per :

[0201] w′ k，Per = w k，dis *w k，1oss

[0202] (4-13) The server calculates the weight w' of the kth client in cooperation fairness according to the weight w k，ColThe weight w' of the kth client in terms of performance fairness obtained in step (4-12) k，Per The weight w of the kth client in terms of performance fairness obtained in step (4-12) k ;

[0203] Specifically, the weight w of the kth client in terms of performance fairness is calculated by the following formula k :

[0204] W k = λ × w' + (1 - λ) × w' k，Per k，Col

[0205] wherein λ represents a factor balancing the weights of cooperation and performance fairness, and its value range is 0 to 1, preferably 0.5.

[0206] (4-14) The server performs weight balancing on the weight w of the kth client in terms of performance fairness obtained in step (4-13) k to obtain the balanced weight w of the kth client in terms of performance fairness

[0207] Specifically, the step includes the following sub-steps:

[0208] (4-14-1) The server calculates the minimum weight W min from the weight w of the kth client in terms of performance fairness obtained in step (4-13) k , wherein w min is the minimum value of the weights of all K clients in terms of performance fairness;

[0209] (4-14-2) The server calculates the balanced weight w equ from the minimum weight w min obtained in step (4-14-1);

[0210] Specifically, the balanced weight w is calculated by the following formula equ :

[0211]

[0212] wherein T represents a pre-set minimum weight threshold, and its value range is 0 to 1, preferably 0.1.

[0213] (4-14-3) The server calculates the balanced weight w of the kth client from the balanced weight w equ obtained in step (4-14-2) and the weight w k of the kth client in terms of performance fairness obtained in step (4-13)

[0214] Specifically, the final normalized weight of the kth client is calculated by the following formula

[0215]

[0216] Wherein, max(a, b) represents the maximum value of a and b.

[0217] (4-15) The server obtains the final normalized weight of the kth client according to the fair weight of the kth client obtained in step (4-14)

[0218] Specifically, the final normalized weight of the kth client is calculated by the following formula

[0219]

[0220] (4-16) The server obtains the final normalized weight of the kth client according to the final normalized weight of the kth client obtained in step (4-15) And the gradient g' of the kth client after optimization in step (3-3) k Calculate the aggregated global model gradient g global ;

[0221] Specifically, the final normalized weight of the kth client is calculated by the following formula global :

[0222]

[0223] (4-17) The server sends the aggregated global model gradient g global Obtained in step (4-16) to the kth client, and sets t = t + 1.

[0224] The advantage of this step (4) is that the present application provides a hierarchical correction method for solving the problem of cooperation fairness and performance fairness in distributed learning, which has the advantages of balancing fairness, improving model performance and ensuring cooperation fairness.

[0225] (5) The kth client replaces the gradient g global of the multi-modal neural image tumor segmentation model W k preliminarily trained in step (2) with the global model gradient g k obtained in step (4).

[0226] (6) The kth client repeats the above steps (3) to (5) for m times, and the process ends.

[0227] ​In particular, the value range in this step is 100 to 300, preferably 200.

[0228] For the problem of achieving collaboration and performance fairness between different modal medical data, the present application adopts a dependence space method to promote collaboration fairness by amplifying the correlation between clients with smaller rewards. The server utilizes the dependence space to measure the correlation between the client and the global model, and amplifies the contribution of clients with moderate dependence. At the same time, the contribution between the client and the global model is corrected in the direction space to ensure that the contribution of the client will not be destroyed. Through multi-level analysis, the server can ensure that the collaboration metric promotes the contribution of those clients with unique correlation with the global model, while maintaining the contribution of other clients, thereby achieving collaboration fairness promotion.

[0229] In order to enhance performance fairness, the server amplifies the weight of the client with poor performance due to significant differences in data distribution in the loss space. In the distance space, the server normalizes the distance between the client and the global model to reduce the bias to the client with large data volume and large gradient update. In this way, the server can treat the performance contribution of different clients more fairly and improve the overall performance fairness.

[0230] The present application adopts an adaptive fairness balancing method to automatically adjust the balance between collaboration and performance fairness without manual adjustment. The server introduces a revised weight adjustment mechanism to prioritize the desired fairness level. At the same time, the server also introduces a fairness balancing mechanism to ensure that the adjustment of the weight achieves an appropriate balance between collaboration and performance fairness. This adaptive fairness balancing method can dynamically adjust according to the actual situation, thereby better meeting the fairness requirements in the collaborative learning scenario.

[0231] In view of the above defects or improvement needs of the prior art, the present application provides a medical image-oriented federated learning fairness improvement method and system. The method of the present application can solve the problem of model construction for high-heterogeneous multi-modal medical data under distributed methods. It can achieve collaboration fairness promotion, performance fairness enhancement and adaptive fairness balancing in the collaborative learning scenario. These technical improvements can improve the performance and accuracy of the model and provide more reliable tools and methods for data analysis and diagnosis in the medical field. It can be widely used in various collaborative learning tasks to improve the fairness and performance of model training, while providing better experience and results for participants.

[0232] Those skilled in the art will readily understand that the above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A medical image-oriented federated learning fairness improvement method, characterized in that, The method comprises the following steps: (1) the kth client in K clients acquires a PET-CT image pair composed of a positron emission computed tomography (PET) image and a corresponding computed tomography (CT) image, and pre-processes the PET-CT image pair to obtain a pre-processed PET-CT image pair, wherein K is any natural number, and k [1, K]; (2) the kth client in the K clients inputs the PET-CT image pair preprocessed in step (1) into the multimodal neural image tumor segmentation model W k for training, obtains the gradient g k of the multimodal neural image tumor segmentation model W k by calculating a loss function, updates the parameters of the multimodal neural image tumor segmentation model W k using the gradient g k to obtain the kth client's preliminarily trained multimodal neural image tumor segmentation model W k , and uploads the gradient g k of the preliminarily trained multimodal neural image tumor segmentation model W k to the server; the multimodal neural image tumor segmentation model is a U-Net network model with local self-attention mechanism, which includes an encoder module composed of two parallel first encoders and second encoders, and an attention component part and a decoder connected in sequence with the encoder module; the server performs adversarial noise perturbation on the gradient g from the kth client to obtain an optimized gradient g' of the kth client k the server performs adversarial noise perturbation on the gradient g from the kth client to obtain an optimized gradient g' of the kth client k ; step (3) comprises the following sub-steps: (3-1) The kth client in the K clients uploads the gradient g k of the preliminarily trained multi-modal neural image tumor segmentation model W k of step (2) to the server; (3-2) The server calculates the perturbed gradient of the kth client according to the gradient uploaded by the kth client wherein the present step is to calculate the perturbation gradient using the following formula and also where S t is a set of all clients, denotes the normalized gradient of the kth client, and ε denotes a scaling adversarial noise constant, which takes a value equal to 1 * 10 -5 , denotes a sign function applied to the normalized gradient of the kth client . (3-3) The server obtains the perturbed gradient of the kth client from (3-2) Perform target optimization to obtain the optimized gradient g' of the kth client k ; (3) where the minimization objective of this step is: where represents the loss function of the kth client originally, represents the loss function of the kth client after adding the adversarial noise disturbance, and g represents the gradient; (4) The server optimizes the gradient g′ of the kth client in step (3) k Fair aggregation is performed to obtain the aggregated global model gradient g global The aggregated global model gradient g global is sent to the kth client. (5) the kth client uses the global model gradient g obtained in step (4) global replacing the multimodal neural image tumor segmentation model W preliminarily trained in step (2) k gradient g k ; (6) the kth client repeats steps (3) to (5) m times, and the process ends.

2. The medical image-oriented federated learning fairness improvement method according to claim 1, wherein the PET image and the CT image acquired in step (1) are in a digital imaging and communications in medicine (DICOM) format; In step (1), first, a PET-CT image pair in a DICOM format is acquired, and then a SimpleITK library is used to convert the PET-CT image pair in the DICOM format into a PET-CT image pair in a neuroimaging informatics technology initiative (NIfTI) format, after which, only the data containing the PET image and the data containing the CT image in the PET-CT image pair in the NIfTI format are removed; then, all PET-CT slices in the PET-CT image pair are extracted, that is, the PET-CT slices without a lesion label are deleted, and the PET-CT slices with a lesion label are retained, and finally, the extracted PET-CT slices are converted into a 512*512 png format PET-CT image pair as the pre-processed PET-CT image pair.

3. The medical image-oriented federated learning fairness improvement method according to claim 1 or 2, wherein the input of the first encoder is the PET image in the pre-processed PET-CT image pair, which is first input into a first double convolution module to obtain a 32-channel feature map, then the 32-channel feature map obtained by the first double convolution module is input into a maximum pooling layer to halve the height and width of the feature map while keeping the number of channels unchanged, thereby realizing spatial down-sampling and obtaining a 32-channel down-sampled feature map; subsequently, the 32-channel down-sampled feature map is input into a second double convolution module to obtain a feature map with an output channel number of 64, and then the feature map is input into the maximum pooling layer to further reduce the size of the feature map, thereby obtaining a 64-channel down-sampled feature map; then, the 64-channel down-sampled feature map is input into a third double convolution module to obtain a feature map with an output channel number of 128; Subsequently, the feature map is input into the maximum pooling layer to halve the size of the feature map, thereby obtaining a 128-channel down-sampled feature map; Subsequently, a Sigmoid activation function is applied to the spatial attention map to map the value of the spatial attention map to between 0 and 1 to obtain a final spatial attention map Ms, and then the feature map F'' output after spatial domain fusion and spatio-temporal domain fusion is calculated according to the following formula: Step (2) comprises the following sub-steps: Subsequently, the 128-channel feature map is input into a fourth double convolution module to obtain a 256-channel feature map; finally, the feature map is input into a maximum pooling layer to reduce the size of the feature map by half, thereby obtaining a final intermediate feature map where H, W, C represent the height, width and dimension of the intermediate feature map F of the PET image respectively pet ​ The structure of the second encoder is completely same as that of the first encoder, and details are not repeated here. An input of the second encoder is a CT image corresponding to the PET image input to the first encoder, and an output of the second encoder is an intermediate feature map of the CT image wherein H, W, C represent a height, a width and a dimension of the intermediate feature map F ct of the CT image respectively. The input of the attention component part is the intermediate feature map F of the PET image output by the first encoder pet and the intermediate feature map F of the CT image output by the second encoder ct which first performs concatenation on the two input intermediate feature maps in the channel dimension to obtain a 512-channel feature map, and then inputs the feature map into a double convolution module with an input channel and an output channel of 512 to obtain a 512-channel PET-CT fusion intermediate feature map F pet-ct After that, the PET-CT fusion intermediate feature map F pet-ct is fused in the channel domain, that is, average pooling and maximum value pooling are respectively performed on it to obtain two channel information, then the two channel information are input into a double convolution module to obtain two 1x1xC channel attention maps, after that, the two channel attention maps are superimposed and input into a Sigmoid activation function to obtain a final channel attention map Mc, and then a feature map fused in the channel domain is calculated according to the channel attention map Mc: wherein denotes a matrix element-wise multiplication operation; Subsequently, maximum pooling and average pooling operations are respectively performed on the obtained feature map F' in the channel dimension to obtain two maximum-pooled feature maps and average-pooled feature maps Then, the obtained maximum-pooled feature map and average-pooled feature map are spliced in the channel dimension, and then a new convolutional layer with a size of 7x7 and an output channel number of 1 is used to extract the spliced two-dimensional feature map to generate a spatial attention map with a size same as that of the spliced two-dimensional feature map and a channel number of 1. ​ The input of the decoder is the feature map F" output by the attention component part after spatial domain fusion and space-time domain fusion. The feature map F" is first input into the fifth double convolution module to obtain a feature map with 512 output channels. Then the 512-channel feature map is input into a transpose convolution module with a size of 2*2 and a stride of 2. The spatial size of the feature map is restored to twice that of the feature map F", and the number of channels is reduced from 512 to 256, thereby obtaining a feature map with a higher resolution and 256 channels. Subsequently, the feature map is spliced with the 256-channel feature map output by the fourth double convolution module of the first encoder in the channel dimension to obtain a new 512-channel feature map F" 512 . Subsequently, the new feature map F" 512 is input into the sixth double convolution module to obtain a feature map with 256 output channels. Then the feature map is input into a transpose convolution module to restore the spatial size of the feature map to twice that of F" 512 , reduce the number of feature map channels from 256 to 128, and splice the 128-channel feature map with the 128-channel feature map output by the third double convolution module in the first encoder in the channel dimension to obtain a new feature map F" with 256 channels. 256 Then the new feature map F" 256 is input into the seventh double convolution module to obtain a feature map with 128 output channels. The feature map is input into a transpose convolution module to restore the spatial size of the feature map to twice that of F" 256 , reduce the number of feature map channels from 128 to 64, and subsequently splice the 64-channel feature map with the 64-channel feature map output by the second double convolution module of the first encoder in the channel dimension to obtain a new feature map F" with 128 channels. 128 Subsequently, the new feature map F" 128 is input into the eighth double convolution module to obtain a feature map with 64 output channels. The feature map is input into a transpose convolution module to restore the spatial size of the feature map to twice that of F" 128 , reduce the number of feature map channels from 64 to 32, and subsequently splice the 32-channel feature map with the 32-channel feature map output by the first double convolution module of the first encoder in the channel dimension to obtain a new feature map F" with 64 channels. 64 Subsequently, the new feature map F" 64 is input into the ninth double convolution module to obtain a feature map F" with 32 output channels. 32 The feature map F" 32 The input is input to an output convolutional layer to reduce the number of channels of the feature map to a single channel through a 1x1 convolutional layer, and then a probability distribution matrix of the tumor position is obtained; subsequently, the probability distribution matrix is mapped to the interval [0, 1] through a softmax function, the element values greater than 0.5 in the probability matrix are set to 1, and the element values less than 0.5 in the probability matrix are set to 0, that is, the position is considered as background, that is, no lesion, so as to obtain a binary matrix containing only 0 and 1; finally, the binary matrix is saved as a binary picture, which is the final lesion segmentation prediction result.

4. The medical image oriented federated learning fairness improvement method according to claim 3, characterized in that, ​ (2-1) The kth client in the K clients divides the preprocessed PET-CT image pairs obtained in step (1) into a training set and a test set in a ratio of 8:2, where K is a natural number greater than or equal to 2, and k ∈ [1, K]; (2-2) The kth client in the K clients initializes the total number of iterations of the training process and the learning rate η of the multi-modal neural image tumor segmentation model; (2-3) The kth client in the K clients inputs the PET image in each PET-CT image pair in the training set obtained in step (2-1) into the first encoder to obtain the intermediate feature map of the PET image Meanwhile, the CT image in each PET-CT image pair in the training set obtained in step (2-1) is input into the second encoder to obtain the intermediate feature map of the CT image Wherein H, W, C represent the height, width and dimension of the intermediate feature map F pet of the PET image respectively (2-4) The kth client in the K clients fuses the intermediate feature maps F pet and the intermediate feature maps F ct The input attention component part is inputted to obtain the feature maps F" fused in the spatial domain and the spatio-temporal domain. (2-5) The kth client in the K clients inputs the feature map F" fused in the channel domain and the spatial domain obtained in step (2-4) into a decoder to obtain a final lesion segmentation prediction result F pred ; (2-6) the kth client in the K clients performs lesion segmentation prediction according to the lesion segmentation prediction result F obtained in step (2-5) pred a loss function is calculated, and a gradient g is calculated using the loss function k , and the calculated gradient g is used k to update the parameters of the multi-modal neural image tumor segmentation model W k , thereby obtaining the preliminary trained multi-modal neural image tumor segmentation model W of the kth client k , Wherein, the loss function in the present step is Dice Loss function, and its calculation formula is: Wherein, ∈ is 1*10 -8 , which is used to prevent the denominator from being 0; gt And F pred Respectively represent the real result and the predicted result, and |F gt |+|F pred | represents the total sum of voxels in the real result and the predicted result.

5. The medical image oriented federated learning fairness improvement method according to claim 4, characterized in that, Step (4) includes the following sub-steps: (4-1) Set the counter t = 1; (4-2) Determine whether t is equal to the preset federal learning round threshold value, if yes, the process ends, otherwise proceed to step (4-3); (4-3) The server obtains the gradient g' of the kth client according to step (3) k Calculate the dependence space score between the gradient of the kth client and the global average gradient Where g k,t Indicates the gradient of the kth client in the tth round of federated learning training, Indicates the average gradient of the K clients in the tth round of federated learning training; wherein the present step is to calculate the dependency spatial score using the following formula Where var represents variance, and cov represents covariance. (4-5) The server calculates the dependence space score between the gradient of the kth client obtained in step (4-3) and the global average gradient The range conversion is performed to obtain the converted dependence space score w of the kth client k,Dep , w k,Dep represents the converted dependence space score of the kth client; wherein the step of obtaining the converted dependent space score w of the kth client is obtained using the following formula k,Dep : Where exp represents the exponential function. (4-5) The server obtains the optimized gradient g'kof the kth client according to step (3-3) k Computes the directional space score of the kth client on the global model wherein the step is calculated using the following formula for the directional space score Where cos represents the cosine similarity. (4-6) The server utilizes an exponential function on the direction space score of the kth client on the global model obtained in step (4-5) Normalization is performed to obtain the normalized direction space score w of the kth client k,Dir ; wherein the step of calculating the normalized direction space score w is calculated using the following formula k,Dir : (4-7) The server calculates the weight w' of the kth client in the cooperation fairness according to the dependence space score w of the kth client obtained in step (4-4) k,Dep and the normalized direction space score w of the kth client obtained in step (4-6) k,Dir k,Col ;​ wherein the weight w' of the kth client in terms of collaboration fairness is calculated by the following formula in this step k,Col : (4-8) The server obtains the gradient g'k of the kth client according to step (3) k The L2 norm scaling factor |g k,t |2 of the kth client gradient wherein the step of calculating the L2 norm scaling factor |g k,t |2: where γ is a regularization coefficient, (g k ) i is the gradient g k as the i-th component of the vector; (4-9) The server sums up the L2 norm scaling factors of the gradients of all clients obtained in step (4-8) to obtain the sum of the L2 norm scaling factors of the gradients of all clients (4-10) The server obtains the L2 norm scaling factor of the kth client gradient |g k，t The sum of the L2 norm scaling factors of all clients' gradients obtained in step (4-9) Computes the distance space weight coefficient w k,dis ; wherein the step is calculated by using the following formula to calculate the distance space weight coefficient w of the kth client k,dis : (4-11) The server calculates the loss function value l k.t of the kth client according to the loss function value l k,t of the kth client in the K clients max k∈[s k.t , and calculates the loss space weight coefficient w k,loss of the kth client, where l k.t represents the loss function value of the kth client in the tth round of federated learning training, s k,t represents the set of all participating k clients in the tth round of federated learning training; wherein the step is calculated using the following formula for the loss space weight coefficient w of the kth client k , loss: (4-12) The server calculates the distance space weight coefficient w of the kth client according to the distance d k ,dis and the loss space weight coefficient w of the kth client obtained in step (4-11) k,loss (4-13) The server calculates the weight w' of the kth client in performance fairness according to the distance space weight coefficient w and the loss space weight coefficient w k,Per ; wherein the step of calculating the weight w' of the kth client in terms of performance fairness is calculated using the following formula k,Per : w' k,Per = w k,dis *w k,loss (4-13) The server calculates the weight w' of the kth client in terms of collaboration fairness according to the result of step (4-7) k,Col and the weight w' of the kth client in terms of performance fairness according to the result of step (4-12) k,Per calculates the fairness weight w of the kth client k ; In particular, the step is to calculate the fair weight w of the kth client using the following formula k : w k = λ x w' k,Per + (1 - λ) x w' k,Col Where λ represents a factor balancing the cooperation and performance fairness weight, and its value range is 0 to 1; (4-14) The server obtains the fair weight w of the kth client from step (4-13) k The weight balancing process is performed to obtain the balanced fair weight w of the kth client (4-15) The server obtains the balanced fair weight of the kth client according to step (4-14) Obtains the final normalized weight of the kth client In particular, the final normalized weight of the kth client is calculated using the following formula (4-16) The server obtains the final normalized weight of the kth client according to step (4-15) and the gradient g' of the kth client after optimization in step (3-3) k calculates the aggregated global model gradient g global ; In particular, the step is to calculate the aggregated global model gradient g using the following formula global : (4-17) The server sends the aggregated global model gradient g global to the kth client, and sets t = t + 1.

6. The medical image oriented federated learning fairness improvement method according to claim 5, characterized in that, Step (4-14) includes the following sub-steps: (4-14-1) The server calculates the fair weight w of the kth client obtained in step (4-13) k Calculate the minimum weight w min , w min is the minimum value of the fair weight of all K clients; (4-14-2) The server obtains the minimum weight w according to step (4-14-1) min Computing the equalization weight w equ ; wherein the step of calculating the equalization weight w is performed using the following formula equ : Where T represents a pre-set minimum weight threshold value, and its value range is 0 to 1; (4-14-3) The server calculates the balanced weight w of the kth client according to the weight w obtained in step (4-14-2) equ and the fair weight w of the kth client obtained in step (4-13) k The balanced weight w of the kth client is calculated wherein the step is calculated using the following formula for the kth client balanced weight Where max(a, b) represents the maximum of a and b.

7. A medical image-oriented federated learning fairness improvement system configured to perform the medical image-oriented federated learning fairness improvement method according to any one of claims 1 to 6. The system comprises: A first module arranged in the kth client in the K clients, used to obtain a positron emission computed tomography (PET) image pair composed of a positron emission computed tomography (PET) image and a corresponding computed tomography (CT) image, and to preprocess the PET-CT image pair to obtain a preprocessed PET-CT image pair; a second module configured on a kth client in the K clients, configured to input the first module pre-processed PET-CT image pair into the multi-modal neural image tumor segmentation model W k for training, to obtain a gradient g k of the multi-modal neural image tumor segmentation model W k by calculating a loss function, to update parameters of the multi-modal neural image tumor segmentation model W k using the gradient g k to obtain a kth client preliminary trained multi-modal neural image tumor segmentation model W k , and to upload the gradient g k of the preliminary trained multi-modal neural image tumor segmentation model W k to the server, wherein k∈[1,K] A third module is arranged on the server, configured to perform the gradient g k The third module is arranged on the server, configured to perform the gradient g k ; A fourth module, arranged on the server, is configured to obtain the gradient g′ of the kth client optimized by the third module k Perform fair aggregation to obtain an aggregated global model gradient g global And send the aggregated global model gradient g global To the kth client; A fifth module configured to use the global model gradient g obtained by the fourth module to update the preliminary trained multi-modal neural image tumor segmentation model W of the second module global The gradient g of the preliminary trained multi-modal neural image tumor segmentation model W of the second module k The gradient g of the preliminary trained multi-modal neural image tumor segmentation model W of the second module k ; A sixth module arranged in the kth client in the K clients, used to repeat the third module to the fifth module m times, and the process ends.

Images