A federated learning method for multi-source heterogeneous medical images
By using amplitude normalization and weight perturbation terms in federated learning, the problem of heterogeneity of medical image data is solved, and the robustness and generalization ability of the model are improved, ensuring that it can still be carried out under heterogeneous data conditions. Medical image processing.
Patent Information
- Application Number
- CN202310283764.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-03-22
- Publication Date
- 2024-12-24
- Estimated Expiration
- 2043-03-22
AI Technical Summary
In the field of medicine, data from multiple medical centers are subject to data heterogeneity caused by different staining conditions and device suppliers, affecting the effectiveness and data security of federated learning.
Using the amplitude normalization and weight perturbation term method, the image is processed by Fourier transform and amplitude normalization on the client to reduce the data heterogeneity problem, and add weight perturbation term when network parameters are updated to smooth the loss Field area, improve model generalization capabilities.
Effectively alleviate the heterogeneity problem of multi-source medical images, improve the robustness and generalization capabilities of the model, and ensure that good performance can still be achieved under heterogeneous data conditions.
Smart Images
Figure CN116524292B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to a federated learning method for multi-source heterogeneous medical images. Background Art
[0002] Federated learning is an emerging basic technology of artificial intelligence. Its design goal is to carry out efficient machine learning among multiple participants or multiple computing nodes under the premise of ensuring information security during big data exchange, protecting terminal data and personal data privacy, and ensuring legality and compliance.
[0003] At present, in the medical industry, there are many types of data and most of them involve user personal privacy and are highly sensitive, such as patients' electrocardiogram, B-ultrasound and other image data. Therefore, balancing the needs of data sharing and the requirements of data security and privacy protection has become a key issue. Federated learning has become a new learning paradigm for effectively utilizing decentralized medical resources and breaking data silos.
[0004] However, there are still some problems in the application of federated learning in the medical field, such as the heterogeneity of data from multiple medical centers.
[0005] This data heterogeneity problem is mostly due to the different data sources of different clients. For example, the appearance of histological images varies due to different staining conditions, so histological images with different staining conditions have the same labels although they have different features. Different equipment suppliers and data acquisition protocols may also cause this data heterogeneity problem. Summary of the invention
[0006] The purpose of this invention is to propose a reasonable new federated learning training scheme for the heterogeneity problem of multi-center medical images, and effectively alleviate the heterogeneity problem of multi-source medical images through two new methods: amplitude normalization and weight perturbation term.
[0007] To achieve the above technical objectives, the present invention is a federated learning method for multi-source heterogeneous medical images, which includes a server for aggregating federated model parameters, the server is respectively connected to multiple client nodes that have training capabilities and perform training on different data sets, and the server assigns training tasks to different client nodes. The client node obtains different model parameters by training the local data set, and uploads them to the server at the same time. The server completes the aggregation and update of the model parameters, and returns the updated parameters to the client, and the client starts the next iteration. The above procedure will be repeated until the entire training process converges.
[0008] The technical solution of the present invention specifically comprises the following steps:
[0009] Step 1: Each client obtains the medical images that need to be learned, such as abdominal area CT images, ovarian tissue images, etc. Build an image processing model based on federated learning, which includes multiple local models and a global model. Each client with training capabilities is set up with a local model, while the global model is only set up in the server.
[0010] Step 2: Construct a local dataset for the data in each client: divide the training set and test set according to a certain ratio, process each image in the dataset to the same size, and perform data enhancement operations; load the data on the client so that the local model can be trained on the client.
[0011] Step 3: Define the loss function and objective function. The objective function is:
[0012]
[0013] in is the client objective function, l i is the loss function defined by the model θ, (x, y) is an image and its label, δ is a weight perturbation term, D i is the distribution followed by the data sampled from the i-th client, is the amplitude-normalized coordination feature distribution, p i is the weight corresponding to the i-th client, p i >0 and Ψ(·) is an amplitude normalization operator that harmonizes various client feature distributions to alleviate the client learning drift problem.
[0014] Step 4: Initialize parameters of the global model.
[0015] Step 5: Send the global model to each client, and each client updates the network parameters of the local model based on the network parameters of the global model.
[0016] Step 6: Start training on the client side.
[0017] Step 7: Use Fourier transform on the client to convert the input image into a frequency domain signal and decompose the amplitude spectrum that captures low-frequency features. Considering the privacy sensitivity of patients and communication cost issues, only the average amplitude during communication is used for normalization. The amplitude carries color information, and amplitude normalization can effectively alleviate the data heterogeneity problem caused by some feature distribution deviations, such as different dyeing conditions.
[0018] Step 8: Train the amplitude-normalized image to obtain the gradient information of the local model, and regularize the gradient information using the L2 norm to obtain the weight perturbation term.
[0019] Step 9: After obtaining the weight perturbation term, update the parameters of the local model according to the gradient and the weight perturbation term. This approach can effectively constrain the model, make the value of the loss function smaller, and make the model move to a flatter loss field.
[0020] Step 10: Repeat step 6 to step 9 until the client completes training.
[0021] Step 11: Send the local model to the server, and the server updates the global model based on the local models of each client.
[0022] Step 12: Repeat steps 5-11 until the number of communications reaches the preset value. Finally, the required global model is obtained on the server side and sent to each client. Test on the client side to verify whether the obtained model can achieve good performance.
[0023] When processing image segmentation tasks, it is suitable for various image segmentation problems in medical images, such as: blood vessel segmentation in retinal images, liver tumor segmentation, prostate segmentation, etc. The local model and global model established in step 3 are both U-Net image segmentation models. This network uses a sliding window to provide the surrounding area of the pixel as the input of the convolutional network, and uses this to predict the label of the pixel. Its feature fusion method determines its U-shaped structure, which is fused with the features extracted by the previous network while upsampling. The introduction of this convolutional network has contributed to the pixel-level segmentation of biomedical images, so that the few-sample image segmentation in the biomedical field can also achieve good results.
[0024] When processing image classification tasks, in order to adapt to various image classification problems in medical images, the local and global models established in step 1 are both DenseNet image classification models. The convolutional network includes shorter connections between the layers close to the input layer and the layers close to the output layer, making the network training deeper, more accurate and more effective.
[0025] In step 2, the ratio of training set and test set is divided into 8:2, that is, 80% of the images are used to train the local model, and 20% of the images are retained. After completing step 12, the obtained global model is tested to detect whether the final global model can meet the needs of each client. The data enhancement method in step 2 is to pre-process the data before training without substantially increasing the data, so that the limited data can produce the value equivalent to more data. This type of method has the effect of avoiding overfitting, improving model robustness, and avoiding sample imbalance. The data enhancement methods used here include: random center cropping, flipping, rotation, translation, jittering, Gaussian blurring, adjusting HSV contrast, and histogram equalization.
[0026] The loss function l used in step 3 i It is the cross entropy loss function, which is a loss function often used in image processing tasks.
[0027] In step 8, the operation of calculating the weight perturbation term is adopted. The calculation formula of the weight perturbation term is as follows:
[0028]
[0029] Among them, θ i,k-1 is the local model of the i-th client at the k-1th training, and α is a parameter that controls the degree of perturbation.
[0030] In step 7, the image amplitude is normalized. The specific operation is:
[0031] For input x∈R (H,W,C) The image is converted into a frequency domain signal using the following formula:
[0032]
[0033] Therefore, we can i (x) and separate the real part R i (x) and the imaginary part L i (x) The two parts, amplitude and phase components can be expressed as:
[0034]
[0035] P i (x) = arctan[L i (x) / R i (x)]
[0036] Among them A i (x) is the amplitude, P i (x) is the phase; the next step is to perform batch normalization on each image. For the kth batch, M images, first use the above formula to decompose its phase and amplitude components, and then use each image x m The amplitude of is calculated as follows:
[0037]
[0038]
[0039] is the average amplitude calculated in the first k-1 batches, σ i,k 2 is the population variance in the first k-1 batches, γ and β are two adjustable parameters, and the updated σ i,k 2, you can use σ i,k 2 and A new amplitude is calculated as follows:
[0040]
[0041] After obtaining the normalized amplitude, the processed image can be calculated as follows:
[0042]
[0043] Among them, F -1 is the inverse Fourier transform, Ψ(x m ) is the processed image.
[0044] Compared with the closest prior art, the technical solution provided by the present invention has the following beneficial effects:
[0045] 1. The present invention effectively alleviates the heterogeneity problem of multi-source medical images by improving the training image processing method and the objective function;
[0046] 2. Before each batch of images and labels are sent to the network, the present invention performs Fourier transform on the images, extracts the amplitude value of the images, and normalizes the amplitude of the images; the amplitude carries color information, and amplitude normalization can effectively alleviate the data heterogeneity problem caused by some feature distribution deviations, such as different dyeing conditions;
[0047] 3. The present invention calculates the weight perturbation term from the gradient before performing gradient back propagation, and adds the weight perturbation term when the network parameters are updated; this approach is conducive to reducing the gradient value, so that the model can reach a flatter loss field; the model trained in this way has stronger generalization ability, stronger robustness, and can achieve a smaller loss function value. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] Figure 1 It is a relationship diagram between local model and global model in federated learning;
[0049] Figure 2 It is the overall flow chart of the training process of the federated learning method for multi-source heterogeneous medical images. DETAILED DESCRIPTION
[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0051] See also Figure 1-Figure 2 , the present invention provides a technical solution:
[0052] A federated learning method for multi-source heterogeneous medical images includes the following steps:
[0053] Step 1: After the client has read in the data, the first thing to do is to select the corresponding global model. The global model here should be based on the network model required for specific image processing. In order to facilitate the server and client to complete the network parameter update, the global model and the local model use the same network structure.
[0054] When you need to complete an image classification task, you can use the DenseNet network structure as the global model, and the local model is also the DenseNet network structure. Due to its dense connection characteristics, this network structure will not experience the gradient disappearance phenomenon even if it is stacked to a very deep network depth, and as the network depth increases, the accuracy that the final model can achieve will also increase. Therefore, this network is selected here as the network structure for the image classification task.
[0055] When it is necessary to complete the image segmentation task, the network structure of U-Net can be used as a global model and a local model. U-Net is an image segmentation network commonly used in the field of medical images. Its characteristic is that the shallow features and deep features are combined through the connection structure in the upsampling process. Medical images often have blurred boundaries, complex gradients, and require more high-resolution information. At the same time, the internal structure of the human body is relatively fixed, and the distribution of segmentation targets in human body images is very regular, with simple and clear semantics, and low-resolution information can be easily located. For medical images, U-Net can use deep features for positioning and shallow features for precise segmentation. Therefore, in the field of medical images, U-Net has a good segmentation effect. Therefore, this type of network is selected as the network structure for image segmentation tasks.
[0056] Step 2: Each client builds a local data set based on its local data, and divides the training set and the test set in a ratio of 8:2. A series of data enhancement operations are performed, including: random center cropping, flipping, rotation, translation, jittering, Gaussian blurring, adjusting HSV contrast, and histogram equalization. This type of method is to pre-process the data before training without substantially increasing the data, so that limited data can produce the value equivalent to more data. This type of method has the effect of avoiding overfitting, improving model robustness, and avoiding sample imbalance.
[0057] Step 3: Define the loss function and objective function. The objective function is:
[0058]
[0059] in is the client objective function, l i is the loss function defined by the model θ, (x, y) is an image and its label, δ is a weight perturbation term, D i is the distribution followed by the data sampled from the i-th client, is the amplitude-normalized coordination feature distribution, p i is the weight corresponding to the i-th client, p i >0 and Ψ(·) is an amplitude normalization operator that harmonizes various client feature distributions to alleviate the client learning drift problem.
[0060] Here i The cross entropy loss commonly used in image processing is used. The specific formula is:
[0061] CEL(p t )=-log p t
[0062] Among them, p t is the probability of the t-class label in the prediction result when the true label is t.
[0063] The calculation formula for the weight perturbation term here is as follows:
[0064]
[0065] Among them, θ i,k-1 is the local model of the i-th client at the k-1th training. α is a parameter that controls the degree of disturbance. In actual training, the appropriate size can be found through multiple debugging.
[0066] Step 4: Initialize parameters of the global model.
[0067] Step 5: Send the global model to each client, and each client updates the network parameters of the local model based on the network parameters of the global model.
[0068] Step 6: Start training on the client. During training, each client uses a simultaneous parallel training method.
[0069] Step 7: Send a batch of images and labels. Before sending the batch of images to the network training, amplitude normalization must be performed first. This is also one of the means used in the present invention to alleviate the heterogeneity problem of multi-source medical images.
[0070] The specific operations are:
[0071] For input x∈R (H,W,C) The image is converted into a frequency domain signal using the following formula:
[0072]
[0073] Therefore, we can i (x) and separate the real part R i (x) and the imaginary part L i (x) The two parts, amplitude and phase components can be expressed as:
[0074]
[0075] P i (x) = arctan[L i (x) / R i (x)]
[0076] Among them A i (x) is the amplitude, P i (x) is the phase. The next step is to perform batch normalization on each image. For the kth batch, M images, first use the above formula to decompose its phase and amplitude components, and then use each image x m The amplitude of is calculated as follows:
[0077]
[0078]
[0079] is the average amplitude calculated in the first k-1 batches, σ i,k 2 is the overall variance in the first k-1 batches, γ and β are two adjustable parameters. σ i,k 2 , you can use σ i,k2 and A new amplitude is calculated as follows:
[0080]
[0081] After obtaining the normalized amplitude, the processed image can be calculated as follows:
[0082]
[0083] Among them, F -1 is the inverse Fourier transform, Ψ(x m ) is the processed image.
[0084] Step 8: Input the amplitude normalized image into the local model whose parameters have been updated in step 5 In which t represents the number of communications, i represents the i-th client, and k represents the number of training times on the client. After forward propagation through the network, the gradient is calculated. Using the weight perturbation term calculation formula in step 3, the weight perturbation term δ is calculated. k .
[0085] Step 9: Update the weight parameters of the local model. The formula is as follows:
[0086]
[0087]
[0088] in, is the learning rate of the i-th client during the k-th parameter update during the t-th communication process, g δ is the gradient after adding the weight perturbation term.
[0089] Step 10: Repeat steps 6 to 9 until the client completes K trainings. During this process, the client's local model has also been updated K times, and has achieved good results on the local dataset. If you want to continue to improve the performance of the local model, you need to add gradient information brought by new data.
[0090] Step 11: After completing K trainings, the local model is sent to the server, which then updates the global model using the following formula:
[0091]
[0092] θ t is the global model, lr g is the global learning rate of the server.
[0093] At the same time, the average amplitude of each client in this training and amplitude mean square error σ i,K 2 It also needs to be sent back to the server to calculate the global average amplitude and the global amplitude mean square error. The formula is as follows:
[0094]
[0095]
[0096] The updated global average amplitude and global amplitude mean square error will be sent to each client for use in the next communication process as the initial values of the average amplitude and amplitude mean square error in each client.
[0097] Step 12: Repeat steps 5 to 11 until the preset T communication processes are completed. After the required global model is finally obtained on the server side, the global model is sent to each client, and the test set set in step 2 is used locally on each client to test whether the expected effect is achieved.
[0098] In summary, the present invention adopts two new methods to constrain the data distribution of each client and the distribution of network parameters.
[0099] Among them, the amplitude normalization operation constrains the amplitude distribution of the image before the image input network performs forward propagation, so that the images in the local data set of each client have similar distributions in color and other amplitude-related features, thereby solving the heterogeneity problem of multi-source medical images in amplitude.
[0100] Another method, adding a weight perturbation term, constrains the network parameters so that they move to a more moderate and flat loss field during the update process. In this way, the global model can achieve a good balance among the local optimal solutions of each client, so that the model can still perform relatively well even when receiving medical images with heterogeneity.
[0101] Finally, it should be noted that the selection of the network structure in the above step 1 is only used to illustrate the technical solution of the present invention rather than to limit it. Although two recommended networks are given for image classification and image segmentation tasks, those skilled in the art should understand that the network structure of the present invention can be modified or replaced by equivalents without departing from the two technical solutions of the present invention, which should all be covered by the scope of the claims of the present invention.
Claims
1. A federated learning method for multi-source heterogeneous medical images, characterized in that The following steps are involved: Step 1: Each client obtains the medical images that need to be learned and builds an image processing model based on federated learning. The image processing model includes multiple local models and a global model. Step 2: Construct a local data set for the data existing in each client; Step 3: Define the loss function and objective function; Step 4: Initialize parameters of the global model; Step 5: Send the global model to each client, and each client updates the network parameters of the local model according to the network parameters of the global model; Step 6: Start training on the client; Step 7: Use Fourier transform on the client to convert the input image into a frequency domain signal, decompose the amplitude spectrum that captures low-frequency features, and perform amplitude normalization; In step 7, an operation of normalizing the image amplitude is adopted, and the specific operation is: For input The image is converted into a frequency domain signal using the following formula: ; Therefore, we can Separate the real part and the imaginary part The two parts, amplitude and phase components can be expressed as: {A}_{i}(x)=\left [ {{R}_{i}{}^{2}(x)+{L}_{i}{}^{2}(x)} \right ]^{\frac {1} {2}}\left [ {} \right ]\left [ {{}} \right ]^{} ; {P}_{i}(x)=arctan[{{L}_{i}(x) / {R}_{i}(x)}] ; in is the amplitude, is the phase; the next step is to perform batch normalization on each image. batch, For each image, first use the above formula to decompose its phase and amplitude components, and then use The amplitude of is calculated as follows: ; It is before The average amplitude calculated in batches, For the front The overall variance in batches, are two adjustable parameters, and the updated , you can use and A new amplitude is calculated as follows: ; After obtaining the normalized amplitude, the processed image can be calculated as follows: ; in, is the inverse Fourier transform, is the processed image; Step 8: Train the amplitude-normalized image to obtain the gradient information of the local model, and regularize the gradient information using the L2 norm to obtain the weight perturbation term; Step 9: After obtaining the weight perturbation term, update the parameters of the local model according to the gradient and the weight perturbation term; Step 10: Repeat steps 6 to 9 until the client completes training; Step 11: Send the local model to the server, and the server updates the global model based on the local models of each client; Step 12: Repeat steps 5-11 until the number of communications reaches a preset value, and finally obtain the global model.
2. The method for federated learning of multi-source heterogeneous medical images according to claim 1, characterized in that: In step 8, an operation of calculating a weight disturbance term is adopted, and the calculation formula of the weight disturbance term is as follows: ; in, It is Client No. The local model during the training, α is a parameter that controls the degree of perturbation.
3. The method for federated learning of multi-source heterogeneous medical images according to claim 1, characterized in that: The objective function in step 3 is: ; in is the client objective function, is the loss function defined by the model θ, is an image and its label, is a weight perturbation term, For The data sampled by each client follows the distribution, is the coordination feature distribution after amplitude normalization, For the The weight corresponding to each client, >0 and , is an amplitude normalization operator.
4. The method for federated learning of multi-source heterogeneous medical images according to claim 1, characterized in that: When processing image segmentation tasks, the local model and the global model in step 3 are both U-Net image segmentation models; when processing image classification tasks, the local model and the global model in step 1 are both DenseNet image classification models.