Semi-supervised federated learning magnetic resonance imaging method

Abstract

The application relates to the technical field of image processing, in particular to a semi-supervised federated learning magnetic resonance imaging method, which comprises the following steps: training a network to obtain spatial data corresponding to a network output image at a local client participating in federated learning; taking pixel point data of an undersampled image as initial pseudo-labels, and updating the initial pseudo-labels based on the spatial data corresponding to the network output image; uploading a local client model to a center server; aggregating the local client model to obtain a global model; and transmitting the global model to the local client. The semi-supervised federated learning magnetic resonance imaging method can solve the network training problem of some local clients lacking full sampling reference data.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a semi-supervised federated learning magnetic resonance imaging method. Background Technology

[0002] The principle of magnetic resonance imaging (MRI) is as follows: when an external magnetic field is applied, radio frequency (RF) pulses of a specific frequency are used to excite protons in the tissue being tested. The protons absorb a certain amount of energy and resonate. When the RF pulse is stopped, the excited protons gradually release the absorbed energy in the form of a scanning signal. By acquiring the scanning signal and processing it using image reconstruction technology, a scan image of the tissue being tested can be obtained.

[0003] The existing federated magnetic resonance imaging reconstruction methods mainly have the following two problems: (1) The federated learning reconstruction methods rely on fully sampled reference data. Existing federated learning reconstruction methods all rely on a strong assumption: that each hospital has fully sampled data as reference values ​​for model training. However, in reality, some hospitals only have undersampled data and do not have corresponding fully sampled data. (2) Most of the federated semi-supervised learning methods used for medical images in the past have been applied to classification or segmentation tasks, and methods suitable for reconstruction tasks have not yet been explored. Summary of the Invention

[0004] This application provides a semi-supervised federated learning magnetic resonance imaging method to address the problem that institutions or hospitals without full sampling reference data cannot participate in the federated learning magnetic resonance reconstruction process.

[0005] To address the aforementioned technical problems, this application provides a semi-supervised federated learning magnetic resonance imaging method, comprising: training a network on a local client participating in federated learning to obtain spatial data corresponding to the network output image; using pixel data of an undersampled image as initial pseudo-labels and updating the initial pseudo-labels based on the spatial data corresponding to the network output image; uploading the local client model to a central server; aggregating the local client model to obtain a global model; and transmitting the global model to the local client.

[0006] In some exemplary embodiments, the local client includes a local client with fully sampled reference data and a local client lacking fully sampled reference data; for a local client with fully sampled reference data, training the network includes: training the network with an undersampled image as network input and a fully sampled image as label; the supervised loss function is shown in equation (1):

[0007]

[0008] Among them, f wk For parameter w k neural networks, x ref This is the full sampling reference data, and k is the local client number.

[0009] In some exemplary embodiments, the local client includes a local client with fully sampled reference data and a local client lacking fully sampled reference data; for a local client lacking fully sampled reference data, training the network includes: performing data augmentation processing on the spatial data corresponding to the undersampled images to obtain at least three undersampled images; using two undersampled images as inputs to two networks respectively, and using one undersampled image other than the two undersampled images as a label to train the two networks.

[0010] In some exemplary embodiments, data augmentation processing is performed on the spatial data corresponding to the undersampled image to obtain three undersampled images; two undersampled images are used as inputs to two networks respectively, and one undersampled image other than the two undersampled images is used as a label to train the two networks, including: undersampling the spatial data corresponding to the undersampled image with three random initial pseudo-labels to obtain a first data subset, a second data subset, and a third data subset; performing inverse Fourier transform on the first data subset and the second data subset to obtain a first undersampled image and a second undersampled image; using the first undersampled image and the second undersampled image as inputs to the two networks, and using the third data subset as a label to train the two networks.

[0011] In some exemplary embodiments, the supervised loss function for training the two networks is shown in equation (2):

[0012]

[0013] in, and These are the undersampled K-space data corresponding to the network output image, where k is the local client number; Three random initial pseudo-labels, These are the first data subset, the second data subset, and the third data subset, respectively.

[0014] In some exemplary embodiments, and The undersampled K-space data corresponding to the network output image are denoted as Y1 and Y2, respectively. Based on the spatial data corresponding to the network output image, the initial pseudo-label is updated, including: determining whether the values ​​of the corresponding pixels in Y1 and Y2 are equal; if so, the corresponding pixels in Y1 and Y2 are determined pixels, and the determined pixels are updated as pseudo-labels; if not, the initial pseudo-label is not updated.

[0015] In some exemplary embodiments, uploading the local client model to the central server includes: for local clients with full-sampled reference data, uploading the local client model to the central server; for local clients lacking full-sampled reference data, uploading either of the two networks to the central server.

[0016] In some exemplary embodiments, equation (3) is used to aggregate the local client model:

[0017]

[0018] Where, ω k This indicates the weights of the local client model. and These represent the local client model of communication round t and the global model of communication round t+1, respectively.

[0019] In some exemplary embodiments, the weights of the local client model are dynamically adjusted based on the root mean square error of the model's predicted values ​​compared to the original input values; the weights of the local client model are represented by equation (4):

[0020]

[0021] Among them, f i (x k ) represents the network prediction value for the local client, x k The image is the original undersampled image, and γ is the dynamic weighting coefficient.

[0022] In some exemplary embodiments, transmitting the global model to a local client includes: for a local client with fully sampled reference data, the global model directly replaces the local client model; for a local client lacking fully sampled reference data, the parameters of both networks are replaced with the parameters of the global model.

[0023] The technical solution provided in this application has at least the following advantages:

[0024] This application primarily addresses the problem that institutions or hospitals lacking fully sampled reference data cannot participate in the federated learning magnetic resonance imaging (MRI) reconstruction process. It provides a semi-supervised federated learning MRI method, which is a federated semi-supervised learning MRI image reconstruction method based on consistent perception determinism and gradually improving pseudo-labels. The advantages of this invention are: it is a semi-supervised learning method within a federated learning framework, avoiding cross-site data access and protecting the data privacy of each local client. Furthermore, it solves the network training problem where some local clients lack fully sampled reference data. Moreover, this invention does not simply extend the semi-supervised learning MRI reconstruction method to the federated learning framework; instead, it progressively supplements pseudo-labels based on the inherent properties of MRI reconstruction, while employing a dynamic weighted aggregation method to avoid the inaccuracy of pseudo-labels affecting the accuracy of the global model. Attached Figure Description

[0025] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations do not constitute a limitation on the embodiments, and unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0026] Figure 1 A flowchart illustrating a semi-supervised federated learning magnetic resonance imaging method provided in an embodiment of this application;

[0027] Figure 2 This is a schematic diagram of a semi-supervised federated learning magnetic resonance imaging method provided for another embodiment of this application. Detailed Implementation

[0028] As can be seen from the background technology, the existing federated magnetic resonance image reconstruction methods mainly have the following two problems: (1) The federated learning reconstruction method relies on the problem of full-sampled reference data. Existing federated learning reconstruction methods all rely on a strong assumption: that each hospital has full-sampled data as reference values ​​for model training. However, in reality, some hospitals only have undersampled data and do not have corresponding full-sampled data. (2) Most of the federated semi-supervised learning methods used for medical images in the past have been applied to classification or segmentation tasks, and methods suitable for reconstruction tasks have not yet been explored.

[0029] To address the above two issues, this invention proposes a federated semi-supervised learning method for magnetic resonance image reconstruction, applicable to situations where some hospitals lack full-sample reference data while others possess it. The main technical problems this method needs to solve are as follows: (1) the problem of pseudo-label generation for clients without full-sample reference data; (2) the problem of pseudo-label determinism; and (3) the problem of balancing the contributions of clients without and with full-sample reference data in the federated learning model.

[0030] Federated learning typically involves collaboratively learning a global model from multiple clients in a decentralized manner. This not only alleviates the problem of insufficient data from individual clients but also effectively protects the data privacy of each client. Some techniques propose federated learning reconstruction methods that preserve the characteristics of institutional data, avoiding performance degradation of local models due to data heterogeneity. Other techniques, while maintaining the characteristics of institutional data, propose data consistency constraints during the testing phase to enhance the model's generalization ability. However, these federated learning magnetic resonance imaging (MRI) reconstruction methods are only applicable to supervised learning settings, which limits the participation of institutions or hospitals without fully sampled reference data in the federated learning process. In real-world scenarios, many hospitals typically lack fully sampled reference data. How to utilize these undersampled datasets that do not contain fully sampled reference data to improve the performance of federated reconstruction models remains an issue that needs further exploration.

[0031] Existing federated semi-supervised learning methods are primarily developed for classification or segmentation tasks of natural images. A simple approach is to apply magnetic resonance imaging (MRI) reconstruction methods to federated learning, but these methods do not explore the knowledge of heterogeneous data from multiple institutions. Other related techniques propose classification methods that utilize inter-institutional information, federated semi-supervised learning classification methods for imbalanced distributions of different institution classes, or hybrid supervised federated learning adaptive weight allocation methods suitable for mixing different levels of segmentation labels. While these methods achieve good classification or segmentation results while protecting data privacy, they cannot be directly applied to MRI reconstruction tasks. Therefore, designing federated semi-supervised learning methods suitable for MRI reconstruction is crucial.

[0032] Based on this, existing federated learning magnetic resonance reconstruction methods have the following technical problems: (1) Existing federated learning magnetic resonance reconstruction methods are all based on the assumption that all local clients have fully sampled reference data during the training process. (2) Existing federated semi-supervised learning techniques are all adapted to natural image classification tasks, and there are no methods designed for medical image reconstruction tasks. (3) Previous semi-supervised learning magnetic resonance reconstruction methods were usually designed for centralized training settings. They are based on the assumption that fully sampled reference data is accessible and provide auxiliary assistance for learning undersampled data. However, if the fully sampled reference data and the undersampled data exist on different local clients, cross-client access to data will infringe on data privacy.

[0033] To address the aforementioned technical problems, this application provides a semi-supervised federated learning magnetic resonance imaging method, comprising: training a network on a local client participating in federated learning to obtain spatial data corresponding to the network output image; using pixel data of an undersampled image as initial pseudo-labels, and updating the initial pseudo-labels based on the spatial data corresponding to the network output image; uploading the local client model to a central server; aggregating the local client models to obtain a global model; and transmitting the global model to the local client. The semi-supervised federated learning magnetic resonance imaging method provided by this invention can solve the problem of network training for some local clients lacking fully sampled reference data.

[0034] The embodiments of this application will now be described in detail with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been provided in the embodiments of this application to facilitate a better understanding of the application. However, the technical solutions claimed in this application can be implemented even without these technical details and various variations and modifications based on the following embodiments.

[0035] See Figure 1 This application provides a semi-supervised federated learning magnetic resonance imaging method, including the following steps:

[0036] Step S1: Train the network on the local client participating in federated learning and obtain the spatial data corresponding to the network output image.

[0037] Step S2: Use the pixel data of the undersampled image as the initial pseudo-label, and update the initial pseudo-label based on the spatial data corresponding to the network output image.

[0038] Step S3: Upload the local client model to the central server.

[0039] Step S4: Aggregate the local client models to obtain the global model.

[0040] Step S5: Transfer the global model to the local client.

[0041] This invention primarily addresses the problem that institutions or hospitals lacking fully sampled reference data cannot participate in the federated learning magnetic resonance imaging (MRI) reconstruction process. It proposes a federated semi-supervised learning MRI image reconstruction method based on consistent perception determinism and gradually improving pseudo-labels. This method simplifies the five steps mentioned above to include local training, pseudo-label updating, local model uploading, global model aggregation, and global model downloading, for both institutions with and without fully sampled reference data. For clients without fully sampled reference data in federated learning, their own undersampled data is used as the initial pseudo-label. Furthermore, this invention proposes identifying equal K-space points (pixels with equal pixel values) among the multiple predicted values ​​reconstructed from multiple subsets of undersampled K-space data as determined K-space points, and continuously updating these determined K-space points as pseudo-labels to improve the accuracy of the pseudo-labels.

[0042] In some exemplary embodiments, the local client includes a local client with fully sampled reference data and a local client lacking fully sampled reference data; for a local client with fully sampled reference data, training the network in step S1 includes: training the network with an undersampled image as network input and a fully sampled image as label; the supervised loss function is shown in equation (1):

[0043]

[0044] Among them, f wk For parameter w k neural networks, x ref This is the full sampling reference data, and k is the local client number.

[0045] In some exemplary embodiments, the local client includes a local client with full-sampled reference data and a local client lacking full-sampled reference data; for a local client lacking full-sampled reference data, training the network in step S1 includes: performing data augmentation processing on the spatial data corresponding to the undersampled image to obtain at least three undersampled images; using two undersampled images as inputs to two networks respectively, and using one undersampled image other than the two undersampled images as a label to train the two networks.

[0046] The following explanation uses the example of performing data augmentation on the spatial data corresponding to the undersampled image to obtain three undersampled images.

[0047] In some exemplary embodiments, data augmentation processing is performed on the spatial data corresponding to the undersampled image to obtain three undersampled images; two undersampled images are used as inputs to two networks respectively, and one undersampled image other than the two undersampled images is used as a label to train the two networks, including: undersampling the spatial data corresponding to the undersampled image with three random initial pseudo-labels to obtain a first data subset, a second data subset, and a third data subset; performing inverse Fourier transform on the first data subset and the second data subset to obtain a first undersampled image and a second undersampled image; using the first undersampled image and the second undersampled image as inputs to the two networks, and using the third data subset as a label to train the two networks.

[0048] In some exemplary embodiments, the supervised loss function for training the two networks is shown in equation (2):

[0049]

[0050] in, and These are the undersampled K-space data corresponding to the network output image, where k is the local client number; Three random initial pseudo-labels, These are the first data subset, the second data subset, and the third data subset, respectively.

[0051] In some exemplary embodiments, and The undersampled K-space data corresponding to the network output image are denoted as Y1 and Y2, respectively. Based on the spatial data corresponding to the network output image, the initial pseudo-label is updated, including: determining whether the values ​​of corresponding pixels in Y1 and Y2 are equal; if so, the corresponding pixels in Y1 and Y2 are determined pixels, and the determined pixels are updated as pseudo-labels; if not, the initial pseudo-labels are not updated. Figure 2 As shown, the determined pixels consist of points from the original undersampled K-space data (black points in Y1 and Y2) and newly added pixel equivalent points (black points within the dashed boxes in Y1 and Y2). As the iteration progresses, the number of black points within the dashed boxes in Y1 and Y2 continuously increases. In each communication round t, the above three initial pseudo-labels... They will all be randomly selected again to improve the accuracy of pseudo-labels.

[0052] In some exemplary embodiments, uploading the local client model to the central server in step S4 includes: for local clients with full-sampled reference data, uploading the local client model to the central server; for local clients lacking full-sampled reference data, uploading either of the two networks to the central server.

[0053] In some exemplary embodiments, equation (3) is used to aggregate the local client model:

[0054]

[0055] Where, ω k This indicates the weights of the local client model. and These represent the local client model of communication round t and the global model of communication round t+1, respectively.

[0056] It should be noted that, considering the potential inaccuracy of pseudo-labels in client-side models lacking full-sample reference data, this invention dynamically adjusts the weights of the local client (also known as aggregated weights) based on the root mean square error of the model's predicted values ​​compared to the original input values. For the aggregated weight ω... k The definition is shown in equation (4).

[0057] In some exemplary embodiments, the weights of the local client model are dynamically adjusted based on the root mean square error of the model's predicted values ​​compared to the original input values; the weights of the local client model are represented by equation (4):

[0058]

[0059] Among them, f i (x k ) represents the network prediction value for the local client, x k The image is the original undersampled image, and γ is the dynamic weighting coefficient.

[0060] In some exemplary embodiments, step S5, transmitting the global model to the local client, includes: for a local client with fully sampled reference data, the global model directly replaces the local client model; for a local client lacking fully sampled reference data, the parameters of both networks are replaced with the parameters of the global model.

[0061] In summary, compared with existing federated learning magnetic resonance imaging reconstruction techniques, the advantages of this invention are:

[0062] (1) Federated semi-supervised learning of magnetic resonance reconstruction was achieved for multiple hospitals with full-sample reference data and multiple hospitals lacking full-sample reference data.

[0063] (2) For hospitals lacking full-sample reference data, using pseudo-labels generated by the supervised models of other hospitals for initial training may introduce errors in the initial training stage and cause error accumulation in subsequent stages. This method uses its own undersampled K-space data as pseudo-labels for the initial training of the model, which will not introduce error information.

[0064] (3) Dynamically adjust the aggregation weights to avoid the inaccuracy of pseudo-labels affecting the accuracy of the global model.

[0065] Based on the above technical solutions, this application primarily addresses the problem that institutions or hospitals lacking fully sampled reference data cannot participate in the federated learning magnetic resonance imaging (MRI) reconstruction process. It provides a semi-supervised federated learning MRI method, which is a federated semi-supervised learning MRI image reconstruction method based on consistent perception determinism and gradually improving pseudo-labels. The advantages of this invention are: it is a semi-supervised learning method within a federated learning framework, avoiding cross-site data access and protecting the data privacy of each local client. Furthermore, it solves the network training problem where some local clients lack fully sampled reference data. Moreover, this invention does not simply extend the semi-supervised learning MRI reconstruction method to the federated learning framework; instead, it progressively supplements pseudo-labels based on the inherent properties of MRI reconstruction, while employing a dynamic weighted aggregation method to avoid the inaccuracy of pseudo-labels affecting the accuracy of the global model.

[0066] Those skilled in the art will understand that the above-described embodiments are specific examples of implementing this application, and in practical applications, various changes in form and detail may be made without departing from the spirit and scope of this application. Any person skilled in the art can make their own modifications and alterations without departing from the spirit and scope of this application; therefore, the scope of protection of this application should be determined by the scope defined in the claims.

Claims

1. A semi-supervised federated learning magnetic resonance imaging method, characterized in that, include: On the local client participating in federated learning, the network is trained, and the K-space data corresponding to the network output image is obtained; The local clients include local clients with full sampled reference data and local clients lacking full sampled reference data; The pixel data of the undersampled image is used as the initial pseudo-label, and the initial pseudo-label is updated based on the K-space data corresponding to the network output image. It includes: and The undersampled K-space data corresponding to the two network output images are denoted as Y1 and Y2, respectively. Based on the K-space data corresponding to the network output images, the initial pseudo-labels are updated, including: determining whether the values ​​of corresponding pixels in Y1 and Y2 are equal; if so, the corresponding pixels in Y1 and Y2 are determined pixels, and the determined pixels are updated as pseudo-labels; if not, the initial pseudo-labels are not updated; where f is the neural network and k is the local client number. These are the first data subset and the second data subset, respectively. Upload the local client model to the central server; The local client models are aggregated to obtain a global model; The global model is then transferred to the local client. For local clients lacking fully sampled reference data, network training includes: performing data augmentation on the K-space data corresponding to the undersampled images to obtain three undersampled images; using two of the undersampled images as inputs to two networks respectively, and using one other undersampled image as a label, the two networks are trained, including: Undersample the K-space data corresponding to the undersampled image using three random initial pseudo-labels to obtain the first data subset, the second data subset, and the third data subset; The first and second data subsets are subjected to inverse Fourier transform to obtain a first undersampled image and a second undersampled image. The first and second undersampled images are used as inputs to two networks, and the third data subset is used as a label to train the two networks.

2. The semi-supervised federated learning magnetic resonance imaging method according to claim 1, characterized in that, For a local client with fully sampled reference data, training the network includes: The network is trained using undersampled images as input and fully sampled images as labels; the supervised loss function is shown in equation (1): （1） in, For parameter w k neural networks, For undersampled images, This is the reference data for the full sampling.

3. The semi-supervised federated learning magnetic resonance imaging method according to claim 1, characterized in that, The supervised loss function for training the two networks is shown in equation (2): （2） in, For the third data subset, The initial pseudo-labels are random.

4. The semi-supervised federated learning magnetic resonance imaging method according to claim 1, characterized in that, Upload the local client model to the central server, including: For local clients with full sampled reference data, the local client model is uploaded to the central server; For local clients lacking full-sample reference data, either of the two networks is uploaded to the central server.

5. The semi-supervised federated learning magnetic resonance imaging method according to claim 1, characterized in that, The local client model is aggregated using equation (3): （3） in, This indicates the weights of the local client model. and These represent the local client model of communication round t and the global model of communication round t+1, respectively.

6. The semi-supervised federated learning magnetic resonance imaging method according to claim 5, characterized in that, The weights of the local client model are dynamically adjusted based on the root mean square error between the model's predicted values ​​and the original input values; the weights of the local client model are expressed by equation (4): （4） in, x is the network prediction value for the local client. k The original undersampled image, These are dynamic weighting coefficients.

7. The semi-supervised federated learning magnetic resonance imaging method according to claim 1, characterized in that, Transferring the global model to the local client includes: For local clients with fully sampled reference data, the global model directly replaces the local client model; For local clients lacking full-sample reference data, the parameters of both networks are replaced with the parameters of the global model.

Images