Contrast-agent-free virtual enhanced magnetic resonance imaging device and data preprocessing device and method thereof

Abstract

The invention provides a contrast-agent-free virtual enhanced magnetic resonance imaging device and a corresponding method, the contrast-agent-free virtual enhanced magnetic resonance imaging device is based on a VCE-MRI model of federated learning (FL), the VCE-MRI model is trained by using large-scale highly heterogeneous multi-center data for data of nasopharyngeal carcinoma (NPC) patients, and the data privacy of the patients is protected while high generalization of the model is ensured. The invention further provides a data preprocessing device and a corresponding method for contrast-agent-free virtual enhanced magnetic resonance imaging so as to obtain a training data set and / or a test data set suitable for the FL model from patient data from different medical institutions, and the training data set and / or the test data set are / is used for model training and / or local verification of the FL model. A clinical evaluation result shows that the VCE-MRI model developed by the invention has high generalization and high clinical use value.

Description

Technical Field

[0001] This application generally relates to medical image diagnostic technology, and more specifically, to contrast-free virtual enhanced magnetic resonance imaging apparatus and its data preprocessing apparatus and method. Background Technology

[0002] Contrast-enhanced magnetic resonance imaging (CE-MRI), also known simply as enhanced MRI, plays an indispensable role in various clinical scenarios, such as disease diagnosis, tumor target delineation, and treatment efficacy analysis. Gadolinium contrast agent (GBCA) is a contrast agent used in MRI containing gadolinium, a highly paramagnetic metal. In MRI, GBCA helps enhance image contrast, allowing doctors to more clearly see blood vessels and their blood supply sites within the body. This gadolinium contrast agent is crucial for diagnosing diseases such as tumors and inflammation. The advent of GBCA has revolutionized MRI practice, providing doctors with invaluable lesion information and significantly improving the accuracy of disease diagnosis. GBCA is primarily used in contrast-enhanced MRI (CE-MRI) imaging; it is estimated that approximately 40% of MRI scans worldwide involve this type of imaging annually, consuming over 30 million doses of GBCA.

[0003] CE-MRI imaging requires the injection of gadolinium contrast agents; however, gadolinium contrast agents pose potential safety risks to patients. Recent studies have reported a strong association between GBCA use and the incidence of renal systemic fibrosis (NSF), a fatal fibrotic disease that occurs in patients with end-stage renal failure. Furthermore, over the past decade, researchers have observed gadolinium deposition in various tissues after GBCA use, including the brain, liver, kidneys, skin, and bones, regardless of kidney function. The mechanisms of gadolinium deposition and its long-term effects on patients remain unclear. These safety concerns have led to a ban on the use of partially linear GBCA in European countries since 2017.

[0004] To avoid the use of GBCA (Guided Magnetic Contrast Alignment), machine learning (ML)-based virtual CE-MRI (VCE-MRI) has emerged as a safe alternative. Alternatives to GBCA, utilizing machine learning (ML) methods, have been developed in this field to bypass its use. ML methods leverage multiple conventional MRI sequences (without contrast agents) containing complementary information, training neural networks to learn the mapping from contrast-free MRI sequences to CE-MRI, thereby synthesizing VCE-MRI images that achieve comparable results to real CE-MRI without contrast agents.

[0005] However, the generalization ability of ML models to external data is a major challenge in VCE-MRI technology, especially when data privacy regulations prevent cross-institutional data sharing to train highly generalizable VCE-MRI models.

[0006] While our previous research achieved promising results on internal test data, the generalization ability of ML models to new data from external institutions remains challenging. Studies have shown that including more diverse datasets, especially those that are underrepresented, can improve model generalization. However, due to patient privacy protection policies such as the US Health Insurance Portability and Accountability Act (HIPAA) and the EU General Data Protection Regulation (GDPR), collecting large-scale multi-institutional datasets to train ML models and enhance their generalization ability presents significant challenges in the healthcare field.

[0007] To address data privacy concerns, Federated Learning (FL) has been used to train multi-institutional models without explicitly sharing data. FL is a distributed model training approach that allows the integration of local model gradients from different medical centers during training, without directly sharing patient data. This ensures patient data privacy while leveraging multi-institutional data for large-scale model training. FL is particularly important in healthcare, especially for non-representative populations (limited data) and rare diseases, where individual institutions may struggle to collect sufficient data for robust model training. Compared to other medical imaging modalities, MRI data is highly heterogeneous (typically due to varying imaging parameters and equipment from different manufacturers), requiring large amounts of heterogeneous data (i.e., data collected from different MRI imaging devices and with varying imaging parameters) to develop highly generalizable models.

[0008] Chinese patent application CN118115407A discloses a system and corresponding method for virtual contrast enhancement of magnetic resonance imaging (MRI) for tumor target delineation. The system includes: a data acquisition module for acquiring MRI image data from two or more medical institutions, wherein the MRI image data includes T1w-MRI, T2w-MRI, and GBCA-enhanced CE-MRI image data without GBCA; a data preprocessing device for resampling the MRI image data to unify the images to the same size and normalizing or standardizing the resampled image data; a model training module for training a neural network model based on a training set selected from a portion of the preprocessed image data, using T1w-MRI and T2w-MRI image data from the training set as input and CE-MRI image data from the training set as the learning target; and a virtual image generation module for generating GBCA-enhanced VCE-MRI image data from GBCA-free MRI image data from external medical institutions. However, this patent application does not address the development of highly generalizable models using large amounts of heterogeneous data acquired from different MRI imaging devices at different medical institutions with different imaging parameters. Summary of the Invention

[0009] One objective of this application is to provide a safe, accurate, and highly generalizable alternative to contrast agents, replacing the use of GBCA.

[0010] To achieve this objective, this application provides a contrast-free virtual enhanced magnetic resonance imaging (VCE-MRI) device and corresponding method for patient privacy protection, based on a federated learning VCE-MRI model trained on large-scale, highly heterogeneous, multicenter data from nasopharyngeal carcinoma (NPC) patients while protecting patient data privacy. Clinical evaluation results demonstrate that the VCE-MRI model developed in this application exhibits high generalization ability and high clinical applicability.

[0011] In one aspect, this application provides a data preprocessing apparatus for virtual enhanced magnetic resonance imaging (VCE-MRI), comprising: a receiving module for receiving and storing VCE-MRI patient data files from different medical institutions, the patient data files containing different data sequences and different views; a data conversion module for converting the VCE-MRI patient data files into three-dimensional MHA files, the three-dimensional MHA files containing image arrays and basic image information, wherein identifiable patient information in the VCE-MRI patient data files is removed during the conversion; and a file selection module for selecting a desired three-dimensional MHA file from the three-dimensional MHA files, including a desired data sequence, the desired data sequence including non-fat-suppressed T1w MRI and fat-suppressed T2w MRI. The system includes: MRI and fat-suppressed CE-MRI; a resampling module that resamples each data sequence in the required data sequence to generate a resampled 3D MHA file; a registration module that applies image registration to the resampled 3D MHA file to generate a registered 3D MHA file; a slicing module that performs MHA-to-NPY conversion on the registered 3D MHA file to convert it into 2D NPY slices; a slice selection module that excludes fragmented slices from the 2D NPY slices to generate selected NPY slices without fragmented slices, ensuring end-to-end mapping of the MHA-to-NPY conversion; a normalization module that applies normalization to the selected NPY slices to generate a normalized image file; and a dataset module that divides the data in the normalized image file into a training dataset and a test dataset.

[0012] In another aspect, this application provides a data preprocessing method for virtual enhanced magnetic resonance imaging (VCE-MRI), comprising: receiving and storing VCE-MRI patient data files from different medical institutions, the patient data files containing different data sequences and different views; converting the VCE-MRI patient data files into three-dimensional MHA files, the three-dimensional MHA files containing image arrays and basic image information, wherein identifiable patient information in the VCE-MRI patient data files is removed during the conversion; selecting a desired three-dimensional MHA file from the three-dimensional MHA files, including a desired data sequence, the desired data sequence including non-fat-suppressed T1w MRI and fat-suppressed T2w MRI. The process involves: MRI and fat-suppressed CE-MRI; resampling each data sequence in the desired data sequence to generate a resampled 3D MHA file; applying image registration to the resampled 3D MHA file to generate a registered 3D MHA file; performing an MHA-NPY conversion on the registered 3D MHA file to convert it into 2D NPY slices; excluding fragmented slices from the 2D NPY slices to generate selected NPY slices without fragmented slices, ensuring end-to-end mapping of the MHA-NPY conversion; applying normalization to the selected NPY slices to generate a normalized image file; and dividing the data in the normalized image file into a training dataset and a test dataset.

[0013] In another aspect, this application provides a multimodal guided collaborative neural network MMgSN-Net for virtual enhanced magnetic resonance imaging, wherein during the training process of the multimodal guided collaborative neural network, T1w MRI and T2w MRI data from the training dataset are used as inputs to MMgSN-Net, while GBCA-based CE-MRI is used as the learning target of MMgSN-Net, wherein the training dataset is obtained according to the aforementioned data preprocessing apparatus for virtual enhanced magnetic resonance imaging.

[0014] In another aspect, this application provides a federated learning training method for an online federated learning training platform for VCE-MRI synthesis, comprising the following steps:

[0015] (i) Global Model Initialization: At the beginning, the central server of the online federated learning training platform initializes the global model weights using a normal distribution; then, the central server assigns the global model weights w g Distributed to user clients of partner organizations;

[0016] (ii) Local model training: Each user client in the user client uses the received global model weights w gInitialize the local model and train it on the local dataset for one round. After one round of training, the i-th user client in the user clients receives the updated local model weights w. i And calculate the gradient update u of the local weights of the i-th user client. i , where u i =w i -w g Then, the gradient updates of the local weights of each user client in the user clients are uploaded to the central server; and

[0017] (iii) Gradient update aggregation: The central server updates according to u g =f(u1,u2,…,u (n-1) ,u n Aggregate the gradient updates of the local model of each user client in the user clients, where f(·) represents the aggregation rule, which is FedProx; then use u g Update global model: w g =w g +α·u g , where α is the learning rate; wherein, after obtaining the updated global model, the central server sends the new global model weights to the user client, and the federated learning training method continues to iterate until the global model converges;

[0018] The local dataset is a training dataset obtained using the aforementioned data preprocessing apparatus for virtual enhanced magnetic resonance imaging.

[0019] This application also provides a computer program product including instructions that, when executed by a computer, cause the computer to perform the aforementioned data preprocessing method for virtual enhanced magnetic resonance imaging.

[0020] This application also provides a contrast-free virtual contrast-enhanced magnetic resonance imaging (VCE-MRI) device, which includes an image generation module and the aforementioned data preprocessing device for contrast-free virtual contrast-enhanced magnetic resonance imaging. The data preprocessing device preprocesses contrast-free MRI image data from different medical institutions to generate training or test datasets, thereby reducing data bias from different medical institutions. The image generation module generates contrast-free virtual contrast-enhanced magnetic resonance imaging (VCE-MRI) image data using the training or test datasets from the data preprocessing device.

[0021] Preferably, the contrast-free virtual enhanced magnetic resonance imaging (VCE-MRI) device further includes a data acquisition module and a generalization testing module; wherein, the data acquisition module acquires magnetic resonance imaging (MRI) image data from two or more medical institutions, the MRI image data including contrast-free longitudinal relaxation time-weighted magnetic resonance imaging (T1w-MRI) image data, transverse relaxation time-weighted magnetic resonance imaging (T2w-MRI) image data, and contrast-enhanced contrast-enhanced magnetic resonance imaging (CE-MRI) image data; the data acquired by the data acquisition module is input into the receiving module of the data preprocessing device for data preprocessing; the generalization testing module improves the generalization of the VCE-MRI device by collecting real cancer patient MRI data with scanning parameters from different medical institutions for training, while reducing the difference between different test datasets and training data.

[0022] Therefore, this application develops a federated learning-based VCE-MRI model to replace GBCA. Compared with existing methods, the model in this application is trained using large-scale, highly heterogeneous NPC data, which ensures high generalization of the model while protecting patient data privacy.

[0023] The solution in this application belongs to a different inventive concept and has different technical points from the aforementioned CN118115407A. CN118115407A mainly uses an image distribution matching algorithm to match the distribution of internal and external datasets, thereby improving the performance of the neural network model on the external dataset without training the model; while the solution in this application focuses on using richer data to train a machine learning model with higher generalization, which is fundamentally different from the technical solution in CN118115407A.

[0024] The foregoing summary of this application and the following detailed description of exemplary embodiments will be better understood by reading in conjunction with the accompanying drawings. Illustrative embodiments of this disclosure are shown in the drawings to illustrate this application. However, it should be understood that this application is not limited to the precise arrangements and means shown in the drawings. Attached Figure Description

[0025] Figure 1 It is a data preprocessing apparatus according to an exemplary embodiment of the present disclosure.

[0026] Figures 2 to 5 This is a screenshot of an online federated learning training platform according to an exemplary embodiment of this disclosure, wherein:

[0027] Figure 2 This is the main page of the online federated learning training platform.

[0028] Figure 3This is the control panel of the online federated learning training platform.

[0029] Figure 4 This is the real-time training monitoring page of the online federated learning training platform.

[0030] Figure 5 This is the training page of the user client of the online federated learning training platform.

[0031] Figure 6 This is a schematic diagram of a federated learning process according to an exemplary embodiment of this disclosure.

[0032] Figure 7 This is a schematic diagram of the structure of an online federated learning training platform according to an exemplary embodiment of this disclosure.

[0033] Figure 8 This includes exemplary embodiments based on this disclosure. Figure 1 The virtual enhanced magnetic resonance imaging device shown is a data preprocessing apparatus. Detailed Implementation

[0034] The technical solution of this application will now be described with reference to the accompanying drawings and specific embodiments.

[0035] The apparatus and method of this application relate to virtual enhanced magnetic resonance imaging (VCE-MRI) technology. Specific embodiments described below synthesize VCE-MRI data for nasopharyngeal carcinoma (NPC) patients using federated learning (FL) technology. The VCE-MRI technology of this invention can be used for radiotherapy target delineation in various cancers, such as nasopharyngeal carcinoma, liver cancer, breast cancer, and brain cancer.

[0036] NPC is a rare disease worldwide, ranking 22nd in incidence among all cancer types, and is primarily found in Southeast Asia, especially southern China. Given the scarcity of NPC cases globally and stringent data privacy policies, collecting large-scale and diverse heterogeneous data for centralized training of virtual enhanced magnetic resonance imaging (VMRI) techniques is extremely challenging.

[0037] By utilizing FL (Flexible Imaging), the virtual enhanced magnetic resonance imaging (fMRI) apparatus and method of this application are able to learn from multicenter heterogeneous data and gain sufficient knowledge from diverse datasets. Through FL, the virtual enhanced magnetic resonance imaging apparatus and method of this application trained the model using unprecedented multicenter heterogeneous data, including 2187 NPC patients from different institutions, using 92 MRI scanning devices covering 34 types of MRI equipment. This is the most significant difference between the research work of this application and previous work in the field.

[0038] One of the objectives of the virtual enhanced magnetic resonance imaging (VCE-MRI) apparatus and method in this application is to address the practical problem of the generalizability of VCE-MRI models. Existing high-precision but low-generalizability VCE-MRI models are only suitable for use within a specific hospital and lack universality, which significantly hinders the widespread clinical application of VCE-MRI technology. In this application's research, to verify the reliability of the synthesized VCE-MRI using the developed FL model, the synthesized VCE-MRI was further clinically evaluated by ten clinicians from eight hospitals, including seven radiologists and three oncologists. Based on the clinical evaluation, this application provides, in general, a safe, accurate, and highly generalizable contrast agent alternative to GBCA. The model in this application is trained using large-scale, highly heterogeneous NPC data while protecting patient data privacy.

[0039] The main technical details of this application are explained below with reference to the accompanying drawings, from the following three aspects: data, neural network, and FL training.

[0040] 1. Data preprocessing:

[0041] The data involved in this application are all from patients with biopsy-confirmed nasopharyngeal carcinoma. Each case includes three MRI sequences: unsuppressed T1-weighted (T1w) MRI, fat-suppressed T2-weighted (T2w) MRI, and GBCA-based fat-suppressed CE-MRI. This application used multicenter heterogeneous data to train the model. MRI images were acquired from MRI imaging equipment from different manufacturers and models, including Siemens (Skyra, Prisma, Aera, etc.), Philips (Signa Excite, Signa Pioneer, Optima MR360, etc.), GE (Ingenia, Achieva, Intera, etc.), and United Imaging (uMR 780 and uMR 790). The scanner magnetic field strengths were 1.5T and 3T.

[0042] This application used 2061 patients for training the FL model (described in detail below), and the remaining 126 patients were used for external validation to evaluate the generalization ability of the FL model. This external validation data was not involved in the training of the FL model. These 126 case data included data from 45 different MRI imaging devices, of which 23 devices (different device models) were not used during training. Therefore, this external validation set is highly representative and more realistically reflects the performance of the FL model in real-world MRI data obtained from scans performed by different devices. In each center's data involved in the training of the FL model (e.g. Figure 3In the central data (positions 1-14 shown), patient cases were randomly divided into a 4:1 ratio for model training and local validation.

[0043] Data Preprocessing: Due to the sensitivity of patient data, it cannot be taken out of the hospital or medical institution. Therefore, preprocessing of patient data is necessary before further use or analysis. Data preprocessing is carried out by staff at various medical institutions. The quality of training data is crucial for the training of the FL model in this application and directly affects its predictive performance. Considering that different operators may have different data preprocessing methods, this application provides a data preprocessing method and corresponding apparatus for virtual enhanced magnetic resonance imaging (fMRI) to obtain training and / or test datasets suitable for the FL model from patient data from different medical institutions, for use in model training and / or local validation of the FL model in this application, respectively.

[0044] like Figure 1 As shown in the data preprocessing apparatus 120, the data preprocessing method and corresponding apparatus of this application establish a standard data preprocessing workflow (HDP), which is applicable to virtual enhanced magnetic resonance imaging to ensure the consistency of post-processed data and reduce data deviation caused by different data preprocessing methods. Figure 1 This is a schematic diagram of the processing flow of the data preprocessing apparatus 120 for the FL model of this application, including the following steps:

[0045] 1.1 Data Type Conversion and Anonymization

[0046] like Figure 1 As shown, in the receiving module 101, VCE-MRI patient data files from different medical institutions are received and stored on the local workstation. These patient data files are typically in DICOM (Digital Imaging and Communications in Medicine) format, containing different data sequences and views. The data sequences include, for example, T1w, T2w, CE-MRI, DWI, ADC, STIR, etc., and the different views include, for example, axial, coronal, and sagittal views. For ease of analysis, in the data conversion module 103, each data sequence is converted into a three-dimensional MHA file, containing an array of images (a three-dimensional volume) and basic image information such as voxel size, spacing, origin, and direction. Converting data from DICOM to MHA helps eliminate identifiable patient information in the metadata. MHA is a volumetric data storage format consisting of a header describing the data and the data itself, generally used for medical images.

[0047] In the anonymization module 102, identifiable patient information is removed from a patient data file, such as a DICOM format file. This identifiable patient information can be stored in another file (e.g., an Excel file), which can be referred to as header information. This header information includes the patient's demographic information (age, gender) and imaging protocol, such as the manufacturer and model of the MRI imaging equipment, magnetic field strength, echo time, repetition time, flip angle, slice thickness, etc. This header information file is private and will be securely stored on the local machine of the relevant medical institution. It can be used, for example, for statistical analysis of patient demographic information. The virtual enhanced magnetic resonance imaging apparatus and method of this application can operate without using the header information file and the identifiable patient information therein, thus protecting the patient's privacy.

[0048] After the patient data data is converted from DICOM to MHA, the file selection module 104 selects the image files of the desired data sequence's transverse views to generate the required 3D MHA file, including the required data sequences (i.e., non-fat-suppressed T1w MRI, fat-suppressed T2w MRI, and fat-suppressed CE-MRI). This selection can be done manually by medical physicists or radiologists at various medical institutions to ensure accurate selection of the required 3D MHA file, or it can be done automatically by the file selection module.

[0049] 1.2 Resampling and Registration

[0050] After selecting the desired MRI data sequences in the required 3D MHA file, the resampling module 105 resamples each data sequence in the desired MRI data sequence to an appropriate size, such as 256*256, generating a resampled 3D MHA file. Then, in the registration module 106, potential sequence mismatch problems caused by respiratory motion are resolved by applying, for example, rigid registration. Image registration is generally divided into rigid registration and non-rigid registration. Rigid registration mainly solves simple overall image movement problems (such as translation, rotation, etc.); non-rigid registration mainly solves flexible image transformation problems, which allows the positional relationship between any two pixels to change during the transformation. The registration module of this application uses rigid registration, where T1w MRI is used as the reference image, and T2w MRI and CE-MRI are used as moving images to generate the registered 3D MHA file 106.

[0051] After rigid registration, in slicing module 107, the registered 3D MHA file undergoes MHA-to-NPY conversion to convert it into 2D NPY slices. The NPY file is a NumPy array file created by a Python package with the NumPy library installed. It contains an arrangement saved in NumPy(NPY) file format. Since some slices may lose information due to rotation during rigid registration, becoming broken slices, the slice selection module 108 can select from the 2D NPY slices obtained in slicing module 107 to exclude broken slices, producing NPY slices without broken slices, thus ensuring accurate end-to-end mapping of the MHA-to-NPY conversion. Medical physicists or radiologists in various medical institutions can manually select 2D NPY slices in slice selection module 108 to exclude broken slices; alternatively, the slice selection module can automatically select and exclude broken slices.

[0052] 1.3 Data Standardization

[0053] After excluding fragmented slices with missing information, standardization is applied in standardization module 109 to address the deviation in pixel intensity distribution of MRI images between different medical institutions, generating standardized image files. Based on past experience, deviations in image pixel intensity distribution between medical institutions or patients can significantly affect the generalization ability of multi-institution models. To mitigate this problem, Z-Score normalization based on individual patients is employed to ensure that the mean of MRI data for each patient in the VCE-MRI patient data file is 0 and the standard deviation is 1. Z-Score normalization, also known as standard deviation normalization, transforms each data point to a standard normal distribution interval with a mean of 0 and a standard deviation of 1 by subtracting the mean from each data point and then dividing by the standard deviation. The applicant's research demonstrates that Z-Score normalization is effective in reducing the deviation in image pixel intensity distribution of MRI data.

[0054] 1.4 Training / Test Dataset

[0055] As described above, for each center data (e.g., for the FL model training involved in this application) Figure 3 The data in the various tasks (named according to their start time) shown in the dataset module 110 are generated by randomly dividing the data in the standardized images 109 obtained from each patient case according to the above processing in, for example, a 4:1 ratio, to produce training datasets and test datasets, respectively, for model training and local validation of the FL model of this application.

[0056] 2. Neural Networks:

[0057] The neural network used in this application is a multimodal guided synergistic neural network (MMgSN-Net), which is a neural network previously developed in this application, and its performance has been confirmed in multiple studies. The training model of MMgSN-Net can be found in the literature published by the inventors of this application: Li W, Xiao H, Li T et al. "Virtual Contrast-enhanced Magnetic Resonance Images Synthesis for Patients with Nasopharyngeal Carcinoma using Multimodality-guided Synergistic Neural Network", Int. J. Radiat. Oncol. 2021; 112(4): 1033-1044, the entire contents of which are incorporated herein by reference.

[0058] During the training of MMgSN-Net, the T1w and T2w MRI data from the aforementioned training dataset were used as inputs to MMgSN-Net, while GBCA-based CE-MRI was used as the learning target. MMgSN-Net used a learning rate of 0.001 and was optimized using the Adam (Adaptive Moment Estimation) optimizer. To handle negative values ​​resulting from Z-score normalization, a LeakyReLU activation function was used after each convolutional layer to prevent truncation of negative values.

[0059] 3. FL training:

[0060] The FL global model of MMGSN-Net was trained using the online federated learning training platform developed in this application, such as... Figures 2-5 As shown, the training process is protected by the firewall of the Hong Kong Polytechnic University (PolyU) in Hong Kong, China. The FL training process is as follows: Figure 6 As shown.

[0061] Figures 2-5 This is a schematic diagram of an online federated learning training platform according to an embodiment of the present disclosure.

[0062] Figure 2 This is the main page of the online federated learning training platform: Administrators can access the control panel by clicking the "Login" button. Partner organizations can download the MMgSN-Net user client to their local workstations by clicking the "Download" button.

[0063] Figure 3 This is the control panel of the online federated learning training platform: Clicking the "Add new task" button opens the configuration setting window, where administrators can enter training parameters. After setting the federated learning training parameters, clicking the "Start Training" button starts a new federated learning training task.

[0064] Figure 4 This is the real-time training monitoring page of the online federated learning training platform: by clicking... Figure 3 The "View details" button for new training tasks allows administrators to monitor the real-time changes in the loss function of each collaborating institution during model training.

[0065] Figure 5 This is the training page of the user client of the online federated learning training platform: after configuring the local training parameters and clicking... Figure 5 After clicking the "Connect Server" button at the bottom of the left column, the local model connects to the server, and the federated learning model will automatically start training after all local clients connect.

[0066] Figure 6 This is a schematic diagram of a federated learning training method according to an embodiment of this disclosure. Figure 6 As shown, once all collaborating medical institutions securely connect to the central server CS located on PolyU, training of the FL global model of MMgSN-Net in this application automatically begins. At the start of training, the central server CS initializes the weights of the neural network using a normal distribution, and then passes the initialized weights to the local models of each collaborating institution. The training process of the FL global model in this application is as follows... Figure 6 As described, the FL global model of MMgSN-Net is not a training platform, but rather utilizes a training platform through... Figure 6 The model obtained by the described training method.

[0067] like Figure 6 As shown, after receiving the initialized weights from the central server CS, each cooperating institution S1, S2, S3, ..., S... 14 Using the received weights W1, W2, W3, ..., W 14 Instantiate the same neural network and perform a round of local training using their respective local data. After the collaborating institutions complete a round of local training, update the trained weight gradients u1, u2, u3, ..., u 14 The data will be uploaded to the central server CS for aggregation.C Once the central server (CS) receives the weight gradient updates from all collaborating institutions, it aggregates these updates and sends the current global model back to each collaborating institution for the next round of training. This process is called a "communication round," representing one iteration of federated training.

[0068] Figure 7 This is a schematic diagram of the structure of an online federated learning training platform according to an exemplary embodiment of this disclosure. Figure 7 As shown, the structure of the online federated learning training platform 701 in this application includes a central server CS 702 and a user client 703. The central server CS 702 mainly includes four modules, namely the account login module 703 (e.g., ... Figure 2 (as shown), client download module 704 (see...) Figure 2 The “Download client” and Training Task Management 705 (see...) are shown. Figure 3 The "Add new task" → "configuration setting" and the training progress monitoring module 706 (see...) are shown. Figure 4 The “view details” shown is shown, and the user client 703 includes a model training module (see [link]). Figure 5 The "Training" shown in this application. Figure 7 The online federated learning training platform 701 shown can be independent of Figure 1 The data preprocessing device shown is used to process image data collected from hospitals or medical institutions. Figure 1 After the data preprocessing device shown performs data preprocessing, it is placed into... Figure 7 Training is conducted on the federated learning platform shown.

[0069] The gradient update aggregation algorithm used in this application is FedProx, which outperforms the FedAvg algorithm (currently the most classic federated learning algorithm) when dealing with differences in equipment used for local model training among collaborating institutions and heterogeneity in training data. To ensure good convergence of the global model, a total of 100 communication rounds of training were performed.

[0070] like Figure 6 As shown, the federated learning includes three main steps: the first step is global model initialization, the second step is local model training, and the third step is gradient update aggregation.

[0071] Step 1: Initially, the central server (CS) initializes the global model weights using a normal distribution. Then, the central server (CS) assigns the global model weights (w...) to the global model weights. gDistribute to user clients S1, S2, S3, ..., S14 of partner organizations.

[0072] Step 2: Each user client S1, S2, S3, ..., S14 uses the received global model weights (w g Initialize the local model and train it on the local dataset for one round. After one round of training, the i-th client receives the updated local model weights (w). i ), and compute the gradient update of the local weights (u i ), where u i =w i -w g Then, the gradient of the local weights is updated u. i It was uploaded to the central server CS.

[0073] Step 3: The central server, based on u g =f(u1,u2,…,u (n-1) ,u n Aggregate gradient updates of the local model (u) i ), where f(·) represents the aggregation rule; this application uses FedProx. Then, u g Update global model: w g =w g +α·u g α is the learning rate. After obtaining the updated global model, the central server CS sends the new global model weights to the user clients S1, S2, S3, ..., S14 of the collaborating institutions. This iterative process continues until the global model converges.

[0074] The aforementioned data preprocessing method, the multimodal guided collaborative neural network MMgSN-Net, or the online federated learning training platform can all be implemented as computer program products, with the instructions in the computer program product executing the data preprocessing method and implementing the MMgSN-Net or the online federated learning training platform.

[0075] Figure 8 This is a virtual enhanced magnetic resonance imaging apparatus 100 according to an exemplary embodiment of the present disclosure, the apparatus 100 including Figure 1 The data preprocessing device 120 shown.

[0076] The applicant's earlier application CN118115407A discloses a system and method for virtual contrast enhancement of magnetic resonance imaging (MRI) for tumor target delineation, the entire contents of which are incorporated herein by reference. The virtual contrast enhancement MRI apparatus of this application substantially corresponds to the system for virtual contrast enhancement of MRI for tumor target delineation in the aforementioned earlier application, but employs the techniques described in this application. Figure 1The data preprocessing apparatus 120 shown replaces the preprocessing module in the system of the prior application to improve the preprocessing of VCE-MRI patient data files from different medical institutions, thereby obtaining better virtual contrast-enhanced magnetic resonance imaging. (From this application) Figure 1 The training and test datasets generated by the dataset module 110 can be input into... Figure 8 The image generation module 140 generates a magnetic resonance virtual contrast-enhanced image.

[0077] like Figure 8 As shown, the virtual enhanced magnetic resonance imaging device of this application includes: Figure 1 The data preprocessing device 120 shown is suitable for virtual contrast-enhanced magnetic resonance imaging (VCE-MRI) and preprocesses contrast-free MRI image data from different medical institutions to generate training or test datasets to ensure consistency of post-processed data and reduce data bias from different medical institutions; and the image generation module 140 is capable of generating virtual contrast-enhanced magnetic resonance imaging (VCE-MRI) image data from the training or test datasets from the data preprocessing device 120.

[0078] like Figure 8 As shown, the virtual enhanced magnetic resonance imaging device of this application may optionally include a data acquisition module 111 and a generalization test module 142.

[0079] The data acquisition module 111 acquires magnetic resonance imaging (MRI) image data from two or more medical institutions. The MRI image data includes contrast-free longitudinal relaxation time-weighted MRI (T1w-MRI) image data, transverse relaxation time-weighted MRI (T2w-MRI) image data, and contrast-enhanced contrast-enhanced MRI (CE-MRI) image data. The data acquired by the data acquisition module 111 can be input to... Figure 1 In the receiving module 101, for data preprocessing.

[0080] exist Figure 8In this invention, the data acquisition module 111 acquires MRI images under various scanning conditions from different medical institutions. Scanning conditions include, but are not limited to, magnetic field strength, type of radio frequency coil, spatial resolution of the image, phase encoding step level, use of fast scan sequences, scan time, TR, TE, NSA, and combinations thereof. In some embodiments of this invention, contrast-free T1w-MRI, T2w-MRI, and contrast-enhanced CE-MRI image data of nasopharyngeal carcinoma patients are collected from different medical institutions. These medical institutions use different imaging equipment, including MRI scanning equipment from companies such as General Electric, Philips, and Siemens.

[0081] The data preprocessing device resamples the MRI image data to unify images of different sizes to the same size, and normalizes or standardizes the resampled image data. The normalization or standardization includes normalization or standardization based on the entire image dataset, based on a single image data, or based on a single patient image data.

[0082] As described in the prior application above, the virtual enhanced magnetic resonance imaging device of this application may further include a model training module. The model training module uses T1w-MRI and T2w-MRI image data from the training dataset or test dataset generated in the dataset module 110 as input to the neural network model, and uses CE-MRI image data from the training dataset or test dataset as the learning target of the neural network model to train the neural network model.

[0083] As described in the prior application above, in order to increase the generalization of the model, the virtual enhanced magnetic resonance imaging device of this application may also include a generalization test module 142, which can be used for training by collecting more real cancer patient MRI data with scanning parameters from different medical institutions, and at the same time reduce the difference between different test datasets and training data to improve the generalization of the model, which will not be elaborated here.

[0084] Based on the above description, the virtual enhanced magnetic resonance imaging device and method of this application mainly involve the following aspects:

[0085] Application Level: This application primarily addresses a practical problem encountered in the clinical application of VCE-MRI technology: insufficient model generalization. Currently, no other work besides this application has been published aimed at improving the generalization of VCE-MRI models. This application aims to improve the generalization of VCE-MRI models. Figure 6The described federated learning process involved the inventors of this application collaborating with multiple medical institutions to train the FL model. Simultaneously, a single-center model was trained using local data from each participating institution. Image data from institutions not participating in the federated learning model training were referred to as external datasets. Both the single-center and FL models were tested on the external datasets, with mean absolute error (MAE) used as the evaluation metric. This application found that on the external datasets, the FL model improved the MAE score by 21.23% compared to the single-center model; this improvement was based on MAE as the evaluation standard.

[0086] Technically: While existing research teams have applied federated learning techniques to the medical field, this application is the first to apply federated learning to VCE-MRI synthesis, thereby enhancing the model's generalization ability and addressing patient data privacy concerns. This application is the first to apply federated learning techniques to the VCE-MRI synthesis task. In other words, utilizing... Figures 2-5 The online federated learning platform shown is an application. Figure 6 The federated learning method shown can produce a global FL model, which can be used for VCE-MRI synthesis.

[0087] At the data level: Thanks to federated learning technology, this application collaborated with multiple medical institutions, ensuring that the data did not leave the hospital or medical institution. This protected patient data privacy while enabling the use of a large amount of multi-center NPC patient data (2187 cases) to train the global model. The resulting FL model showed a significant improvement in generalization ability compared to single-center models. This application used NPC data from multiple centers to develop the FL model; to date, no other research in the field has used NPC data from so many centers to train a model.

[0088] Hardware and Software: To implement federated learning, this application used an internal laboratory server to build its own federated learning platform (MNPP-VCE), i.e. Figures 2-5 The training platform shown.

[0089] This platform is located within the Hong Kong Polytechnic University (PolyU) in Hong Kong, China, and is managed by members of the inventors' team. The network connection is protected by the PolyU firewall. This application independently built a federated learning training platform for real-world federated learning training. Unlike most published federated learning works, which involve splitting a specific dataset and using a single workstation to simulate federated learning training, this application demonstrates a different approach.

[0090] Compared to single-center models, the federated learning model in this application can be trained using multi-center data. Thanks to the increased amount of data and the heterogeneity of multi-center data, the trained FL model has higher accuracy and generalization ability, and will be more reliable and have greater clinical value in real clinical use.

[0091] This application relates to the application of federated learning in VCE-MRI, specifically for synthesizing VCE-MRI of nasopharyngeal carcinoma patients. Using MMgSN-Net as the network, the model in this application is trained with T1w MRI and T2w MRI as inputs and CE-MRI as the ground truth. Figures 1 to 8 The description illustrates the technical process for developing FL models for synthesizing VCE-MRI. This process can be applied to the synthesis of VCE-MRI for different types of cancer.

[0092] To further improve the accuracy and generalization of the model, this application may collect more patient data in the future, develop better-performing deep neural networks, and use better federated learning aggregation algorithms to further improve the predictive performance of VCE-MRI. In addition to its application to nasopharyngeal carcinoma patients, this application will also collect data from other cancer patients, such as brain cancer and liver cancer, for model training, applying the device and method of this application to different cancer types.

[0093] The current technical process described in this application is a FL model for nasopharyngeal carcinoma. This FL model has been completed, and subsequent data collection on other cancer types will easily extend it to other diseases, such as liver cancer and brain cancer. Future technological developments, such as neural networks and federated learning aggregation algorithms, can also be easily updated based on the current framework to achieve better predictive results.

[0094] To provide a more comprehensive understanding of the concepts of this application, numerous specific details of embodiments thereof have been described above. However, it will be apparent to those skilled in the art that the inventive concepts within this disclosure can be practiced without these specific details. In other instances, well-known features have not been described in detail to avoid unnecessarily complicating this disclosure.

[0095] As used herein, any reference to "an embodiment" or "embodiment" means that a particular element, feature, structure, or characteristic described in connection with the embodiment is included in at least one implementation. The appearance of the phrase "in an embodiment" in various places in the specification does not necessarily refer to the same embodiment. Furthermore, it is understood that features of one embodiment may be combined with features of other embodiments, even if not explicitly stated or described as a combination.

Claims

1. A data preprocessing apparatus for contrast-free virtual enhanced magnetic resonance imaging, comprising: The receiving module is used to receive and store virtual enhanced magnetic resonance imaging (VCE-MRI) patient data files from different medical institutions, wherein the patient data files contain different data sequences and different views; The data conversion module is used to convert the VCE-MRI patient data file into a three-dimensional MHA file, the three-dimensional MHA file containing an image array and basic image information, wherein the conversion removes identifiable patient information from the VCE-MRI patient data file; The file selection module selects the desired three-dimensional MHA file from the three-dimensional MHA files, including the required data sequence, which includes non-fat-suppressed T1w MRI, fat-suppressed T2w MRI, and fat-suppressed contrast-enhanced magnetic resonance imaging (CE-MRI). The resampling module resamples each data sequence in the required data sequence to generate a resampled 3D MHA file; The registration module applies image registration to the resampled 3D MHA file and generates a registered 3D MHA file. The slicing module performs MHA-to-NPY conversion on the registered 3D MHA file to convert the registered 3D MHA file into a 2D NPY slice. The slice selection module is used to exclude fragmented slices in the two-dimensional NPY slices to generate NPY slices without fragmented slices, ensuring the end-to-end mapping of MHA to NPY conversion; The standardization module applies standardization to process the selected NPY slices to produce a standardized image file; as well as The dataset module divides the data in the standardized image files into a training dataset and a test dataset.

2. The data preprocessing apparatus according to claim 1 further includes an anonymization module for removing identifiable patient information from the patient data file and storing the identifiable patient information in a header information file, wherein, The header information file is private and is securely stored on the local machine of the relevant medical institution.

3. The data preprocessing apparatus of claim 1, wherein the VCE-MRI patient data file is in DICOM format, and the registration module applies rigid registration to the resampled three-dimensional MHA file, wherein T1w MRI is used as a reference image, and T2w MRI and CE-MRI are used as moving images.

4. The data preprocessing apparatus of claim 1, wherein the standardization is performed using Z-Score normalization based on a single patient to ensure that the mean MRI data of each patient in the VCE-MRI patient data file is 0 and the standard deviation is 1.

5. The data preprocessing apparatus according to claim 1, wherein the dataset module randomly divides the data in the standardized image file into a training dataset and a test dataset at a ratio of 4:

1.

6. A data preprocessing method for contrast-free virtual-enhanced magnetic resonance imaging, comprising: Receive and store virtual enhanced magnetic resonance imaging (VCE-MRI) patient data files from different medical institutions, the patient data files containing different data sequences and different views; The VCE-MRI patient data file is converted into a three-dimensional MHA file, which contains an array of images and basic image information, wherein identifiable patient information in the VCE-MRI patient data file is removed during the conversion. Select the desired 3D MHA file from the 3D MHA files, which includes the desired data sequences, including non-fat-suppressed T1w MRI, fat-suppressed T2w MRI, and fat-suppressed CE-MRI. Each data sequence in the required data sequence is resampled to produce a resampled 3D MHA file; Image registration is applied to the resampled 3D MHA file to generate a registered 3D MHA file; The registered 3D MHA file is converted to NPY to convert the registered 3D MHA file into a 2D NPY slice. Fragmented slices in the two-dimensional NPY slices are excluded to generate selected NPY slices that do not contain fragmented slices, ensuring end-to-end mapping of MHA to NPY conversion; The selected NPY slices are processed using standardization to produce a standardized image file; as well as The data in the standardized image files are divided into training datasets and test datasets.

7. The data preprocessing method according to claim 6 further includes removing identifiable patient information from the patient data file and storing the identifiable patient information in a header information file, wherein, The header information file is private and is securely stored on the local machine of the relevant medical institution.

8. The data preprocessing method of claim 6, wherein the VCE-MRI patient data file is in DICOM format, and the image registration is a rigid registration applied to the resampled three-dimensional MHA file, wherein T1w MRI is used as a reference image, and T2w MRI and CE-MRI are used as moving images.

9. The data preprocessing method of claim 6, wherein the standardization is performed using Z-Score normalization based on a single patient to ensure that the mean of the MRI data for each patient in the VCE-MRI patient data file is 0 and the standard deviation is 1.

10. The data preprocessing method of claim 6, wherein dividing the data in the standardized image file into a training dataset and a test dataset is done randomly in a 4:1 ratio.

11. A multimodal guided collaborative neural network MMgSN-Net for contrast-free virtual enhanced magnetic resonance imaging, wherein during the training process of the multimodal guided collaborative neural network, T1w MRI and T2w MRI data from the training dataset are used as inputs to MMgSN-Net, while GBCA-based CE-MRI is used as the learning target of MMgSN-Net. in, The training dataset is obtained using the data preprocessing apparatus for virtual enhanced magnetic resonance imaging according to any one of claims 1-5.

12. The multimodal guided collaborative neural network of claim 11, wherein a learning rate of 0.001 is used, and optimization is performed using the Adam optimizer; wherein, To handle negative values ​​resulting from Z-Score normalization, the LeakyReLU activation function is used after each convolutional layer to prevent negative values ​​from being truncated.

13. A federated learning training method for an online federated learning training platform for contrast-free virtual enhanced magnetic resonance imaging (VCE-MRI) synthesis, comprising the following steps: (i) Global Model Initialization: At the beginning, the central server of the online federated learning training platform initializes the global model weights using a normal distribution; then, the central server assigns the global model weights w g Distributed to user clients of partner organizations; (ii) Local model training: Each user client in the user client uses the received global model weights w g Initialize the local model and perform one round of training on the local dataset; After one round of training, the i-th user client in the user clients obtains the updated local model weights w. i And calculate the gradient update u of the local weights of the i-th user client. i , where u i =w i -w g Then, the gradient update of the local weights of each user client in the user client is uploaded to the central server. as well as (iii) Gradient update aggregation: The central server updates according to u g =f(u1,u2,…,u (n-1) ,u n Aggregate the gradient updates of the local model of each user client in the user clients, where f(·) represents the aggregation rule, which is FedProx; then use u g Update global model: w g =w g +α·u g , where α is the learning rate; wherein, after obtaining the updated global model, the central server sends the new global model weights to the user client, and the federated learning training method continues to iterate until the global model converges; The local dataset is the training dataset obtained by the data preprocessing apparatus for virtual enhanced magnetic resonance imaging according to any one of claims 1-5.

14. A computer program product comprising instructions that, when executed by a computer, cause the computer to perform the data preprocessing method for contrast-free virtual-enhanced magnetic resonance imaging as described in any one of claims 6-10.

15. A contrast-free virtual enhancement magnetic resonance imaging (VCE-MRI) device, the VCE-MRI device comprising an image generation module and a data preprocessing device for contrast-free virtual enhancement magnetic resonance imaging according to any one of claims 1-5, wherein, The data preprocessing device preprocesses contrast-free MRI image data from different medical institutions to generate training or test datasets, thereby reducing data bias from different medical institutions; the image generation module generates contrast-free virtual contrast-enhanced magnetic resonance imaging (VCE-MRI) image data using the training or test datasets from the data preprocessing device.

16. The contrast-free virtual enhanced magnetic resonance imaging (VCE-MRI) device according to claim 15 further includes a data acquisition module and a generalization testing module; wherein, The data acquisition module acquires MRI image data from two or more medical institutions. The MRI image data includes contrast-free longitudinal relaxation time-weighted MRI T1w-MRI image data, transverse relaxation time-weighted MRI T2w-MRI image data, and contrast-enhanced contrast-enhanced MRI (CE-MRI) image data. The data acquired by the data acquisition module is input into the receiving module of the data preprocessing device for data preprocessing. The generalization testing module improves the generalization of the VCE-MRI device by collecting real cancer patient MRI data with scanning parameters from different medical institutions for training, while simultaneously reducing the difference between different test datasets and training data.

Images