An MRI image enhancement method and system based on multi-modal fusion

Abstract

The present invention provides a method and system for MRI image enhancement based on multimodal fusion, belonging to the technical field of image enhancement, including establishing a medical image database and obtaining diagnostic texts; obtaining original MRI image data and performing structured processing on T1W / T2W and ADC image data; establishing a fusion enhancement algorithm model pool, where the fusion enhancement algorithm model pool includes a forward enhancement model, a reverse optimization model, and a CT-assisted enhancement model, and performing a matching operation; if the matching is successful, enabling the CT-assisted enhancement model and performing fusion enhancement with the forward enhancement model; if the matching fails, separately enabling the forward enhancement model for fusion enhancement; forwarding the enhanced MRI image data to the medical image database and simultaneously outputting it in reverse to the reverse optimization model, and periodically updating and optimizing the forward enhancement model and the CT-assisted enhancement model; providing a method for selectively enhancing MRI images based on CT images.

Description

Technical Field

[0001] The present invention belongs to the technical field of image enhancement, and more specifically, particularly relates to a method and system for enhancing MRI images based on multi-modal fusion. Background Art

[0002] In the field of modern medical image diagnosis, MRI has become an important means for diagnosing various diseases such as brain diseases, neurological diseases, cardiovascular diseases, and tumors due to its significant advantages such as no radiation, high soft tissue resolution, and multi-directional imaging. By analyzing MRI images, doctors can clearly observe the morphology, structure, and functional changes of human tissues and organs, thus providing key evidence for the early detection, accurate diagnosis, and effective treatment of diseases. However, during the MRI imaging process, it is susceptible to various factors such as magnetic field inhomogeneity and radio frequency noise, resulting in problems such as low contrast, blurring, and artifacts in the images. Traditional MRI image enhancement methods often cannot selectively enhance MRI images based on CT images, and the enhancement effect is weak. Summary of the Invention

[0003] To solve the above technical problems, the present invention provides a method and system for enhancing MRI images based on multi-modal fusion to solve the technical problems in the prior art that traditional MRI image enhancement methods often cannot selectively enhance MRI images based on CT images and the enhancement effect is weak.

[0004] The purpose and efficacy of a method and system for enhancing MRI images based on multi-modal fusion of the present invention are achieved by the following specific technical means:

[0005] A method for enhancing MRI images based on multi-modal fusion includes the following steps:

[0006] Establish a medical image database for storing CT image data, enhanced MRI image data, original MRI image data, and historical medical image data, establish a cross-regional data circulation mechanism, and obtain diagnostic texts;

[0007] Based on the medical image database, obtain the original MRI image data, which includes T1W / T2W and ADC image data, and perform structured processing on the T1W / T2W and ADC image data based on the diagnostic texts;

[0008] Establish a fusion enhancement algorithm model pool, which includes a forward enhancement model, a reverse optimization model, and a CT-assisted enhancement model. Train the relevant models in the model pool based on the historical medical image data in the medical image database, and perform matching operations based on the diagnostic texts and the medical image database;

[0009] If the matching is successful, enable the CT-assisted enhancement model, and fuse and enhance the structured T1W / T2W and ADC image data with the forward enhancement model to obtain enhanced MRI image data;

[0010] If the matching fails, separately enable the forward enhancement model to fuse and enhance the structured T1W / T2W and ADC images to obtain enhanced MRI image data;

[0011] Forward output the enhanced MRI image data to the medical image database, and at the same time reverse output it to the reverse optimization model to generate updated parameters, and regularly update and optimize the forward enhancement model and the CT-assisted enhancement model.

[0012] As a further solution of the present invention, separately enabling the forward enhancement model to fuse and enhance the structured T1W / T2W image data and ADC image data includes:

[0013] The forward enhancement model extracts the structural feature data in the structured T1W image data based on the 3D U-Net network, extracts the texture feature data in the structured T2W image data based on the 3D ResNet network, and extracts the quantitative feature data in the structured ADC image data based on the lightweight CNN;

[0014] Dynamically assign weights to the structural feature data, texture feature data, and quantitative feature data;

[0015] Based on the dynamic weights of the structural feature data, texture feature data, and quantitative feature data and the forward enhancement target loss function, perform image enhancement operations on the structured T1W / T2W and ADC image data, and generate enhanced MRI image data based on convolutional fusion.

[0016] As a further solution of the present invention, enabling the CT-assisted enhancement model and fusing and enhancing the structured T1W / T2W and ADC image data with the forward enhancement model includes:

[0017] Based on the forward enhancement model and the CT-assisted enhancement model, perform alternating enhancement on the structured T1W / T2W and ADC image data, and perform a total of six rounds of image enhancement operations;

[0018] The first five rounds of image enhancement operations are performed on the structured T1W / T2W and ADC image data based on the forward enhancement model. After the enhancement is completed, obtain the enhanced T1W / T2W and ADC image data;

[0019] When performing the sixth round of image enhancement operation, call the CT image data that matches the T1W / T2W and ADC image data, extract the CT feature data in the CT image data based on the CT-assisted enhancement model, and perform secondary enhancement on the enhanced T1W / T2W and ADC image data according to the CT feature data, and generate enhanced MRI image data based on convolutional fusion.

[0020] As a further solution of the present invention, and perform secondary enhancement on the enhanced T1W / T2W and ADC image data according to the CT feature data, including:

[0021] Assign weights to the CT feature data and the enhanced T1W / T2W and ADC image data. The weight of the CT feature data is 20%, and the weight of the T1W / T2W and ADC image data is 80%. Select and retain the structural feature data, texture feature data, and quantitative feature data in the T1W / T2W and ADC image data, and iteratively adjust the feature fusion ratio of the CT feature data with the structural feature data, texture feature data, and quantitative feature data based on the joint optimization objective function.

[0022] As a further solution of the present invention, perform a matching operation based on the diagnostic text and the medical image database, including:

[0023] Query the corresponding CT image data in the medical image library based on the patient's unique identity ID;

[0024] If the patient's unique identity ID stores corresponding CT image data in the medical image database, and the interval between the CT image data and the original MRI image data is less than or equal to 7 days, enter the registration operation;

[0025] If the interval between the CT image data and the original MRI image data is greater than seven days or there is no corresponding CT image data in the medical image database, it is determined that the matching fails;

[0026] The registration operation means registering the CT image data with the original MRI image data;

[0027] If the registration of the CT image data and the original MRI image data is successful, that is, the Dice coefficient is greater than or equal to 0.85, it is determined that the CT image data and the original MRI image data match successfully;

[0028] If the registration of the CT image data and the original MRI image data fails, that is, the Dice coefficient is less than 0.85, it is determined that the CT image data and the original MRI image data match fails.

[0029] As a further solution of the present invention, the registration operation means registering the CT image data with the original MRI image data, including:

[0030] The registration operation includes rigid registration and elastic registration;

[0031] The rigid registration based on translation operation and rotation operation performs rigid adjustment on the CT image data, so that the CT image data is roughly aligned with the original MRI image data;

[0032] After the rigid registration is completed, the elastic registration based on the B-spline algorithm performs deformation compensation on the CT image data, ensures that the lesion areas of the CT image data and the original MRI image data are aligned, and at the same time, obtains the Dice coefficient for elastic registration.

[0033] As a further solution of the present invention, the T1W / T2W and ADC image data are structurally processed based on the diagnostic text, including:

[0034] Based on the diagnostic text, the unique patient identity ID is obtained. Based on the unique patient identity ID, the original MRI image data in the medical image database is retrieved. Artifact detection is performed on the T1W / T2W and ADC image data in the original MRI image data. Artifact repair is performed on the T1W / T2W and ADC image data based on non-local means denoising operation and interpolation algorithm. Normalization operation is performed on the T1W / T2W and ADC image data, and the T1W / T2W and ADC image data are synchronously adjusted to the same resolution, so as to generate the processed T1W / T2W and ADC image data;

[0035] Based on natural language processing algorithms, word segmentation and entity recognition are performed on the diagnostic text, keywords of lesion areas and keywords of lesion degrees are extracted. An image space coordinate range matrix is generated based on the keywords of lesion areas, and a dynamic space constraint is generated based on the keywords of lesion degrees. The image space coordinate range matrix is data-constrained by the dynamic space constraint, and the constrained image space coordinate range matrix is mapped to the processed T1W / T2W and ADC image data, so as to mark the lesion areas and generate structured T1W / T2W and ADC image data.

[0036] As a further solution of the present invention, a cross-region data circulation mechanism is established, including:

[0037] The medical image database is divided into four independent storage areas. The data in each independent storage area is interrelated and cross-storage area data circulation can be carried out. The medical image database is interrelated with the external case system, and the diagnostic text in the external case system can be called. Users can also call the image data stored in the four independent storage areas in the medical image database based on the external case system, so as to view CT images, enhanced MRI images, original MRI images and historical medical images.

[0038] As a further solution of the present invention, the medical image database is divided into four independent storage areas, including:

[0039] The four independent storage areas are respectively a CT storage area, a raw storage area, an enhanced storage area and a historical storage area. The CT storage area is used to store CT images, the raw storage area is used to store raw MRI images, the enhanced storage area is used to store enhanced MRI images, and the historical storage area is used to store historical medical images;

[0040] When storing medical image data, the CT images, enhanced MRI images and raw MRI images are regularly transferred from the corresponding independent storage areas to the historical storage area, and at the same time, the image data of the CT images, enhanced MRI images and raw MRI images are deleted;

[0041] The above transfer and deletion operations are performed every 30 days.

[0042] An MRI image enhancement system based on multimodal fusion includes:

[0043] A database module, which is used to establish a medical image database. The medical image database is used to store CT image data, enhanced MRI image data, raw MRI image data and historical medical image data, and the medical image database is interconnected with an external case system and can call the diagnostic text in the external case system;

[0044] An acquisition module, which is used to acquire relevant data in the medical image database;

[0045] A matching module, which is used to perform matching operations;

[0046] A processing module, which is used to perform structuring operations on T1W / T2W and ADC image data to obtain structured T1W / T2W and ADC image data;

[0047] An enhancement module, which is used to establish a fusion enhancement algorithm model pool. The fusion enhancement algorithm model pool includes a forward enhancement model, a reverse optimization model and a CT-assisted enhancement model. Based on the forward enhancement model and the reverse optimization model, the structured T1W / T2W and ADC image data are fused and enhanced to obtain enhanced MRI image data. Based on the reverse optimization model, update parameters are generated, and the forward enhancement model and the CT-assisted enhancement model are updated and optimized regularly;

[0048] A training module, which is used to train relevant models in the fusion enhancement algorithm model pool.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] First, establish a medical image database and a cross-region data circulation mechanism so that the medical image database can be associated with an external case system. Obtain diagnostic texts through the external case system. Then, based on the medical image database, obtain the original MRI image data, which includes T1W / T2W and ADC image data. Perform structured processing on the T1W / T2W and ADC image data based on the diagnostic texts. Through structured processing, the T1W / T2W and ADC image data can be made more convenient for subsequent enhancement operations. Subsequently, establish a fusion enhancement algorithm model pool, which includes a forward enhancement model, a reverse optimization model, and a CT-assisted enhancement model. Perform matching operations based on the diagnostic texts and the medical image database; if the matching is successful, enable the CT-assisted enhancement model and perform fusion enhancement on the structured T1W / T2W and ADC image data with the forward enhancement model to obtain enhanced MRI image data; if the matching fails, separately enable the forward enhancement model to perform fusion enhancement on the structured T1W / T2W and ADC images to obtain enhanced MRI image data; output the enhanced MRI image data forward to the medical image database and simultaneously output it backward to the reverse optimization model to generate updated parameters. Regularly update and optimize the forward enhancement model and the CT-assisted enhancement model. Through this method, the original MRI images can be enhanced selectively based on CT images, and when the CT-assisted enhancement model and the forward enhancement model perform fusion enhancement on the original MRI image data with matching CT image data, the enhancement effect is improved. Description of the Drawings

[0051] Figure 1 is a flowchart of the steps of a method for enhancing MRI images based on multi-modal fusion according to the present invention;

[0052] Figure 2 is a data flow diagram of a method for enhancing MRI images based on multi-modal fusion according to the present invention;

[0053] Figure 3 is a schematic diagram of a system for enhancing MRI images based on multi-modal fusion according to the present invention. Detailed Embodiments

[0054] The following further describes in detail the embodiments of the present invention in conjunction with the drawings and embodiments. The following embodiments are used to illustrate the technical solutions of the present invention, but cannot be used to limit the protection scope of the present invention.

[0055] Embodiment 1: As shown in the appendix Figure 1 、 Figure 2 shown:

[0056] The present invention provides a method for enhancing MRI images based on multi-modal fusion, which is applicable to image enhancement and includes the following steps:

[0057] S101: Establish a medical image database for storing CT image data, enhanced MRI image data, original MRI image data, and historical medical image data. Establish a cross-region data circulation mechanism to obtain diagnostic texts.

[0058] Specifically, establishing a cross-region data circulation mechanism includes:

[0059] The medical image database is divided into four independent storage areas. The data within each independent storage area is interrelated, enabling cross-storage area data circulation. Moreover, the medical image database is interconnected with an external case system, allowing the invocation of diagnostic texts in the external case system. Users can also, based on the external case system, call the image data stored in the four independent storage areas within the medical image database to view CT images, enhanced MRI images, original MRI images, and historical medical images.

[0060] Specifically, the medical image database can be implemented through a standardized medical interface protocol. It uses protocols such as HL7 FHIR RESTful API to transmit structured text data, such as diagnostic texts, and can achieve two-way invocation of images and data through protocols such as Web Access to DICOM Objects.

[0061] Furthermore, the four independent storage areas are respectively a CT storage area, an original storage area, an enhanced storage area, and a historical storage area. The CT storage area is used to store CT images, the original storage area is used to store original MRI images, the enhanced storage area is used to store enhanced MRI images, and the historical storage area is used to store historical medical images.

[0062] Furthermore, the four independent areas achieve cross-region association through a unified metadata index engine built based on Elasticsearch. Using the patient's unique identification ID as the index key, it supports intelligent cross-modal queries. For example, it can automatically associate relevant CT images, original MRI images, and enhanced MRI images according to the keyword "liver cancer", and can asynchronously process high-concurrency requests through a message queue, such as RabbitMQ, to ensure that the response time is less than 3 seconds when calling multi-modal data.

[0063] When storing medical image data, regularly transfer the CT images, enhanced MRI images, and original MRI images from their corresponding independent storage areas to the historical storage area, and simultaneously delete the image data of the CT images, enhanced MRI images, and original MRI images.

[0064] Execute the above transfer and deletion operation every 30 days.

[0065] Specifically, the system performs an automated transfer and deletion operation every 30 days. By triggering the data life cycle management process through a scheduled task scheduler, CT images, enhanced MRI images, and original MRI images in the CT storage area that are over 30 days old are sorted by timestamp, compressed to generate an archive package, and batch transferred to the cold storage layer of the historical storage area. At the same time, a blockchain deposit record is generated, which includes the data hash value, operator ID, and timestamp. After the transfer is completed, the corresponding data in the source storage area is immediately physically deleted to release storage space and update the metadata index. The deletion operation records the audit log throughout the process, including the data path, size, and deletion time. At the same time, through redundant verification, such as SHA-256 hash comparison, data loss or tampering is prevented.

[0066] It can be understood that using dynamic optimization of storage resources can reduce the occupancy of redundant data. For example, only active data within 30 days is retained in the original storage area, and historical data is archived to a low-cost storage area, that is, the historical storage area, which can reduce the cost of hardware investment. And through compression to generate an archive package and blockchain deposit, the long-term traceability of cross-modal data is ensured. For example, when researchers need to analyze the evolution of liver cancer lesions, longitudinal analysis within ten years can be supported. At the same time, clinicians can quickly access active images within 30 days, while researchers can efficiently retrieve historical archived data.

[0067] S102: Obtain the original MRI image data based on the medical image database. The original MRI image data includes T1W / T2W and ADC image data, and perform structured processing on the T1W / T2W and ADC image data based on the diagnostic text.

[0068] Specifically, obtain the patient's unique identity ID based on the diagnostic text, retrieve the original MRI image data in the medical image database based on the patient's unique identity ID, perform artifact detection on the T1W / T2W and ADC image data in the original MRI image data, perform artifact repair on the T1W / T2W and ADC image data based on non-local mean denoising operation and interpolation algorithm, perform normalization operation on the T1W / T2W and ADC image data, and synchronously adjust the T1W / T2W and ADC image data to the same resolution, so as to generate the processed T1W / T2W and ADC image data.

[0069] Furthermore, based on U-Net, common artifact types are identified, such as motion artifacts, susceptibility distortions, etc., and an artifact distribution heat map, that is, a probability heat map, is output. The probability is between 0 and 1. For the artifact area, non-local mean denoising is used to suppress noise, block matching and 3D filtering are performed on the T1W / T2W image data to suppress noise, while retaining the edge sharpness, and the gradient difference is greater than 15%. For the artifact area in the ADC image with a probability greater than 0.9, the diffusion coefficient is reconstructed and constrained to conform to the biological range, such as white matter in the brain. , based on gray normalization, map the T1W / T2W pixel values to the closed interval of 0 to 1, perform Z-score normalization on the ADC image data, and resample the multi-modal data to a unified resolution through affine transformation.

[0070] Based on natural language processing algorithms, segment the diagnostic text and identify entities, extract lesion area keywords and lesion degree keywords, generate an image space coordinate range matrix based on the lesion area keywords, generate dynamic space constraints based on the lesion degree keywords, perform data constraints on the image space coordinate range matrix through the dynamic space constraints, and map the constrained image space coordinate range matrix to the processed T1W / T2W and ADC image data, thereby marking the lesion area and generating structured T1W / T2W and ADC image data.

[0071] Furthermore, use the pre-trained BioClinicalBERT model to segment the diagnostic text and identify entities, and extract the identified entities, including lesion area keywords and lesion degree keywords. The lesion area keywords include anatomical location keywords and lesion type keywords, while the lesion degree keywords include malignancy grading keywords and size description keywords. For example, from "An irregular mass-like abnormal signal is seen in the left frontal lobe, with a slightly low signal on T1W, a high signal on T2W, inhomogeneous enhancement after enhancement, and a decreased ADC value (about ), consistent with glioblastoma (WHO grade IV), with a maximum diameter of about 3.2 cm", extract the anatomical location keyword "left frontal lobe", the lesion type keyword "glioblastoma", the malignancy grading keyword "WHO grade IV", and the size description keyword "maximum diameter of about 3.2 cm". Subsequently, generate a three-dimensional bounding box according to the keyword "left frontal lobe", generate an image space coordinate range matrix based on the three-dimensional bounding box, apply dynamic space constraints according to the keyword "WHO grade IV", expand the three-dimensional bounding box outward by 10 mm based on the infiltration characteristics of the keyword "glioblastoma" to cover the potential infiltration area, perform data constraints on the image space coordinate range matrix based on the outward expansion of the three-dimensional bounding box, and at the same time generate a probability mask based on the characteristic of decreased ADC value. Finally, register the image space coordinate range matrix and the probability mask to the patient's individual image space through affine transformation and overlay them with the preprocessed T1W / T2W / ADC image data to form a multi-dimensional structured label containing anatomical location, malignancy degree, and functional parameters, realizing the accurate semantic mapping from text description to image space and providing fine-grained annotation data for subsequent model enhancement and multi-modal fusion diagnosis.

[0072] S103: Establish a fusion enhancement algorithm model pool, which includes a forward enhancement model, a reverse optimization model, and a CT-assisted enhancement model. Train the relevant models in the model pool based on the historical medical image data in the medical image database, and perform matching operations based on the diagnostic text and the medical image database.

[0073] Specifically, query the corresponding CT image data in the medical image library based on the patient's unique identification ID.

[0074] If the patient's unique identification ID stores corresponding CT image data in the medical image database, and the interval between the CT image data and the original MRI image data is less than or equal to 7 days, enter the registration operation.

[0075] If the interval between the CT image data and the original MRI image data is more than seven days or there is no corresponding CT image data in the medical image database, it is determined that the matching fails.

[0076] It can be understood that the time interval between the CT image data and the original MRI image data is set to be less than or equal to 7 days mainly based on the balance between the change rate of the human anatomical structure and the tolerance of the image registration error. For most clinical scenarios, such as tumor monitoring and postoperative evaluation, the anatomical structure changes of patients are usually controllable within 7 days. For example, the peripheral growth of solid tumors ≤ 2 mm, and the regression rate of postoperative edema ≤ 1 mm / day, ensuring that the bony landmarks and soft tissue contours of CT and MRI can be initially aligned through translation / rotation during the rigid body registration stage; at the same time, the deformation compensation ability of elastic registration (the B-spline algorithm can correct up to ±15 mm non-rigid deformation) is sufficient to cover physiological deformations within the 7-day time window, such as the displacement of the diaphragm caused by respiratory movement and the displacement of abdominal organs caused by intestinal peristalsis. After more than 7 days, the progression of the lesion, such as the infiltrative growth of glioma, or the treatment response, such as the expansion of the necrotic focus after chemotherapy, may cause the anatomical structure difference to exceed the algorithm correction range. For example, the volume change of cranial tumors > 30%. At this time, forced registration will introduce an unreliable deformation field, resulting in a decrease in the Dice coefficient and distortion of the subsequent fusion enhancement results.

[0077] The registration operation refers to registering the CT image data and the original MRI image data.

[0078] The registration operation includes rigid body registration and elastic registration.

[0079] Perform rigid adjustment on the CT image data through rigid body registration based on translation and rotation operations, and align the global anatomical structures of the CT image data and the MRI image data, such as the skull contour and spinal sequence, by maximizing the mutual information, so that the CT image data and the original MRI image data are roughly aligned.

[0080] After the rigid registration is completed, elastic registration based on the B-spline algorithm compensates for the deformation of the CT image data, ensures the alignment of the lesion areas of the CT image data and the original MRI image data, and meanwhile, obtains the Dice coefficient of the elastic registration.

[0081] It can be understood that the B-spline algorithm generates a continuous and smooth deformation field by establishing a grid of uniformly distributed control points and defining the displacement vectors of each control point. The objective function consists of a similarity metric and a regularization term. After registration, the alignment accuracy of the lesion area is quantified by the Dice coefficient.

[0082] The calculation formula of the Dice coefficient is: ;

[0083] where is denoted as the Dice coefficient, is the binary mask of the CT bone window or calcified area, is denoted as the expert annotation mask of the corresponding anatomical or pathological area of the original MRI image data.

[0084] If the registration of the CT image data and the original MRI image data is successful, that is, the Dice coefficient is greater than or equal to 0.85, it is determined that the CT image data and the original MRI image data match successfully.

[0085] If the registration of the CT image data and the original MRI image data fails, that is, the Dice coefficient is less than 0.85, it is determined that the CT image data and the original MRI image data do not match.

[0086] It can be understood that the determination threshold of is based on a comprehensive consideration of clinical practice and image registration research. The Dice coefficient (ranging from 0 to 1) quantifies the overlap degree of two sets of segmentation masks. 0.85 means that the overlap rate of the anatomical or lesion areas of the CT image data and the original MRI image data reaches 85%. For example, in the registration of brain tumors, when Dice≥0.85, the consistency of the delineation of the lesion boundary by researchers, that is, the Kappa value can reach above 0.8. The radiotherapy target area error within the clinical safety range is less than or equal to 2mm, while less than 0.8 may lead to a positioning deviation of key structures exceeding 3mm, increasing the treatment risk. Meanwhile, within the range of a control point density of 20mm and available computing resources, the average Dice of the registration of common anatomical sites (such as the brain and pelvis) by the B-spline elastic registration algorithm is about . Setting 0.85 can cover about 95% of the qualified cases. For example, the Dice distribution of the CT-MRI registration of lung cancer liver metastases is 0.83 - 0.92, balancing the algorithm performance and clinical acceptability.

[0087] S1041: If the match is successful, enable the CT-assisted enhancement model, and fuse and enhance the structured T1W / T2W and ADC image data with the forward enhancement model to obtain enhanced MRI image data.

[0088] Based on the forward enhancement model and the CT-assisted enhancement model, perform alternating enhancement on the structured T1W / T2W and ADC image data, and perform a total of six rounds of image enhancement operations.

[0089] The first five rounds of image enhancement operations are performed on the structured T1W / T2W and ADC image data based on the forward enhancement model. After the enhancement is completed, the enhanced T1W / T2W and ADC image data are obtained.

[0090] When performing the sixth round of image enhancement operation, call the CT image data that matches the T1W / T2W and ADC image data, extract the CT feature data in the CT image data based on the CT-assisted enhancement model, and re-enhance the enhanced T1W / T2W and ADC image data according to the CT feature data, and generate enhanced MRI image data based on convolutional fusion.

[0091] Assign weights to the CT feature data and the enhanced T1W / T2W and ADC image data. The weight of the CT feature data is 20%, and the weight of the T1W / T2W and ADC image data is 80%. Select to retain the structural feature data, texture feature data, and quantitative feature data in the T1W / T2W and ADC image data, and iteratively adjust the feature fusion ratio of the CT feature data with the structural feature data, texture feature data, and quantitative feature data based on the joint optimization objective function.

[0092] Specifically, in the first five rounds of image enhancement operations, the forward enhancement model extracts the structural feature data based on the 3D U-Net network, extracts the texture feature data based on the 3D ResNet network, extracts the quantitative feature data based on the lightweight CNN, and dynamically assigns weights, and performs enhancement based on the forward enhancement objective loss function. After each round of enhancement, the updated enhanced T1W / T2W and ADC image data are output and input to the forward enhancement model again for enhancement until the five rounds of image enhancement operations are completed.

[0093] In the sixth-round operation, a CT-assisted enhancement sub-model is introduced. The CT image data registered with MRI is called, and CT feature data such as skull edge gradient and calcification point distribution is extracted. Subsequently, through a hierarchical attention fusion strategy, the MRI enhancement features accumulated in the previous five rounds are dynamically integrated with the CT features. 20% weight is assigned to the CT features in the bone region shown by CT to correct the susceptibility artifact, and 80% weight of the MRI features is retained in the soft tissue region to maintain the native contrast. Based on the joint optimization objective function, the feature contribution ratio is iteratively adjusted. Finally, the fused super-resolution image is output in the sixth round. In addition, an anomaly detection mechanism can be set, such as abnormal gradient magnitude and ADC value exceeding the threshold. The over-enhanced region is marked through the anomaly detection mechanism and local filtering or feature re-extraction is triggered to ensure that the fusion result not only retains the biological significance of MRI functional imaging, such as the ADC value reflecting the true degree of diffusion restriction, but also integrates the spatial accuracy of CT anatomical structures, such as sub-millimeter alignment of the skull edge, generating a multi-modal enhanced image that can be directly used for precise diagnosis and radiotherapy target delineation.

[0094] Among them, the joint optimization objective function includes an adversarial loss function, a cycle consistency loss function, and an ADC physical constraint loss function.

[0095] S1042: If the matching fails, the forward enhancement model is enabled alone to fuse and enhance the structured T1W / T2W and ADC images to obtain enhanced MRI image data.

[0096] Specifically, the forward enhancement model extracts the structural feature data (such as ventricle morphology, gray matter-white matter junction) in the structured T1W image data based on the 3D U-Net network and outputs a high-dimensional feature map with 512 channels. The texture feature data (such as wavelet energy distribution in the T2-FLAIR hyperintense region) in the structured T2W image data is extracted based on the 3D ResNet network and outputs a 1024-dimensional texture encoding vector. The quantitative feature data in the structured ADC image data is extracted based on the lightweight CNN.

[0097] Weights are dynamically assigned to the structural feature data, texture feature data, and quantitative feature data.

[0098] Based on the dynamic weights of the structural feature data, texture feature data, and quantitative feature data and the forward enhancement objective loss function, image enhancement operations are performed on the structured T1W / T2W and ADC image data, and enhanced MRI image data is generated based on convolutional fusion.

[0099] Specifically, a fused super-resolution image is generated based on a 3×3 convolutional layer. In practical applications, the number of enhancement rounds of the forward enhancement model for structured T1W / T2W and ADC image data can be set, such as five or six rounds. In addition, an anomaly detection mechanism can be added, such as using the gradient magnitude threshold method to detect oversharpening and an ADC value range validator to screen for abnormal diffusion regions for local correction.

[0100] S105: Output the enhanced MRI image data forward to the medical image database and simultaneously output it backward to the reverse optimization model to generate updated parameters, and periodically update and optimize the forward enhancement model and the CT-assisted enhancement model.

[0101] Specifically, calculate the reconstruction error through the GANomaly network of the reverse optimization model, mark the over-enhanced regions, such as blood vessel wall artifacts and abnormal ADC value fluctuation regions, generate an anomaly heat map, find the abnormal regions based on the anomaly heat map, and calculate the gradient information according to the abnormal regions to generate a model parameter update vector. Every time 100 new model parameter update vectors are accumulated, it will automatically trigger fine-tuning of the forward enhancement model and the CT-assisted enhancement model. In practical applications, users can also retrain the forward enhancement model and the CT-assisted enhancement model based on historical medical image data every quarter. Periodic model update and optimization can improve the adaptability and diagnostic accuracy of model image enhancement.

[0102] Among them, please refer to, for example Figure 2As shown in the figure, the External case system and the Medical image database are interconnected in data, and can call the relevant data stored internally with each other. The four independent storage areas in the Medical image database, namely the Enhance storage area, the Historical storage area, the CT storage area, and the Original storage area, are interconnected in data. The Medical image database can store or call relevant data in the four independent storage areas. The data in the Original storage area is input into the Fusion enhanced algorithm model pool after Structured processing, and the data in the CT storage area is also input into the Fusion enhanced algorithm model pool. After the Fusion enhanced algorithm model pool completes the image enhancement operation, it outputs the Enhance MRI image data in the forward direction, and optimizes the model in the reverse direction. The Enhance MRI image data is stored in the Enhance storage area.

[0103] An MRI image enhancement system based on multimodal fusion, comprising:

[0104] A database module 10, used to establish a medical image database, which is used to store CT image data, enhanced MRI image data, original MRI image data, and historical medical image data, and the medical image database is associated with an external case system and can call the diagnostic text in the external case system;

[0105] An acquisition module 11, used to acquire relevant data in the medical image database;

[0106] A matching module 12, used to perform matching operations;

[0107] A processing module 13 for performing structured operations on T1W / T2W and ADC image data to obtain structured T1W / T2W and ADC image data;

[0108] An enhancement module 14 for establishing a fusion enhancement algorithm model pool, which includes a forward enhancement model, a reverse optimization model, and a CT-assisted enhancement model. Based on the forward enhancement model and the reverse optimization model, the structured T1W / T2W and ADC image data are fused and enhanced to obtain enhanced MRI image data. Update parameters are generated based on the reverse optimization model, and the forward enhancement model and the CT-assisted enhancement model are updated and optimized regularly;

[0109] A training module 15 for training relevant models in the fusion enhancement algorithm model pool.

[0110] The specific usage and function of the first embodiment:

[0111] First, a medical image database is established, and a cross-region data circulation mechanism is established so that the medical image database can be associated with an external case system. Diagnostic texts are obtained through the external case system. Then, based on the medical image database, original MRI image data is obtained, which includes T1W / T2W and ADC image data. The T1W / T2W and ADC image data are structurally processed based on the diagnostic texts. Through the structural processing, the T1W / T2W and ADC image data are made more convenient for subsequent enhancement operations. Subsequently, a fusion enhancement algorithm model pool is established, which includes a forward enhancement model, a reverse optimization model, and a CT-assisted enhancement model. Matching operations are performed based on the diagnostic texts and the medical image database; if the match is successful, the CT-assisted enhancement model is enabled, and it is fused and enhanced with the forward enhancement model for the structured T1W / T2W and ADC image data to obtain enhanced MRI image data; if the match fails, the forward enhancement model is separately enabled to fuse and enhance the structured T1W / T2W and ADC images to obtain enhanced MRI image data; the enhanced MRI image data is output forward to the medical image database and simultaneously output backward to the reverse optimization model to generate update parameters, and the forward enhancement model and the CT-assisted enhancement model are updated and optimized regularly. Through this method, the original MRI images can be enhanced selectively based on the CT images, and when the CT-assisted enhancement model and the forward enhancement model are used to fuse and enhance the original MRI image data with matching CT image data, the enhancement effect is improved.

[0112] An electronic device, comprising:

[0113] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method proposed in Embodiment 1 of the present invention.

[0114] The following is a specific introduction to the various components of the electronic device:

[0115] Among them, the processor is the control center of the electronic device, which can be a single processor or a collective term for multiple processing elements. For example, the processor is one or more central processing units (CPUs), or can be an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement Embodiment 1 of the present invention, such as: one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs).

[0116] Among them, the processor can execute various functions of the electronic device by running or executing software programs stored in the memory and calling data stored in the memory.

[0117] The memory is used to store the software program for implementing the solution of the present invention and is controlled by the processor for execution. The specific implementation manner can refer to the above method embodiments and will not be elaborated here.

[0118] The memory may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only (CD-ROM), or other optical disc storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media, or other magnetic storage devices, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may be integrated with the processor or exist independently and be coupled to the processor through the interface circuit of the electronic device. The embodiments of the present invention do not make specific limitations thereto.

[0119] The above embodiments may be implemented in whole or in part by software, hardware (such as circuits), firmware, or any other combination. When implemented using software, the above embodiments may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions described in the embodiments of the present invention are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wired (such as infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that includes one or more collections of available media. The available media may be magnetic media (such as floppy disks, hard disks, magnetic tapes), optical media (such as DVDs), or semiconductor media. The semiconductor media may be a solid-state drive.

[0120] It should be understood that the term "and / or" in this text is merely a description of the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. Here, A and B can be singular or plural. Additionally, the character " / " in this text generally represents an "or" relationship between the associated objects before and after, but it may also represent an "and / or" relationship. The specific meaning can be understood by referring to the context before and after.

[0121] It should be understood that in the embodiments of the present invention, the magnitudes of the serial numbers of the above processes do not mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

[0122] The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for enhancing MRI images based on multimodal fusion, characterized in that, It includes the following steps: Establish a medical image database for storing CT image data, enhanced MRI image data, original MRI image data, and historical medical image data, establish a cross-region data circulation mechanism, and obtain diagnostic texts; Obtain the original MRI image data based on the medical image database. The original MRI image data includes T1W / T2W and ADC image data, and perform structured processing on the T1W / T2W and ADC image data based on the diagnostic text; Establish a fusion enhancement algorithm model pool. The fusion enhancement algorithm model pool includes a forward enhancement model, a reverse optimization model, and a CT-assisted enhancement model. Train the relevant models in the model pool based on the historical medical image data in the medical image database, and perform matching operations based on the diagnostic text and the medical image database; If the matching is successful, enable the CT-assisted enhancement model, and perform fusion enhancement on the structured T1W / T2W and ADC image data with the forward enhancement model to obtain enhanced MRI image data; If the matching fails, separately enable the forward enhancement model to perform fusion enhancement on the structured T1W / T2W and ADC images to obtain enhanced MRI image data; The forward enhancement model extracts the structural feature data in the structured T1W image data based on the 3D U-Net network, extracts the texture feature data in the structured T2W image data based on the 3D ResNet network, and extracts the quantitative feature data in the structured ADC image data based on the lightweight CNN; Dynamically assign weights to the structural feature data, texture feature data, and quantitative feature data; Perform image enhancement operations on the structured T1W / T2W and ADC image data based on the dynamic weights of the structural feature data, texture feature data, and quantitative feature data and the forward enhancement target loss function, and generate enhanced MRI image data based on convolutional fusion; Output the enhanced MRI image data forward to the medical image database, and at the same time output it backward to the reverse optimization model to generate updated parameters, and regularly update and optimize the forward enhancement model and the CT-assisted enhancement model.

2. The MRI image enhancement method based on multimodal fusion according to claim 1, wherein Enable the CT-assisted enhancement model, and perform fusion enhancement on the structured T1W / T2W and ADC image data with the forward enhancement model, including: Perform alternating enhancement on the structured T1W / T2W and ADC image data based on the forward enhancement model and the CT-assisted enhancement model, and perform a total of six rounds of image enhancement operations; The first five rounds of image enhancement operations are performed on the structured T1W / T2W and ADC image data based on the forward enhancement model. After the enhancement is completed, obtain the enhanced T1W / T2W and ADC image data; When performing the sixth round of image enhancement operation, call the CT image data matching the T1W / T2W and ADC image data, extract the CT feature data in the CT image data based on the CT-assisted enhancement model, and perform secondary enhancement on the enhanced T1W / T2W and ADC image data according to the CT feature data, and generate enhanced MRI image data based on convolutional fusion.

3. A method for enhancing MRI images based on multimodal fusion according to claim 2, characterized in that, And perform secondary enhancement on the enhanced T1W / T2W and ADC image data according to the CT feature data, including: Assign weights to the CT feature data and the enhanced T1W / T2W and ADC image data. The weight of the CT feature data is 20%, and the weight of the T1W / T2W and ADC image data is 80%. Select and retain the structural feature data, texture feature data, and quantitative feature data in the T1W / T2W and ADC image data, and iteratively adjust the feature fusion ratio between the CT feature data and the structural feature data, texture feature data, and quantitative feature data based on the joint optimization objective function.

4. A method for enhancing MRI images based on multimodal fusion according to claim 1, characterized in that, Perform matching operations based on the diagnostic text and the medical image database, including: Query the corresponding CT image data in the medical image library based on the patient's unique identification ID; If the corresponding CT image data is stored in the medical image database for the patient's unique identification ID and the interval between the CT image data and the original MRI image data is less than or equal to 7 days, proceed to the registration operation; If the interval between the CT image data and the original MRI image data is greater than seven days or there is no corresponding CT image data in the medical image database, it is determined that the matching fails; The registration operation means registering the CT image data and the original MRI image data; If the registration of the CT image data and the original MRI image data is successful, that is, the Dice coefficient is greater than or equal to 0.85, it is determined that the CT image data and the original MRI image data match successfully; If the registration of the CT image data and the original MRI image data fails, that is, the Dice coefficient is less than 0.85, it is determined that the CT image data and the original MRI image data match fails.

5. A method for enhancing MRI images based on multimodal fusion according to claim 4, characterized in that, The registration operation means registering the CT image data and the original MRI image data, including: The registration operation includes rigid registration and elastic registration; Perform rigid adjustment on the CT image data based on the rigid registration of the translation operation and the rotation operation, so that the CT image data is roughly aligned with the original MRI image data; After the rigid registration is completed, perform deformation compensation on the CT image data based on the elastic registration of the B-spline algorithm, and ensure that the lesion areas of the CT image data and the original MRI image data are aligned. At the same time, obtain the Dice coefficient for the elastic registration.

6. A method for enhancing MRI images based on multimodal fusion according to claim 1, characterized in that, Perform structured processing on the T1W / T2W and ADC image data based on the diagnostic text, including: Obtain the patient's unique identification ID based on the diagnostic text, retrieve the original MRI image data in the medical image database based on the patient's unique identification ID, perform artifact detection on the T1W / T2W and ADC image data in the original MRI image data, perform artifact repair on the T1W / T2W and ADC image data based on the non-local means denoising operation and the interpolation algorithm, perform normalization operation on the T1W / T2W and ADC image data, and synchronously adjust the T1W / T2W and ADC image data to the same resolution, so as to generate the processed T1W / T2W and ADC image data; Segment the diagnostic text and identify entities based on natural language processing algorithms, extract keywords for lesion regions and keywords for lesion severity, generate an image spatial coordinate range matrix based on the keywords for lesion regions, generate dynamic spatial constraints based on the keywords for lesion severity, perform data constraints on the image spatial coordinate range matrix through the dynamic spatial constraints, and map the constrained image spatial coordinate range matrix to the T1W / T2W and ADC image data being processed, thereby marking the lesion regions and generating structured T1W / T2W and ADC image data.

7. A method for enhancing MRI images based on multimodal fusion according to claim 1, characterized in that, Establish a cross-region data circulation mechanism, including: The medical image database is divided into four independent storage areas. The data within each independent storage area is interrelated and data circulation across storage areas is possible. Moreover, the medical image database is interrelated with an external case system, enabling the invocation of diagnostic text in the external case system. Users can also invoke the image data stored in the four independent storage areas within the medical image database based on the external case system to view CT images, enhanced MRI images, original MRI images, and historical medical images.

8. A method for enhancing MRI images based on multimodal fusion according to claim 7, characterized in that The medical image database is divided into four independent storage areas, including: The four independent storage areas are the CT storage area, the original storage area, the enhanced storage area, and the historical storage area. The CT storage area is used to store CT images, the original storage area is used to store original MRI images, the enhanced storage area is used to store enhanced MRI images, and the historical storage area is used to store historical medical images; When storing medical image data, periodically transfer the CT images, enhanced MRI images, and original MRI images from their corresponding independent storage areas to the historical storage area, and simultaneously delete the image data of the CT images, enhanced MRI images, and original MRI images; Execute the above transfer and deletion operations every 30 days.

9. An MRI image enhancement system based on multi-modal fusion, characterized in that, Including: A database module for establishing a medical image database, which is used to store CT image data, enhanced MRI image data, original MRI image data, and historical medical image data. The medical image database is interrelated with an external case system and can invoke diagnostic text in the external case system; An acquisition module for acquiring relevant data within the medical image database; A matching module for performing matching operations; A processing module for structuring the T1W / T2W and ADC image data to obtain structured T1W / T2W and ADC image data; An enhancement module for establishing a fusion enhancement algorithm model pool. The fusion enhancement algorithm model pool includes a forward enhancement model, a reverse optimization model, and a CT-assisted enhancement model. Perform fusion enhancement on the structured T1W / T2W and ADC image data based on the forward enhancement model and the reverse optimization model to obtain enhanced MRI image data. Generate update parameters based on the reverse optimization model and periodically update and optimize the forward enhancement model and the CT-assisted enhancement model; The positive enhancement model extracts structural feature data from structured T1W image data based on the 3D U-Net network, extracts texture feature data from structured T2W image data based on the 3D ResNet network, and extracts quantitative feature data from structured ADC image data based on the lightweight CNN; dynamically assigns weights to the structural feature data, texture feature data, and quantitative feature data; Performs image enhancement operations on the structured T1W / T2W and ADC image data based on the dynamic weights of the structural feature data, texture feature data, and quantitative feature data and the positive enhancement objective loss function, and generates enhanced MRI image data based on convolutional fusion; A training module for training relevant models in the fusion enhancement algorithm model pool.

Images