MRI (Magnetic Resonance Imaging) image enhancement method and system based on multi-modal fusion

Abstract

The invention provides an MRI image enhancement method and system based on multi-modal fusion, and belongs to the technical field of image enhancement, and the method comprises the steps: building a medical image database, and obtaining a diagnosis text; original MRI image data are obtained, and structured processing is carried out on T1W / T2W and ADC image data; a fusion enhancement algorithm model pool is established, a forward enhancement model, a reverse optimization model and a CT auxiliary enhancement model are included in the fusion enhancement algorithm model pool, and matching operation is carried out; if the matching is successful, starting a CT auxiliary enhancement model, and performing fusion enhancement with the forward enhancement model; if the matching fails, independently starting a forward enhancement model to carry out fusion enhancement; forward outputting the enhanced MRI image data into the medical image database, reversely outputting the enhanced MRI image data into the reverse optimization model, and regularly updating and optimizing the forward enhancement model and the CT auxiliary enhancement model; a method for enhancing an MRI image based on CT image selectivity is provided.

Description

Technical Field

[0001] The present invention belongs to the technical field of image enhancement, and more specifically, particularly relates to a method and system for enhancing MRI images based on multimodal fusion. Background Art

[0002] In the field of modern medical image diagnosis, MRI has become an important means for diagnosing various diseases such as brain diseases, nervous system diseases, cardiovascular diseases, and tumors due to its significant advantages such as radiation-free, high soft tissue resolution, and multi-directional imaging. By analyzing MRI images, doctors can clearly observe the morphology, structure, and functional changes of human tissues and organs, thus providing key evidence for the early detection, accurate diagnosis, and effective treatment of diseases. During the MRI imaging process, it is vulnerable to various factors such as magnetic field inhomogeneity and radio frequency noise, resulting in problems such as low contrast, blurring, and artifacts in the images. Traditional MRI image enhancement methods often cannot selectively enhance MRI images based on CT images, and the enhancement effect is weak. Summary of the Invention

[0003] To solve the above technical problems, the present invention provides a method and system for enhancing MRI images based on multimodal fusion to solve the technical problems in the prior art that traditional MRI image enhancement methods often cannot selectively enhance MRI images based on CT images and the enhancement effect is weak.

[0004] The purpose and efficacy of a method and system for enhancing MRI images based on multimodal fusion of the present invention are achieved by the following specific technical means: A method for enhancing MRI images based on multimodal fusion includes the following steps: Establish a medical image database for storing CT image data, enhanced MRI image data, original MRI image data, and historical medical image data, establish a cross-region data circulation mechanism, and obtain diagnostic texts; Obtain original MRI image data based on the medical image database. The original MRI image data includes T1W / T2W and ADC image data, and perform structured processing on the T1W / T2W and ADC image data based on the diagnostic texts; Establish a fusion enhancement algorithm model pool. The fusion enhancement algorithm model pool includes a forward enhancement model, a reverse optimization model, and a CT-assisted enhancement model. Train the relevant models in the model pool based on the historical medical image data in the medical image database, and perform matching operations based on the diagnostic texts and the medical image database; If the matching is successful, enable the CT-assisted enhancement model, and perform fusion enhancement on the structured T1W / T2W and ADC image data with the forward enhancement model to obtain enhanced MRI image data; If the matching fails, the forward enhancement model is enabled alone to fuse and enhance the structured T1W / T2W and ADC images, and the enhanced MRI image data is obtained; The enhanced MRI image data is output forward to the medical image database, and at the same time, it is output backward to the reverse optimization model to generate updated parameters, and the forward enhancement model and the CT-assisted enhancement model are updated and optimized regularly.

[0005] As a further solution of the present invention, the forward enhancement model is enabled alone to fuse and enhance the structured T1W / T2W image data and ADC image data, including: The forward enhancement model extracts the structural feature data in the structured T1W image data based on the 3D U-Net network, extracts the texture feature data in the structured T2W image data based on the 3D ResNet network, and extracts the quantitative feature data in the structured ADC image data based on the lightweight CNN; Dynamically assign weights to the structural feature data, texture feature data, and quantitative feature data; Based on the dynamic weights of the structural feature data, texture feature data, and quantitative feature data and the forward enhancement target loss function, perform image enhancement operations on the structured T1W / T2W and ADC image data, and generate enhanced MRI image data based on convolutional fusion.

[0006] As a further solution of the present invention, the CT-assisted enhancement model is enabled, and it is fused and enhanced with the forward enhancement model for the structured T1W / T2W and ADC image data, including: Based on the forward enhancement model and the CT-assisted enhancement model, perform alternating enhancement on the structured T1W / T2W and ADC image data, and a total of six rounds of image enhancement operations are performed; The first five rounds of image enhancement operations are performed based on the forward enhancement model for the structured T1W / T2W and ADC image data. After the enhancement is completed, the enhanced T1W / T2W and ADC image data are obtained; When performing the sixth round of image enhancement operations, call the CT image data matching the T1W / T2W and ADC image data, extract the CT feature data in the CT image data based on the CT-assisted enhancement model, and perform secondary enhancement on the enhanced T1W / T2W and ADC image data according to the CT feature data, and generate enhanced MRI image data based on convolutional fusion.

[0007] As a further solution of the present invention, and perform secondary enhancement on the enhanced T1W / T2W and ADC image data according to the CT feature data, including: Weigh the CT feature data and the enhanced T1W / T2W and ADC image data. The weight of the CT feature data is 20%, and the weight of the T1W / T2W and ADC image data is 80%. Select and retain the structural feature data, texture feature data, and quantitative feature data in the T1W / T2W and ADC image data, and iteratively adjust the feature fusion ratio of the CT feature data with the structural feature data, texture feature data, and quantitative feature data based on the joint optimization objective function.

[0008] As a further aspect of the present invention, perform a matching operation based on the diagnostic text and the medical image database, including: Query the corresponding CT image data in the medical image library based on the patient's unique identity ID; If the patient's unique identity ID stores corresponding CT image data in the medical image database, and the interval between the CT image data and the original MRI image data is less than or equal to 7 days, proceed to the registration operation; If the interval between the CT image data and the original MRI image data is greater than seven days or there is no corresponding CT image data in the medical image database, it is determined that the matching fails; The registration operation is expressed as registering the CT image data and the original MRI image data; If the registration of the CT image data and the original MRI image data is successful, that is, the Dice coefficient is greater than or equal to 0.85, it is determined that the CT image data and the original MRI image data match successfully; If the registration of the CT image data and the original MRI image data fails, that is, the Dice coefficient is less than 0.85, it is determined that the CT image data and the original MRI image data match fails.

[0009] As a further aspect of the present invention, the registration operation is expressed as registering the CT image data and the original MRI image data, including: The registration operation includes rigid registration and elastic registration; Perform rigid adjustment on the CT image data based on the rigid registration of translation operation and rotation operation, so that the CT image data is roughly aligned with the original MRI image data; After the rigid registration is completed, perform deformation compensation on the CT image data based on the elastic registration of the B-spline algorithm, and ensure that the lesion areas of the CT image data and the original MRI image data are aligned. At the same time, obtain the Dice coefficient of the elastic registration.

[0010] As a further aspect of the present invention, perform structured processing on the T1W / T2W and ADC image data based on the diagnostic text, including: Obtain the patient's unique identity ID based on the diagnostic text, retrieve the original MRI image data from the medical image database based on the patient's unique identity ID, perform artifact detection on the T1W / T2W and ADC image data in the original MRI image data, perform artifact repair on the T1W / T2W and ADC image data based on non-local mean denoising operation and interpolation algorithm, perform normalization operation on the T1W / T2W and ADC image data, and synchronously adjust the T1W / T2W and ADC image data to the same resolution, so as to generate the processed T1W / T2W and ADC image data; Perform word segmentation and entity recognition on the diagnostic text based on natural language processing algorithms, extract the key words of the lesion area and the key words of the lesion degree, generate an image space coordinate range matrix based on the key words of the lesion area, generate a dynamic space constraint based on the key words of the lesion degree, perform data constraint on the image space coordinate range matrix through the dynamic space constraint, and map the constrained image space coordinate range matrix to the processed T1W / T2W and ADC image data, so as to mark the lesion area and generate structured T1W / T2W and ADC image data.

[0011] As a further solution of the present invention, a cross-region data circulation mechanism is established, including: The medical image database is divided into four independent storage areas. The data in each independent storage area is mutually related and can perform cross-storage area data circulation. Moreover, the medical image database is mutually related with the external case system and can call the diagnostic text in the external case system. Users can also call the image data stored in the four independent storage areas in the medical image database based on the external case system, so as to view CT images, enhanced MRI images, original MRI images, and historical medical images.

[0012] As a further solution of the present invention, the medical image database is divided into four independent storage areas, including: The four independent storage areas are respectively a CT storage area, an original storage area, an enhanced storage area, and a historical storage area. The CT storage area is used to store CT images, the original storage area is used to store original MRI images, the enhanced storage area is used to store enhanced MRI images, and the historical storage area is used to store historical medical images; When storing medical image data, regularly transfer and store the CT images, enhanced MRI images, and original MRI images from the corresponding independent storage areas to the historical storage area, and at the same time delete the image data of the CT images, enhanced MRI images, and original MRI images; Execute the above transfer and deletion operation every 30 days.

[0013] An MRI image enhancement system based on multi-modal fusion, including: A database module for establishing a medical image database, which is used to store CT image data, enhanced MRI image data, original MRI image data and historical medical image data, and the medical image database is interconnected with an external case system and can call diagnostic texts in the external case system; An acquisition module for acquiring relevant data in the medical image database; A matching module for performing matching operations; A processing module for performing a structuring operation on T1W / T2W and ADC image data to obtain structured T1W / T2W and ADC image data; An enhancement module for establishing a fusion enhancement algorithm model pool, which includes a forward enhancement model, a reverse optimization model and a CT-assisted enhancement model. Based on the forward enhancement model and the reverse optimization model, the structured T1W / T2W and ADC image data are fused and enhanced to obtain enhanced MRI image data. Based on the reverse optimization model, updated parameters are generated to periodically update and optimize the forward enhancement model and the CT-assisted enhancement model; A training module for training relevant models in the fusion enhancement algorithm model pool.

[0014] Compared with the prior art, the present invention has the following beneficial effects: First, establish a medical image database and a cross-region data circulation mechanism so that the medical image database can be associated with an external case system. Obtain diagnostic texts through the external case system. Then, based on the medical image database, obtain the original MRI image data, which includes T1W / T2W and ADC image data. Perform structured processing on the T1W / T2W and ADC image data based on the diagnostic texts. Through structured processing, the T1W / T2W and ADC image data can be made more convenient for subsequent enhancement operations. Subsequently, establish a fusion enhancement algorithm model pool, which includes a forward enhancement model, a reverse optimization model, and a CT-assisted enhancement model. Perform a matching operation based on the diagnostic texts and the medical image database; if the match is successful, enable the CT-assisted enhancement model and perform fusion enhancement on the structured T1W / T2W and ADC image data with the forward enhancement model to obtain enhanced MRI image data; if the match fails, separately enable the forward enhancement model to perform fusion enhancement on the structured T1W / T2W and ADC images to obtain enhanced MRI image data; output the enhanced MRI image data forward to the medical image database and simultaneously output it backward to the reverse optimization model to generate updated parameters, and regularly update and optimize the forward enhancement model and the CT-assisted enhancement model. Through this method, the original MRI images can be enhanced selectively based on CT images, and when the CT-assisted enhancement model and the forward enhancement model perform fusion enhancement on the original MRI image data with matching CT image data, the enhancement effect is improved. Description of the Drawings

[0015] Figure 1 is a flowchart of the steps of a method for enhancing MRI images based on multimodal fusion according to the present invention; Figure 2 is a data flow diagram of a method for enhancing MRI images based on multimodal fusion according to the present invention; Figure 3 is a schematic diagram of a system for enhancing MRI images based on multimodal fusion according to the present invention. Detailed Embodiment

[0016] The following further describes in detail the embodiments of the present invention in conjunction with the drawings and examples. The following examples are used to illustrate the technical solutions of the present invention but cannot be used to limit the protection scope of the present invention.

[0017] Embodiment 1: As shown in the attached Figure 1 、 Figure 2 : The present invention provides a method for enhancing MRI images based on multimodal fusion, which is applicable to image enhancement and includes the following steps: S101: Establish a medical image database for storing CT image data, enhanced MRI image data, original MRI image data, and historical medical image data. Establish a cross-region data circulation mechanism to obtain diagnostic texts.

[0018] Specifically, establishing a cross-region data circulation mechanism includes: The medical image database is divided into four independent storage areas. The data within each independent storage area is interrelated, enabling cross-storage area data circulation. Moreover, the medical image database is interconnected with an external case system, allowing the invocation of diagnostic texts in the external case system. Users can also, based on the external case system, invoke the image data stored in the four independent storage areas within the medical image database to view CT images, enhanced MRI images, original MRI images, and historical medical images.

[0019] Specifically, the medical image database can be implemented through a standardized medical interface protocol. It uses, for example, the HL7 FHIR RESTful API to transmit structured text data such as diagnostic texts, and can achieve two-way invocation of images and data through protocols such as Web Access to DICOM Objects.

[0020] Furthermore, the four independent storage areas are respectively a CT storage area, an original storage area, an enhanced storage area, and a historical storage area. The CT storage area is used to store CT images, the original storage area is used to store original MRI images, the enhanced storage area is used to store enhanced MRI images, and the historical storage area is used to store historical medical images.

[0021] Furthermore, the four independent areas achieve cross-region association through a unified metadata index engine built based on Elasticsearch. Using the patient's unique identity ID as the index key, it supports intelligent cross-modal queries. For example, it can automatically associate relevant CT images, original MRI images, and enhanced MRI images according to the keyword "liver cancer", and can asynchronously process high-concurrency requests through a message queue such as RabbitMQ to ensure that the response time is less than 3 seconds when invoking multi-modal data.

[0022] When storing medical image data, regularly transfer and store CT images, enhanced MRI images, and original MRI images from their corresponding independent storage areas to the historical storage area, and simultaneously delete the image data of CT images, enhanced MRI images, and original MRI images.

[0023] Execute the above transfer and deletion operation every 30 days.

[0024] Specifically, the system performs an automated transfer and deletion operation every 30 days. By triggering the data life cycle management process through a scheduled task scheduler, CT images, enhanced MRI images, and original MRI images in the CT storage area that are over 30 days old are sorted by timestamp, compressed to generate an archive package, and batch transferred to the cold storage layer of the historical storage area. At the same time, a blockchain deposit record is generated, which includes the data hash value, operator ID, and timestamp. After the transfer is completed, the corresponding data in the source storage area is immediately physically deleted to release storage space and update the metadata index. The deletion operation records the audit log throughout the process, including the data path, size, and deletion time. At the same time, through redundant verification, such as SHA-256 hash comparison, data loss or tampering is prevented.

[0025] It can be understood that using dynamic optimization of storage resources can reduce the occupancy of redundant data. For example, only active data within 30 days is retained in the original storage area, and historical data is archived to a low-cost storage area, that is, the historical storage area, which can reduce the cost of hardware investment. And through compression to generate an archive package and blockchain deposit, the long-term traceability of cross-modal data is ensured. For example, when researchers need to analyze the evolution of liver cancer lesions, longitudinal analysis within ten years can be supported. At the same time, clinicians can quickly access active images within 30 days, while researchers can efficiently retrieve historical archived data.

[0026] S102: Obtain the original MRI image data based on the medical image database. The original MRI image data includes T1W / T2W and ADC image data, and perform structured processing on the T1W / T2W and ADC image data based on the diagnostic text.

[0027] Specifically, obtain the unique patient identity ID based on the diagnostic text, retrieve the original MRI image data in the medical image database based on the unique patient identity ID, perform artifact detection on the T1W / T2W and ADC image data in the original MRI image data, perform artifact repair on the T1W / T2W and ADC image data based on non-local mean denoising operation and interpolation algorithm, perform normalization operation on the T1W / T2W and ADC image data, and synchronously adjust the T1W / T2W and ADC image data to the same resolution, thereby generating the processed T1W / T2W and ADC image data.

[0028] Furthermore, based on U-Net, common artifact types are identified, such as motion artifacts, susceptibility distortions, etc., and an artifact distribution heat map, that is, a probability heat map, is output. The probability is between 0 and 1. For the artifact area, non-local mean denoising is used to suppress noise, block matching and 3D filtering are performed on the T1W / T2W image data to suppress noise, while retaining the edge sharpness, and the gradient difference is greater than 15%. For the artifact area in the ADC image with a probability greater than 0.9, the diffusion coefficient is reconstructed and constrained to conform to the biological range, such as white matter in the brain , based on gray normalization, map the T1W / T2W pixel values to the closed interval of 0 to 1, perform Z-score normalization on the ADC image data, and resample the multimodal data to a unified resolution through affine transformation.

[0029] Based on natural language processing algorithms, tokenize and perform entity recognition on the diagnostic text, extract lesion area keywords and lesion degree keywords, generate an image space coordinate range matrix based on the lesion area keywords, generate dynamic space constraints based on the lesion degree keywords, perform data constraints on the image space coordinate range matrix through the dynamic space constraints, and map the constrained image space coordinate range matrix to the processed T1W / T2W and ADC image data, so as to mark the lesion area and generate structured T1W / T2W and ADC image data.

[0030] Furthermore, use the pre-trained BioClinicalBERT model to tokenize and recognize entities in the diagnostic text, and extract the recognized entities, including lesion area keywords and lesion degree keywords. The lesion area keywords include anatomical location keywords and lesion type keywords, while the lesion degree keywords include malignancy grading keywords and size description keywords. For example, from "An irregular mass-like abnormal signal is seen in the left frontal lobe, with slightly low signal on T1W, high signal on T2W, inhomogeneous enhancement after enhancement, and decreased ADC value (about ), consistent with glioblastoma (WHO grade IV), with a maximum diameter of about 3.2 cm", extract the anatomical location keyword "left frontal lobe", the lesion type keyword "glioblastoma", the malignancy grading keyword "WHO grade IV", and the size description keyword "maximum diameter of about 3.2 cm". Subsequently, generate a three-dimensional bounding box according to the keyword "left frontal lobe", generate an image space coordinate range matrix based on the three-dimensional bounding box, apply dynamic space constraints according to the keyword "WHO grade IV", expand the three-dimensional bounding box outward by 10 mm for the infiltration characteristics of the keyword "glioblastoma" to cover the potential infiltration area, perform data constraints on the image space coordinate range matrix based on the outward expansion of the three-dimensional bounding box, and at the same time generate a probability mask based on the decreased ADC value feature. Finally, register the image space coordinate range matrix and the probability mask to the patient's individual image space through affine transformation and superimpose them on the preprocessed T1W / T2W / ADC image data to form a multi-dimensional structured label containing anatomical location, malignancy degree, and functional parameters, realizing accurate semantic mapping from text description to image space and providing fine-grained annotation data for subsequent model enhancement and multimodal fusion diagnosis.

[0031] S103: Establish a fusion enhancement algorithm model pool, which includes a forward enhancement model, a reverse optimization model, and a CT-assisted enhancement model. Train the relevant models in the model pool based on the historical medical image data in the medical image database, and perform matching operations based on the diagnostic text and the medical image database.

[0032] Specifically, query the corresponding CT image data in the medical image library based on the patient's unique identification ID.

[0033] If the patient's unique identification ID stores corresponding CT image data in the medical image database and the interval between the CT image data and the original MRI image data is less than or equal to 7 days, enter the registration operation.

[0034] If the interval between the CT image data and the original MRI image data is more than seven days or there is no corresponding CT image data in the medical image database, it is determined that the matching fails.

[0035] It can be understood that setting the time interval between the CT image data and the original MRI image data to be less than or equal to 7 days is mainly based on the balance between the rate of change of the human anatomical structure and the tolerance of image registration error. For most clinical scenarios, such as tumor monitoring and postoperative evaluation, the anatomical structure changes of patients are usually controllable within 7 days. For example, the peripheral growth of solid tumors ≤ 2 mm, and the regression rate of postoperative edema ≤ 1 mm / day, ensuring that the bony landmarks and soft tissue contours of CT and MRI can be initially aligned through translation / rotation in the rigid body registration stage; at the same time, the deformation compensation ability of elastic registration (the B-spline algorithm can correct up to ±15 mm of non-rigid deformation) is sufficient to cover physiological deformations within a 7-day time window, such as the displacement of the diaphragm caused by respiratory movement and the displacement of abdominal organs caused by intestinal peristalsis. However, after more than 7 days, the progression of the lesion, such as the infiltrative growth of glioma, or the treatment response, such as the expansion of the necrotic focus after chemotherapy, may lead to anatomical differences beyond the correction range of the algorithm. For example, the volume change of intracranial tumors > 30%. At this time, forced registration will introduce an unreliable deformation field, resulting in a decrease in the Dice coefficient and distortion of the subsequent fusion enhancement results.

[0036] The registration operation refers to registering the CT image data and the original MRI image data.

[0037] The registration operation includes rigid body registration and elastic registration.

[0038] Perform rigid adjustment on the CT image data through rigid body registration based on translation and rotation operations, and align the global anatomical structures of the CT image data and the MRI image data, such as the skull contour and spinal column sequence, by maximizing the mutual information, so that the CT image data and the original MRI image data are roughly aligned.

[0039] After the rigid registration is completed, elastic registration based on the B-spline algorithm compensates for the deformation of the CT image data, ensures the alignment of the lesion areas of the CT image data and the original MRI image data. At the same time, the Dice coefficient of the elastic registration is obtained.

[0040] It can be understood that the B-spline algorithm generates a continuous and smooth deformation field by establishing a uniformly distributed control point grid and defining the displacement vector of each control point. The objective function consists of a similarity metric and a regularization term. After registration, the alignment accuracy of the lesion area is quantified by the Dice coefficient.

[0041] The calculation formula of the Dice coefficient is: ; where is denoted as the Dice coefficient, is the binary mask of the CT bone window or calcified area, is denoted as the expert annotation mask of the corresponding anatomical or pathological area of the original MRI image data.

[0042] If the registration of the CT image data and the original MRI image data is successful, that is, the Dice coefficient is greater than or equal to 0.85, it is determined that the CT image data and the original MRI image data match successfully.

[0043] If the registration of the CT image data and the original MRI image data fails, that is, the Dice coefficient is less than 0.85, it is determined that the CT image data and the original MRI image data do not match.

[0044] It can be understood that the determination threshold of is based on a comprehensive consideration of clinical practice and image registration research. The Dice coefficient (value range 0-1) quantifies the overlap degree of two groups of segmentation masks. 0.85 means that the overlap rate of the anatomical or lesion areas of the CT image data and the original MRI image data reaches 85%. For example, in the registration of brain tumors, when Dice≥0.85, the consistency of the delineation of the lesion boundary by researchers, that is, the Kappa value can reach above 0.8. The radiotherapy target area error within the clinical safety range is less than or equal to 2mm, while less than 0.8 may lead to a positioning deviation of key structures exceeding 3mm, increasing the treatment risk. At the same time, within the range of a control point density of 20mm and available computing resources, the average Dice of the registration of common anatomical parts (such as the brain and pelvis) by the B-spline elastic registration algorithm is about Setting 0.85 can cover about 95% of the qualified cases. For example, the Dice distribution of CT-MRI registration of lung cancer liver metastases is 0.83-0.92, balancing the algorithm performance and clinical acceptability.

[0045] S1041: If the match is successful, enable the CT-assisted enhancement model, and fuse and enhance the structured T1W / T2W and ADC image data with the forward enhancement model to obtain enhanced MRI image data.

[0046] Perform alternating enhancement on the structured T1W / T2W and ADC image data based on the forward enhancement model and the CT-assisted enhancement model, and perform a total of six rounds of image enhancement operations.

[0047] The first five rounds of image enhancement operations are performed on the structured T1W / T2W and ADC image data based on the forward enhancement model. After the enhancement is completed, the enhanced T1W / T2W and ADC image data are obtained.

[0048] When performing the sixth round of image enhancement operation, call the CT image data that matches the T1W / T2W and ADC image data, extract the CT feature data in the CT image data based on the CT-assisted enhancement model, and re-enhance the enhanced T1W / T2W and ADC image data according to the CT feature data, and generate enhanced MRI image data based on convolutional fusion.

[0049] Assign weights to the CT feature data and the enhanced T1W / T2W and ADC image data. The weight of the CT feature data is 20%, and the weight of the T1W / T2W and ADC image data is 80%. Select and retain the structural feature data, texture feature data, and quantitative feature data in the T1W / T2W and ADC image data, and iteratively adjust the feature fusion ratio of the CT feature data with the structural feature data, texture feature data, and quantitative feature data based on the joint optimization objective function.

[0050] Specifically, in the first five rounds of image enhancement operations, the forward enhancement model extracts the structural feature data based on the 3D U-Net network, extracts the texture feature data based on the 3D ResNet network, extracts the quantitative feature data based on the lightweight CNN, and dynamically assigns weights, and performs enhancement based on the forward enhancement objective loss function. After each round of enhancement, the updated enhanced T1W / T2W and ADC image data are output and input to the forward enhancement model again for enhancement until the five rounds of image enhancement operations are completed.

[0051] In the sixth round of operation, a CT-assisted enhancement sub-model is introduced. CT image data registered with MRI is called, and CT feature data such as skull edge gradient and calcification point distribution is extracted. Subsequently, through a hierarchical attention fusion strategy, the MRI enhancement features accumulated in the previous five rounds are dynamically integrated with the CT features. 20% weight is given to the CT features in the bone region shown by CT to correct susceptibility artifacts, and 80% weight of the MRI features is retained in the soft tissue region to maintain the original contrast. Based on the joint optimization objective function, the feature contribution ratio is iteratively adjusted. Finally, a fused super-resolution image is output in the sixth round. In addition, an anomaly detection mechanism can be set, such as abnormal gradient magnitude and ADC value exceeding the threshold. Through the anomaly detection mechanism, over-enhanced regions are marked and local filtering or feature re-extraction is triggered to ensure that the fusion result not only retains the biological significance of MRI functional imaging, such as the ADC value reflecting the true degree of diffusion restriction, but also fuses the spatial accuracy of the CT anatomical structure, such as sub-millimeter alignment of the skull edge, generating a multi-modal enhanced image that can be directly used for precise diagnosis and radiotherapy target delineation.

[0052] Among them, the joint optimization objective function includes an adversarial loss function, a cycle consistency loss function, and an ADC physical constraint loss function.

[0053] S1042: If the matching fails, the forward enhancement model is separately enabled to fuse and enhance the structured T1W / T2W and ADC images to obtain enhanced MRI image data.

[0054] Specifically, the forward enhancement model extracts the structural feature data (such as ventricular morphology, gray matter-white matter junction) in the structured T1W image data based on a 3D U-Net network, and outputs a high-dimensional feature map with 512 channels. The texture feature data (such as wavelet energy distribution in the T2-FLAIR hyperintense region) in the structured T2W image data is extracted based on a 3D ResNet network, and a 1024-dimensional texture encoding vector is output. The quantitative feature data in the structured ADC image data is extracted based on a lightweight CNN.

[0055] Weights are dynamically assigned to the structural feature data, texture feature data, and quantitative feature data.

[0056] Based on the dynamic weights of the structural feature data, texture feature data, and quantitative feature data and the forward enhancement objective loss function, image enhancement operations are performed on the structured T1W / T2W and ADC image data, and enhanced MRI image data is generated based on convolutional fusion.

[0057] Specifically, a fused super-resolution image is generated based on a 3×3 convolutional layer. In practical applications, the number of enhancement rounds of the forward enhancement model for structured T1W / T2W and ADC image data can be set, such as five or six rounds. In addition, an anomaly detection mechanism can be added, such as using the gradient magnitude threshold method to detect oversharpening and an ADC value range validator to screen for abnormal diffusion regions for local correction.

[0058] S105: Forward the enhanced MRI image data to the medical image database and, at the same time, output it backward to the reverse optimization model to generate updated parameters, and periodically update and optimize the forward enhancement model and the CT-assisted enhancement model.

[0059] Specifically, calculate the reconstruction error through the GANomaly network of the reverse optimization model, mark the over-enhanced regions, such as blood vessel wall artifacts and abnormal ADC value fluctuation regions, generate an anomaly heat map, find the abnormal regions based on the anomaly heat map, calculate the gradient information according to the abnormal regions, and generate a model parameter update vector. Every time 100 new model parameter update vectors are accumulated, it will automatically trigger fine-tuning of the forward enhancement model and the CT-assisted enhancement model. In practical applications, users can also retrain the forward enhancement model and the CT-assisted enhancement model based on historical medical image data every quarter. Periodic model update and optimization can improve the adaptability and diagnostic accuracy of model image enhancement.

[0060] Among them, please refer to, for example Figure 2As shown, the External case system and the Medical image database are interconnected and can call the relevant data stored internally with each other. The four independent storage areas in the Medical image database, namely the Enhance storage area, the Historical storage area, the CT storage area, and the Original storage area, are interconnected. The Medical image database can store or call relevant data in the four independent storage areas. The data in the Original storage area is input into the Fusion enhanced algorithm model pool after being Structured processing, and the data in the CT storage area is also input into the Fusion enhanced algorithm model pool. After the Fusion enhanced algorithm model pool performs image enhancement operations, it outputs the Enhance MRI image data in the forward direction and optimizes the model in the reverse direction. The Enhance MRI image data is stored in the Enhance storage area.

[0061] An MRI image enhancement system based on multimodal fusion, comprising: A database module 10 for establishing a medical image database, which is used to store CT image data, enhanced MRI image data, original MRI image data, and historical medical image data. The medical image database is associated with the external case system and can call the diagnostic text in the external case system. An acquisition module 11 for acquiring relevant data in the medical image database. A matching module 12 for performing matching operations. A processing module 13 for performing structured operations on T1W / T2W and ADC image data to obtain structured T1W / T2W and ADC image data. Enhancement module 14 is used to establish a fusion enhancement algorithm model pool, which includes a forward enhancement model, a reverse optimization model, and a CT-assisted enhancement model. Based on the forward enhancement model and the reverse optimization model, the structured T1W / T2W and ADC image data are fused and enhanced to obtain enhanced MRI image data. The reverse optimization model is used to generate update parameters to periodically update and optimize the forward enhancement model and the CT-assisted enhancement model. Training module 15 is used to train the relevant models in the fusion enhancement algorithm model pool.

[0062] The specific usage and function of the first embodiment: First, a medical image database is established, and a cross-region data circulation mechanism is established so that the medical image database can be associated with an external case system. Diagnostic texts are obtained through the external case system. Then, based on the medical image database, the original MRI image data is obtained, which includes T1W / T2W and ADC image data. The T1W / T2W and ADC image data are structured based on the diagnostic texts. Through the structuring process, the T1W / T2W and ADC image data are made more convenient for subsequent enhancement operations. Subsequently, a fusion enhancement algorithm model pool is established, which includes a forward enhancement model, a reverse optimization model, and a CT-assisted enhancement model. A matching operation is performed based on the diagnostic texts and the medical image database. If the match is successful, the CT-assisted enhancement model is enabled, and it is used together with the forward enhancement model to fuse and enhance the structured T1W / T2W and ADC image data to obtain enhanced MRI image data. If the match fails, only the forward enhancement model is enabled to fuse and enhance the structured T1W / T2W and ADC images to obtain enhanced MRI image data. The enhanced MRI image data is output forward to the medical image database and simultaneously output backward to the reverse optimization model to generate update parameters, and the forward enhancement model and the CT-assisted enhancement model are periodically updated and optimized. Through this method, the original MRI images can be selectively enhanced based on the CT images, and when the CT-assisted enhancement model and the forward enhancement model are used to fuse and enhance the original MRI image data with matching CT image data, the enhancement effect is improved.

[0063] An electronic device includes: At least one processor; and a memory communicatively connected to at least one processor. The memory stores instructions executable by the at least one processor. When the instructions are executed by the at least one processor, the at least one processor is enabled to execute the method proposed in the first embodiment of the present invention.

[0064] The following is a specific introduction to the components of the electronic device: Among them, the processor is the control center of the electronic device, which can be a single processor or a collective term for multiple processing elements. For example, the processor can be one or more central processing units (CPUs), or can be an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement Embodiment 1 of the present invention. For example: one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs).

[0065] Among them, the processor can execute various functions of the electronic device by running or executing software programs stored in the memory and calling data stored in the memory.

[0066] The memory is used to store the software program for implementing the solution of the present invention and is controlled by the processor for execution. The specific implementation manner can refer to the above method embodiments and will not be elaborated here.

[0067] The memory can be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or can also be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only (CD-ROM), or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory can be integrated with the processor or exist independently and be coupled to the processor through the interface circuit of the electronic device. The embodiments of the present invention do not make specific limitations on this.

[0068] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware, or any combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions described in the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wired (such as infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that contains one or more collections of available media. The available media can be magnetic media (such as floppy disks, hard disks, magnetic tapes), optical media (such as DVDs), or semiconductor media. The semiconductor media can be a solid-state drive.

[0069] It should be understood that the term "and / or" in this document is merely a description of the association relationship between associated objects, indicating that there can be three relationships. For example, A and / or B can mean that A exists alone, A and B exist simultaneously, or B exists alone. Here, A and B can be singular or plural. In addition, the character " / " in this document generally represents an "or" relationship between the associated objects before and after, but it may also represent an "and / or" relationship, which can be specifically understood with reference to the context.

[0070] It should be understood that in the embodiments of the present invention, the magnitudes of the serial numbers of the above processes do not mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

[0071] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments or perform equivalent replacements for some of the technical features. These modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention and should all be included in the protection scope of the present invention.

Claims

1. A MRI image enhancement method based on multimodal fusion, characterized in that: The following steps are included: Establish a medical image database to store CT image data, enhanced MRI image data, original MRI image data and historical medical image data, establish a cross-regional data circulation mechanism, and obtain diagnostic texts; Acquire original MRI image data based on a medical image database, the original MRI image data including T1W / T2W and ADC image data, and perform structured processing on the T1W / T2W and ADC image data based on the diagnosis text; Establish a fusion enhancement algorithm model pool, which includes a forward enhancement model, a reverse optimization model, and a CT-assisted enhancement model. Train relevant models in the model pool based on historical medical image data in the medical image database, and perform matching operations based on diagnostic text and the medical image database. If the match is successful, the CT-assisted enhancement model is enabled, and the structured T1W / T2W and ADC image data are fused and enhanced with the forward enhancement model to obtain enhanced MRI image data; If the matching fails, the forward enhancement model is used alone to perform fusion enhancement on the structured T1W / T2W and ADC images to obtain enhanced MRI image data; The enhanced MRI image data is forward output to the medical image database and reversely output to the reverse optimization model to generate updated parameters, and the forward enhancement model and CT-assisted enhancement model are regularly updated and optimized.

2. The MRI image enhancement method based on multimodal fusion according to claim 1, characterized in that: The forward enhancement model is enabled separately to perform fusion enhancement on structured T1W / T2W image data and ADC image data, including: The forward enhancement model extracts structural feature data from structured T1W image data based on the 3D U-Net network, extracts texture feature data from structured T2W image data based on the 3D ResNet network, and extracts quantitative feature data from structured ADC image data based on lightweight CNN. Dynamically assign weights to structural feature data, texture feature data, and quantitative feature data; Image enhancement operations are performed on structured T1W / T2W and ADC image data based on the dynamic weights of structural feature data, texture feature data and quantitative feature data and the forward enhancement target loss function, and enhanced MRI image data are generated based on convolution fusion.

3. The MRI image enhancement method based on multimodal fusion according to claim 1, characterized in that: Enable the CT-assisted enhancement model and fuse and enhance the structured T1W / T2W and ADC image data with the forward enhancement model, including: The structured T1W / T2W and ADC image data were alternately enhanced based on the forward enhancement model and the CT-assisted enhancement model, with a total of six rounds of image enhancement operations; The first five rounds of image enhancement operations were performed on the structured T1W / T2W and ADC image data based on the forward enhancement model. After the enhancement was completed, the enhanced T1W / T2W and ADC image data were obtained; When performing the sixth round of image enhancement operation, the CT image data matching the T1W / T2W and ADC image data is called, the CT feature data in the CT image data is extracted based on the CT-assisted enhancement model, and the enhanced T1W / T2W and ADC image data are enhanced again according to the CT feature data, and the enhanced MRI image data is generated based on convolution fusion.

4. The MRI image enhancement method based on multimodal fusion according to claim 3, characterized in that: The enhanced T1W / T2W and ADC image data are further enhanced according to the CT feature data, including: Weights were assigned to CT feature data and enhanced T1W / T2W and ADC image data. The weight of CT feature data was 20%, and the weight of T1W / T2W and ADC image data was 80%. The structural feature data, texture feature data and quantitative feature data in T1W / T2W and ADC image data were retained. The feature fusion ratio of CT feature data and structural feature data, texture feature data and quantitative feature data was iteratively adjusted based on the joint optimization objective function.

5. The MRI image enhancement method based on multimodal fusion according to claim 1, characterized in that: Matching operations based on diagnostic text and medical image databases include: Query the corresponding CT image data in the medical image library based on the patient's unique identity ID; If the patient's unique ID has corresponding CT image data stored in the medical image database, and the interval between the CT image data and the original MRI image data is less than or equal to 7 days, proceed to the registration operation; If the interval between the CT image data and the original MRI image data is greater than seven days or there is no corresponding CT image data in the medical image database, it is determined to be a match failure; The registration operation is represented as registering the CT image data with the original MRI image data; If the CT image data and the original MRI image data are successfully registered, that is, the Dice coefficient is greater than or equal to 0.85, then it is determined that the CT image data and the original MRI image data are successfully matched; If the CT image data and the original MRI image data fail to be registered, that is, the Dice coefficient is less than 0.85, it is determined that the CT image data and the original MRI image data fail to match.

6. The MRI image enhancement method based on multimodal fusion according to claim 5, characterized in that: The registration operation is represented by registering the CT image data with the original MRI image data, including: The registration operation includes rigid registration and elastic registration; The CT image data is rigidly adjusted based on the rigid body registration of translation and rotation operations so that the CT image data is roughly aligned with the original MRI image data; After the rigid body registration is completed, the elastic registration based on the B-spline algorithm is used to compensate the deformation of the CT image data and ensure that the lesion area of ​​the CT image data is aligned with the original MRI image data. At the same time, the Dice coefficient for elastic registration is obtained.

7. The MRI image enhancement method based on multimodal fusion according to claim 1, characterized in that: Structural processing of T1W / T2W and ADC image data based on diagnostic text, including: The unique ID of the patient is obtained based on the diagnosis text, and the original MRI image data in the medical image database is retrieved based on the unique ID of the patient, and artifact detection is performed on the T1W / T2W and ADC image data in the original MRI image data, and artifact repair is performed on the T1W / T2W and ADC image data based on the non-local mean denoising operation and the interpolation algorithm, and the T1W / T2W and ADC image data are normalized, and the T1W / T2W and ADC image data are synchronously adjusted to the same resolution, so as to generate processed T1W / T2W and ADC image data; Based on the natural language processing algorithm, the diagnostic text is segmented and entity recognized, the lesion area keywords and lesion degree keywords are extracted, the image space coordinate range matrix is ​​generated based on the lesion area keywords, the dynamic space constraints are generated based on the lesion degree keywords, the image space coordinate range matrix is ​​data constrained by the dynamic space constraints, and the constrained image space coordinate range matrix is ​​mapped to the T1W / T2W and ADC image data to be processed, so as to mark the lesion area and generate structured T1W / T2W and ADC image data.

8. The MRI image enhancement method based on multimodal fusion according to claim 1, characterized in that: Establish a cross-regional data circulation mechanism, including: The medical image database is divided into four independent storage areas. The data in each independent storage area is interrelated, and data can flow across storage areas. The medical image database is interrelated with the external case system, and the diagnostic text in the external case system can be called. Users can also call the image data stored in the four independent storage areas in the medical image database based on the external case system to view CT images, enhanced MRI images, original MRI images and historical medical images.

9. The MRI image enhancement method based on multimodal fusion according to claim 8, characterized in that: The medical image database is divided into four independent storage areas, including: The four groups of independent storage areas are CT storage area, original storage area, enhanced storage area and history storage area. The CT storage area is used to store CT images, the original storage area is used to store original MRI images, the enhanced storage area is used to store enhanced MRI images, and the history storage area is used to store historical medical images. When storing medical image data, regularly transferring CT images, enhanced MRI images and original MRI images from corresponding independent storage areas to historical storage areas, and simultaneously deleting image data of CT images, enhanced MRI images and original MRI images; The above-mentioned transfer and deletion operations are performed every 30 days.

10. An MRI image enhancement system based on multimodal fusion, characterized in that: include: A database module is used to establish a medical image database, which is used to store CT image data, enhanced MRI image data, original MRI image data and historical medical image data. The medical image database is interconnected with an external case system and can call the diagnosis text in the external case system; An acquisition module, used for acquiring relevant data in a medical image database; A matching module, used for performing matching operations; A processing module, used for performing structured operations on T1W / T2W and ADC image data to obtain structured T1W / T2W and ADC image data; The enhancement module is used to establish a fusion enhancement algorithm model pool, which includes a forward enhancement model, a reverse optimization model and a CT-assisted enhancement model. Based on the forward enhancement model and the reverse optimization model, the structured T1W / T2W and ADC image data are fused and enhanced to obtain enhanced MRI image data. Based on the reverse optimization model, update parameters are generated, and the forward enhancement model and the CT-assisted enhancement model are regularly updated and optimized. The training module is used to train relevant models in the fusion enhancement algorithm model pool.

Images