Multimodal real-time imaging fusion method and system for interventional MRI therapy

Abstract

This application discloses a multimodal real-time imaging fusion method and system for MRI interventional therapy, relating to the technical field of medical diagnostics. It includes acquiring the patient's real-time respiratory gating signal, triggering an acquisition process to obtain ultrasound and MRI images; fusing the ultrasound and MRI images to obtain a multimodal image data stream; outlining vascular-neural structures and highlighting their borders, triggering an alarm when an interventional device approaches the highlighted area; spatially registering the multimodal image data stream to obtain spatially aligned fused image data; and overlaying the ultrasound and MRI images of the target lesion area, with a virtual knob controlling the display transparency of the ultrasound and MRI images. This application improves the accuracy and efficiency of multimodal image fusion during MRI interventional therapy.

Description

Technical Field

[0001] This application relates to the technical field of medical diagnostics, and in particular to a multimodal real-time imaging fusion method and system for interventional magnetic resonance imaging therapy. Background Technology

[0002] Magnetic resonance imaging (MRI) is one of the core tools for precision diagnosis in modern medicine. In interventional procedures using MRI, it is often necessary to integrate images from other modalities, such as ultrasound, to provide complementary information.

[0003] In existing technologies, conventional ultrasound equipment is easily interfered with in the strong magnetic field of MRI, making it difficult to achieve stable synchronization with MRI scans and resulting in difficulties in the spatiotemporal registration of multi-source images. Secondly, image fusion often relies on complex post-processing software algorithms, resulting in poor real-time performance and susceptibility to artifacts caused by patient breathing and other physiological movements. Furthermore, the simultaneous display of multiple images leads to information clutter, increasing the cognitive burden on physicians and hindering the rapid and accurate identification of lesions and key anatomical structures, while also lacking proactive warnings for intraoperative instruments and dangerous areas. Therefore, how to efficiently, accurately, and systematically fuse multimodal images from MRI interventional procedures has become an urgent problem to be solved. Summary of the Invention

[0004] The purpose of this invention is to provide a multimodal real-time imaging fusion method and system for interventional magnetic resonance imaging (MRI) therapy, in order to solve the problems mentioned in the background art.

[0005] In a first aspect, this application provides a multimodal real-time imaging fusion method for interventional magnetic resonance imaging (MRI) therapy, the method comprising: The patient's real-time respiratory gating signal is acquired, and the acquisition process of the magnetically compatible ultrasound probe and MRI scan sequence hardware is triggered based on the respiratory gating signal in a preset respiratory phase to obtain ultrasound images and MRI images. The first timestamp of the ultrasound image and the second timestamp of the MRI image are extracted. After synchronizing and aligning the first timestamp and the second timestamp, the ultrasound image and the MRI image are fused to obtain a multimodal image data stream. Based on the ultrasound and MRI images, the vascular-nerve structure is delineated and marked with a highlighted border. When the interventional device approaches the area marked with the highlighted border, an alarm is triggered. The imaging features of the interventional device in the MRI image and the ultrasound image are obtained. Using the preset marker points on the interventional device as a common spatial reference, the multimodal image data stream is spatially registered to obtain spatially aligned fused image data. The fused image data is used to identify lesions to obtain target lesion areas. The ultrasound image and the MRI image of the target lesion area are then superimposed and displayed. A virtual knob is set to control the display transparency of the ultrasound image and the MRI image.

[0006] Preferably, the steps of acquiring the patient's real-time respiratory gating signal, and obtaining ultrasound and MRI images based on the respiratory gating signal in a preset respiratory phase-triggered acquisition process of the magnetically compatible ultrasound probe and MRI scan sequence hardware, are as follows: The patient's real-time respiratory cycle is acquired, and the patient's real-time respiratory gating signal is obtained based on the real-time respiratory cycle. The time point at the end of the patient's expiration is marked as the respiratory phase. An image acquisition signal is generated based on the respiratory gating signal during the respiratory phase, and the image acquisition signal is sent to the magnetically compatible ultrasound probe and MRI scan sequence hardware. After receiving the image acquisition signal, the MRI scanning sequence hardware performs a rapid single-frame scan of the patient to obtain an MRI image; The magnetically compatible ultrasound probe continuously monitors the patient with ultrasound and, upon receiving the image acquisition signal, performs instantaneous image acquisition to obtain an ultrasound image.

[0007] Preferably, after the step of sending the image acquisition signal to the magnetically compatible ultrasound probe and the MRI scan sequence hardware, the method further includes: The ultrasound acquisition process involves pre-acquiring images using a magnetically compatible ultrasound probe for instantaneous image acquisition, and the MRI acquisition process involves rapid single-frame scanning using MRI scanning sequence hardware. Record the ultrasound image generation process nodes in the ultrasound acquisition process and the nuclear magnetic resonance (NMR) image generation process nodes in the NMR acquisition process; The ultrasound image generation process node and the MRI image generation process node are timestamped, and the start time difference between the ultrasound acquisition process and the MRI acquisition process is recorded. Based on the start time difference, a redundant time period is generated. Extract the ultrasound start time point of the ultrasound acquisition process and the MRI start time point of the MRI acquisition process, and determine whether the ultrasound start time point is earlier than the MRI start time point. If it is determined that the ultrasound start point is earlier than the MRI start point, then the redundant time period is added before the MRI acquisition process; If it is determined that the ultrasound start point is later than the MRI start point, then the redundant time period is added before the ultrasound acquisition process.

[0008] Preferably, the step of extracting the first timestamp of the ultrasound image and the second timestamp of the MRI image, synchronizing and aligning the first timestamp and the second timestamp, and then fusing the ultrasound image and the MRI image to obtain a multimodal image data stream specifically includes: Based on the ultrasound image generation process node and the MRI image generation process node, the first timestamp of the ultrasound image and the second timestamp of the MRI image are obtained respectively; The ultrasound image and the MRI image are streamed to obtain an ultrasound data stream and an MRI data stream, respectively. After aligning the first timestamp with the second timestamp, the ultrasound data stream and the MRI data stream are fused to obtain a multimodal image data stream.

[0009] Preferably, the step of outlining the vascular-nerve structure based on the ultrasound image and the MRI image, highlighting the outline, and triggering an alarm when the interventional device approaches the highlighted area is as follows: Image recognition is performed on the ultrasound image to obtain the patient's blood flow imaging image in the ultrasound image; Image recognition is performed on the MRI images to obtain the patient's angiography and nerve distribution images from the MRI images; By combining the blood flow imaging images, the angiography images, and the nerve distribution images, the patient's vascular-nerve structure is constructed; The minimum threat range of blood vessels and nerves is extracted separately, and the blood vessel-nerve structure is marked with a border highlight based on the minimum threat range; When using interventional instruments, the system identifies whether the instrument is close to the highlighted area of ​​the border marker. If the instrument enters the highlighted area, a threat alarm is triggered to the inspector.

[0010] Preferably, the step of acquiring the imaging features of the interventional device in the MRI image and the ultrasound image, and using preset markers on the interventional device as a common spatial reference to spatially register the multimodal image data stream to obtain spatially aligned fused image data specifically includes: The first imaging feature and the second imaging feature of the interventional device are acquired from the MRI image and the ultrasound image, respectively. Based on the first imaging feature and the second imaging feature, the position of the first marker point, the direction of the first marker point, the position of the second marker point, and the direction of the second marker point on the interventional device are respectively identified. A common layer reference is generated based on the positions of the first and second marker points, and a common direction reference is generated based on the directions of the first and second marker points. A common spatial reference is obtained based on the common layer reference and the common direction reference, and spatial registration is performed on the multimodal image data stream based on the common spatial reference to obtain fused image data.

[0011] Preferably, the steps of identifying lesions in the fused image data to obtain target lesion regions, overlaying the ultrasound image and the MRI image of the target lesion region, and setting a virtual knob to control the display transparency of the ultrasound image and the MRI image are as follows: The fused image data is used to identify lesions to obtain target lesion regions, and the target ultrasound image region and target MRI image region of the target lesion region are extracted. The target ultrasound image region and the target MRI image region are made semi-transparent and stained, and then superimposed for display. A virtual ultrasound knob and a virtual MRI knob are respectively provided for the target ultrasound image region and the target MRI image region; The ultrasound virtual knob and the MRI virtual knob respectively control the stepless adjustment of the transparency of the target ultrasound image region and the target MRI image region; The image region outside the target ultrasound image and the target MRI image region is marked as the background image region. The background image region includes the ultrasound image or the MRI image, and a virtual change button is provided for rotating the ultrasound image and the MRI image.

[0012] Secondly, this application provides a multimodal real-time imaging fusion system for magnetic resonance imaging interventional therapy, the system comprising: Image acquisition module: used to acquire the patient's real-time respiratory gating signal, and based on the respiratory gating signal, trigger the acquisition process of the magnetically compatible ultrasound probe and MRI scan sequence hardware in a preset respiratory phase to obtain ultrasound images and MRI images; Time fusion module: used to extract the first timestamp of the ultrasound image and the second timestamp of the MRI image, synchronize and align the first timestamp and the second timestamp, and then fuse the ultrasound image and the MRI image to obtain a multimodal image data stream; Danger alarm module: used to delineate vascular-nerve structures based on the ultrasound image and the MRI image, and mark them with a highlighted border. When the interventional device approaches the area marked with the highlighted border, an alarm is triggered. Image fusion module: used to acquire the imaging features of interventional instruments in the MRI image and the ultrasound image, and to perform spatial registration of the multimodal image data stream using preset marker points on the interventional instrument as a common spatial reference to obtain spatially aligned fused image data; Fusion display module: used to identify lesions in the fused image data, obtain the target lesion area, overlay the ultrasound image and the MRI image of the target lesion area, and set a virtual knob to control the display transparency of the ultrasound image and the MRI image.

[0013] In summary, this application includes at least one of the following beneficial technical effects: By acquiring the patient's real-time respiratory cycle, the real-time respiratory gating signal is obtained. The acquisition process of the magnetically compatible ultrasound probe and MRI scanning sequence hardware is triggered at a preset respiratory phase to acquire ultrasound and MRI images. The first timestamp of the ultrasound image and the second timestamp of the MRI image are extracted, aligned, and then fused to obtain a multimodal image data stream. Vascular-nerve structures are then delineated based on the ultrasound and MRI images, and highlighted with borders. An alarm is triggered when the interventional device approaches a blood vessel or nerve. The imaging characteristics of the interventional device in the MRI and ultrasound images are identified, and spatial registration of the multimodal image data stream is performed using preset markers on the interventional device as the industrial control spatial reference, resulting in spatially aligned fused image data. The fused image data is then used to identify the target lesion area. The ultrasound and MRI images of the target lesion area are overlaid and displayed, with two virtual knobs for each to control the transparency of the overlay. This improves the accuracy and efficiency of multimodal image fusion during MRI interventional therapy. Attached Figure Description

[0014] Figure 1 This is a flowchart of the steps of the multimodal real-time imaging fusion method for interventional magnetic resonance imaging provided in the embodiments of this application; Figure 2 This is a block diagram of a multimodal real-time imaging fusion system for interventional magnetic resonance therapy provided in this application embodiment.

[0015] Explanation of reference numerals in the attached diagram: 1. Image acquisition module; 2. Time fusion module; 3. Hazard alarm module; 4. Image fusion module; 5. Fusion display module. Detailed Implementation

[0016] The following is in conjunction with the appendix Figures 1-2 This application will be described in further detail, but the embodiments of the present invention are not limited thereto.

[0017] This application discloses a multimodal real-time imaging fusion method and system for interventional magnetic resonance imaging therapy.

[0018] In this embodiment, a multimodal real-time imaging fusion method for interventional magnetic resonance imaging (MRI) therapy is provided, comprising: S100: Acquires the patient's real-time respiratory gating signal, and triggers the acquisition process of the magnetically compatible ultrasound probe and MRI scan sequence hardware based on the respiratory gating signal in the preset respiratory phase to obtain ultrasound images and MRI images; S200: Extract the first timestamp of the ultrasound image and the second timestamp of the MRI image. After synchronizing and aligning the first timestamp and the second timestamp, fuse the ultrasound image and the MRI image to obtain a multimodal image data stream. S300: Based on ultrasound and MRI images, outline the vascular-nerve structure and highlight the borders. When the interventional device approaches the highlighted area, an alarm is triggered. S400: Acquire the imaging features of interventional devices in MRI and ultrasound images, use the preset markers on the interventional devices as a common spatial reference, perform spatial registration on the multimodal image data stream, and obtain spatially aligned fused image data; S500: Performs lesion identification on fused image data to obtain the target lesion area, overlays the ultrasound and MRI images of the target lesion area, and sets a virtual knob to control the display transparency of the ultrasound and MRI images.

[0019] The steps for acquiring the patient's real-time respiratory gating signal, triggering the acquisition process of the magnetically compatible ultrasound probe and MRI scan sequence hardware based on the respiratory gating signal at a preset respiratory phase, and obtaining ultrasound and MRI images are as follows: The patient's real-time respiratory cycle is acquired, and the patient's real-time respiratory gating signal is obtained based on the real-time respiratory cycle. The time point at the end of the patient's expiration is marked as the respiratory phase. Image acquisition signals are generated during the respiratory phase based on respiratory gating signals, and the image acquisition signals are sent to the magnetically compatible ultrasound probe and MRI scan sequence hardware; After receiving the image acquisition signal, the MRI scanning sequence hardware performs a rapid single-frame scan of the patient to obtain an MRI image; The magnetically compatible ultrasound probe continuously monitors the patient with ultrasound and acquires instantaneous images upon receiving the image acquisition signal, thus obtaining ultrasound images.

[0020] In practice, taking a 65-year-old liver cancer patient as an example, the system needs to acquire the patient's real-time respiratory gating signal for synchronous image acquisition. First, the system continuously monitors the patient's respiratory movements using pressure sensors in the chest and abdomen to obtain the real-time respiratory cycle. For example, a complete respiratory cycle is monitored to be 5 seconds, with approximately 2 seconds of inspiration and 3 seconds of expiration. The system generates a respiratory gating signal based on the real-time respiratory cycle and marks the end of expiration as the respiratory phase. Specifically, when the patient completes expiration and the chest cavity is relatively still, the system records this time point as the respiratory phase. For example, at timestamp 09:30:15.000, the system detects the end of expiration and immediately generates an image acquisition signal. This signal is simultaneously sent to a magnetically compatible ultrasound probe and MRI scanning sequence hardware. Upon receiving the signal, the MRI scanning sequence hardware immediately performs a rapid single-frame scan of the patient's liver region, with the scan time controlled within 200 milliseconds, obtaining a clear MRI image. Meanwhile, the magnetically compatible ultrasound probe continuously monitors the same area with ultrasound and instantly acquires an image upon receiving the image acquisition signal, with an acquisition time of approximately 50 milliseconds, resulting in a single real-time ultrasound image. In this way, both images are acquired in the same phase when the patient's breathing is relatively stable, minimizing artifacts caused by respiratory motion and laying the foundation for subsequent precise fusion.

[0021] After the step of sending the image acquisition signal to the magnetically compatible ultrasound probe and MRI scan sequence hardware, the following is also included: The ultrasound acquisition process involves pre-acquiring images using a magnetically compatible ultrasound probe for instantaneous image acquisition, and the MRI acquisition process involves rapid single-frame scanning using MRI scanning sequence hardware. Record the ultrasound image generation process nodes in the ultrasound acquisition process and the MRI image generation process nodes in the MRI acquisition process; The ultrasound image generation process node and the MRI image generation process node are timestamped, the start time difference between the ultrasound acquisition process and the MRI acquisition process is recorded, and a redundant time period is generated based on the start time difference. Extract the ultrasound start time point of the ultrasound acquisition process and the MRI start time point of the MRI acquisition process, and determine whether the ultrasound start time point is earlier than the MRI start time point. If it is determined that the ultrasound start point is earlier than the MRI start point, then the redundant time period is added before the MRI acquisition process. If it is determined that the ultrasound start point is later than the MRI start point, then the redundant time period is added before the ultrasound acquisition process.

[0022] In application, taking a 65-year-old liver cancer patient as an example, after the system sends the image acquisition signal, to ensure complete time synchronization between ultrasound and MRI images, the system pre-analyzes the acquisition processes of both devices. The system first records the complete process of instantaneous image acquisition by the magnetically compatible ultrasound probe, including probe initialization, ultrasound transmission, echo reception, data conversion, and image generation, marking the key process node for ultrasound image generation as T1. Simultaneously, the system records the process of rapid single-frame scanning of the MRI scan sequence hardware, including gradient field switching, radio frequency pulse transmission, signal acquisition, and image reconstruction, marking the key process node for MRI image generation as T2. Next, the system timestamps T1 and T2, finding that due to hardware startup delay, the start time of the ultrasound acquisition process is 0.1 seconds earlier than the start time of the MRI acquisition process, i.e., a 0.1-second start time difference. Based on this time difference, the system generates a 0.2-second redundant time period as a buffer. Since the ultrasound start point is earlier than the MRI start point, the system adds this 0.2-second redundant time period before the start of the MRI acquisition process. This means that when the next respiratory phase is triggered, the MRI scan sequence hardware will enter the preparation state 0.2 seconds in advance, waiting for the image acquisition signal to arrive, thereby ensuring that the two image acquisition actions can be triggered at the same time, achieving microsecond-level time synchronization, and effectively avoiding image misalignment caused by asynchronous device response time.

[0023] The steps for extracting the first timestamp of the ultrasound image and the second timestamp of the MRI image, synchronizing and aligning the first and second timestamps, and then fusing the ultrasound image and the MRI image to obtain a multimodal image data stream are as follows: Based on the ultrasound image generation process nodes and the MRI image generation process nodes, the first timestamp of the ultrasound image and the second timestamp of the MRI image are obtained respectively. The ultrasound images and MRI images were streamed to obtain ultrasound data stream and MRI data stream, respectively. After aligning the first and second timestamps, the ultrasound data stream and the MRI data stream are fused to obtain a multimodal image data stream.

[0024] In application, taking a 65-year-old liver cancer patient as an example, after completing time-synchronized acquisition, the system needs to fuse the acquired ultrasound and MRI images. First, the system extracts the timestamps of the two images based on the previously recorded ultrasound and MRI image generation process nodes. For example, the ultrasound image timestamp (first timestamp) is 09:30:15.123, and the MRI image timestamp (second timestamp) is 09:30:15.125. Since the two timestamps are very close but still have slight differences, the system calibrates the second timestamp to be completely consistent with the first timestamp, i.e., both are adjusted to 09:30:15.123, completing the timestamp alignment. Subsequently, the system performs data streaming processing on the ultrasound and MRI images separately. For the ultrasound image, its pixel matrix is ​​converted into a continuous binary data stream to form the ultrasound data stream. For the MRI image, its DICOM format data is similarly converted into a continuous binary data stream to form the MRI data stream. Finally, the system fuses the timestamp-aligned ultrasound and MRI data streams. The fusion process uses timestamps as indexes to overlay and register two sets of data streams at the same time, generating a continuous multimodal image data stream containing multimodal information (such as real-time structure of ultrasound and anatomical details of MRI), providing a unified data source for subsequent real-time display and analysis.

[0025] Based on ultrasound and MRI images, the vascular-nerve structure is delineated and highlighted with borders. An alarm is triggered when the interventional device approaches the highlighted area. The specific steps are as follows: Image recognition is performed on ultrasound images to obtain blood flow imaging images of the patient within the ultrasound images; Image recognition was performed on MRI images to obtain angiographic and nerve distribution images of the patient within the MRI images; By combining blood flow imaging images, angiography images, and nerve distribution images, the patient's vascular-neural structure was constructed; Extract the minimum threat range of blood vessels and nerves respectively, and highlight the borders of blood vessel-nerve structures according to the minimum threat range; When using interventional instruments, the system identifies whether the instrument is approaching the highlighted area marked on the border. If the instrument enters the highlighted area, a threat alarm is triggered to the inspector.

[0026] In practice, taking a 65-year-old liver cancer patient as an example, the system needs to delineate the vascular and neural structures of the patient's liver region to mitigate surgical risks. First, the system performs image recognition on the acquired ultrasound images, using Doppler imaging technology to identify and extract blood flow signals, generating a clear blood flow image. The image displays blood flow velocity and direction using color coding. Simultaneously, the system recognizes MRI images, extracting the liver's vascular network structure through angiography sequences (such as MRA) to generate angiography images; and delineates the nerve distribution images of important areas such as the porta hepatis through specific neural imaging sequences or registration based on anatomical atlases. Next, the system overlays and registers the blood flow image, angiography image, and nerve distribution image to construct a comprehensive three-dimensional vascular-neural structure model including the location, direction, diameter of blood vessels, and the direction of nerves. Then, the system sets safety thresholds: for major blood vessels (such as the hepatic artery), a 3mm radius around them is set as the lowest threat range; for important nerve plexuses, a 2mm radius around them is set as the lowest threat range. Based on these ranges, the system highlights the edges of blood vessels and nerve structures in the fused image, displaying them as a striking flashing red border. As the physician moves the interventional device (such as a radiofrequency ablation needle) across the image, the system monitors the distance between the device tip and the highlighted border in real time. Once the device tip enters the highlighted area (i.e., less than 3 mm or 2 mm from a blood vessel or nerve), the system immediately issues a "approaching danger zone" alarm to the physician via sound and screen flashing, prompting the physician to adjust the operating path.

[0027] The steps for acquiring the imaging features of interventional devices in MRI and ultrasound images, using pre-defined markers on the interventional devices as a common spatial reference, and spatially registering the multimodal image data streams to obtain spatially aligned fused image data are as follows: The first and second imaging features of the interventional device were acquired from MRI and ultrasound images, respectively. Based on the first imaging feature and the second imaging feature, the position and direction of the first marker point, the position and direction of the second marker point on the interventional device are identified respectively; A common layer reference is generated based on the positions of the first and second marker points, and a common direction reference is generated based on the directions of the first and second marker points. A common spatial reference is obtained based on the common layer reference and the common direction reference. The multimodal image data stream is then spatially registered based on the common spatial reference to obtain fused image data.

[0028] In this application, taking a 65-year-old liver cancer patient as an example, to achieve precise spatial alignment between ultrasound and MRI images, the system uses preset markers on the interventional device as a reference for registration. In this example, the interventional device is a magnetically compatible biopsy needle with two ring-shaped reflective markers. The system first automatically identifies the biopsy needle in both MRI and ultrasound images. In the MRI image, due to metal artifacts and signal characteristics, the biopsy needle appears as a specific low-signal linear shadow. The system acquires its imaging features (first imaging features), such as length, curvature, and pixel coordinates of the two markers (e.g., point A(150,200,50), point B(155,205,55)) and direction vector. In the ultrasound image, the biopsy needle appears as a hyperechoic bright line. The system acquires its imaging features (second imaging features), such as length, direction of the bright line, and pixel coordinates of the two markers (e.g., point A'(152,201), point B'(157,206)) and direction. Next, based on the coordinates of two identified marker points in the two images, the system calculates a common layer reference using a spatial transformation matrix, ensuring that points A and A', and points B and B', coincide spatially. Simultaneously, the system calculates and unifies the directional reference based on the direction vector of the line connecting the two marker points. Ultimately, the system obtains a precise common spatial reference (including the origin, coordinate axes, and scaling factor). Based on this common spatial reference, the system performs frame-by-frame spatial geometric transformations and interpolation operations on the multimodal image data stream composed of ultrasound and MRI data streams, ensuring complete alignment of the same anatomical structure in the two images. This results in the output of spatially precisely registered fused image data, providing doctors with misaligned fusion visual guidance.

[0029] The steps for identifying lesions in the fused image data to obtain the target lesion region, overlaying the ultrasound and MRI images of the target lesion region, and setting a virtual knob to control the display transparency of the ultrasound and MRI images are as follows: Lesion identification is performed on the fused image data to obtain the target lesion region, and the target ultrasound image region and target MRI image region of the target lesion region are extracted. The target ultrasound image region and the target MRI image region are made semi-transparent and stained, and then superimposed for display. The target ultrasound image region and the target MRI image region are respectively equipped with ultrasound virtual knobs and MRI virtual knobs; The ultrasound virtual knob and the MRI virtual knob respectively control the stepless adjustment of the transparency of the target ultrasound image area and the target MRI image area; Image areas outside the target ultrasound and MRI image areas are marked as background image areas. The background image areas include ultrasound or MRI images and are equipped with virtual change buttons for switching between ultrasound and MRI images.

[0030] In this application, taking a 65-year-old liver cancer patient as an example, after obtaining spatially aligned fused image data, the system identifies and optimizes the display of the lesion area. The system uses a pre-trained deep learning model to automatically analyze the fused image data, identifying a suspected tumor area of ​​approximately 2.5 cm in diameter in the right lobe of the liver and marking it as the target lesion area. The system extracts the corresponding ultrasound image sub-region (target ultrasound image area) and MRI image sub-region (target MRI image area) from this area. To highlight the lesion, the system renders the target ultrasound image area as a semi-transparent orange-yellow and the target MRI image area as a semi-transparent blue-purple. These two stained semi-transparent areas are then superimposed pixel-wise, resulting in a unique mixed color for the fused lesion area, creating a striking contrast with the surrounding normal tissue. In the sidebar of the display interface, the system provides virtual knobs for "Ultrasound Transparency Adjustment" and "MRI Transparency Adjustment," respectively. Doctors can drag the knobs with the mouse to adjust the display transparency of the two areas steplessly from 0% to 100%. For example, adjusting the transparency of the ultrasound area to 30% and the MRI area to 70% can emphasize the observation of fine anatomical structures provided by the MRI image; conversely, adjusting the transparency can emphasize the observation of real-time blood flow information provided by the ultrasound image. Images outside the lesion area are marked as background image areas, initially displayed as grayscale MRI images. The interface also includes a virtual "Switch Background" button, which allows for a one-click switch to the ultrasound image background for easy comparison by doctors.

[0031] This invention provides a multimodal real-time imaging fusion system for interventional MRI therapy, using any of the above-described multimodal real-time imaging fusion methods for interventional MRI therapy. The system includes the following: Image acquisition module 1: Used to acquire the patient's real-time respiratory gating signal, and based on the respiratory gating signal, trigger the acquisition process of the magnetically compatible ultrasound probe and MRI scan sequence hardware at a preset respiratory phase to obtain ultrasound images and MRI images; Time fusion module 2: used to extract the first timestamp of the ultrasound image and the second timestamp of the MRI image, synchronize and align the first timestamp and the second timestamp, and then fuse the ultrasound image and the MRI image to obtain a multimodal image data stream; Danger alarm module 3: It is used to delineate vascular-nerve structures based on ultrasound and MRI images and highlight the borders. When the interventional device approaches the area highlighted by the border, an alarm is triggered. Image fusion module 4: Used to acquire the imaging features of interventional devices in MRI and ultrasound images, and to spatially register the multimodal image data streams using preset markers on the interventional devices as a common spatial reference, so as to obtain spatially aligned fused image data; Fusion display module 5: Used to identify lesions from fused image data, obtain target lesion areas, overlay ultrasound and MRI images of the target lesion areas, and set a virtual knob to control the display transparency of ultrasound and MRI images.

[0032] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A multimodal real-time imaging fusion method for interventional magnetic resonance imaging (MRI) therapy, characterized in that, include: The patient's real-time respiratory gating signal is acquired, and the acquisition process of the magnetically compatible ultrasound probe and MRI scan sequence hardware is triggered based on the respiratory gating signal in a preset respiratory phase to obtain ultrasound images and MRI images. The first timestamp of the ultrasound image and the second timestamp of the MRI image are extracted. After synchronizing and aligning the first timestamp and the second timestamp, the ultrasound image and the MRI image are fused to obtain a multimodal image data stream. Based on the ultrasound and MRI images, the vascular-nerve structure is delineated and marked with a highlighted border. When the interventional device approaches the area marked with the highlighted border, an alarm is triggered. The imaging features of the interventional device in the MRI image and the ultrasound image are obtained. Using the preset marker points on the interventional device as a common spatial reference, the multimodal image data stream is spatially registered to obtain spatially aligned fused image data. The fused image data is used to identify lesions to obtain target lesion areas. The ultrasound image and the MRI image of the target lesion area are then superimposed and displayed. A virtual knob is set to control the display transparency of the ultrasound image and the MRI image.

2. The multimodal real-time imaging fusion method for interventional magnetic resonance imaging therapy according to claim 1, characterized in that, The steps for acquiring the patient's real-time respiratory gating signal, and obtaining ultrasound and MRI images based on the respiratory gating signal in a preset respiratory phase-triggered acquisition process using a magnetically compatible ultrasound probe and MRI scanning sequence hardware, are as follows: The patient's real-time respiratory cycle is acquired, and the patient's real-time respiratory gating signal is obtained based on the real-time respiratory cycle. The time point at the end of the patient's expiration is marked as the respiratory phase. An image acquisition signal is generated based on the respiratory gating signal during the respiratory phase, and the image acquisition signal is sent to the magnetically compatible ultrasound probe and MRI scan sequence hardware. After receiving the image acquisition signal, the MRI scanning sequence hardware performs a rapid single-frame scan of the patient to obtain an MRI image; The magnetically compatible ultrasound probe continuously monitors the patient with ultrasound and, upon receiving the image acquisition signal, performs instantaneous image acquisition to obtain an ultrasound image.

3. The multimodal real-time imaging fusion method for interventional magnetic resonance imaging therapy according to claim 2, characterized in that, After the step of sending the image acquisition signal to the magnetically compatible ultrasound probe and MRI scan sequence hardware, the method further includes: The ultrasound acquisition process involves pre-acquiring images using a magnetically compatible ultrasound probe for instantaneous image acquisition, and the MRI acquisition process involves rapid single-frame scanning using MRI scanning sequence hardware. Record the ultrasound image generation process nodes in the ultrasound acquisition process and the nuclear magnetic resonance (NMR) image generation process nodes in the NMR acquisition process; The ultrasound image generation process node and the MRI image generation process node are timestamped, and the start time difference between the ultrasound acquisition process and the MRI acquisition process is recorded. Based on the start time difference, a redundant time period is generated. Extract the ultrasound start time point of the ultrasound acquisition process and the MRI start time point of the MRI acquisition process, and determine whether the ultrasound start time point is earlier than the MRI start time point. If it is determined that the ultrasound start point is earlier than the MRI start point, then the redundant time period is added before the MRI acquisition process; If it is determined that the ultrasound start point is later than the MRI start point, then the redundant time period is added before the ultrasound acquisition process.

4. The multimodal real-time imaging fusion method for interventional magnetic resonance imaging therapy according to claim 3, characterized in that, The steps of extracting the first timestamp of the ultrasound image and the second timestamp of the MRI image, synchronizing and aligning the first timestamp and the second timestamp, and then fusing the ultrasound image and the MRI image to obtain a multimodal image data stream are as follows: Based on the ultrasound image generation process node and the MRI image generation process node, the first timestamp of the ultrasound image and the second timestamp of the MRI image are obtained respectively; The ultrasound image and the MRI image are streamed to obtain an ultrasound data stream and an MRI data stream, respectively. After aligning the first timestamp with the second timestamp, the ultrasound data stream and the MRI data stream are fused to obtain a multimodal image data stream.

5. The multimodal real-time imaging fusion method for interventional magnetic resonance imaging therapy according to claim 4, characterized in that, Based on the ultrasound and MRI images, the vascular-nerve structures are delineated and highlighted with borders. An alarm is triggered when the interventional device approaches the highlighted area. Specifically, the steps are as follows: Image recognition is performed on the ultrasound image to obtain the patient's blood flow imaging image in the ultrasound image; Image recognition is performed on the MRI images to obtain the patient's angiography and nerve distribution images from the MRI images; By combining the blood flow imaging images, the angiography images, and the nerve distribution images, the patient's vascular-nerve structure is constructed; The minimum threat range of blood vessels and nerves is extracted separately, and the blood vessel-nerve structure is marked with a border highlight based on the minimum threat range; When using interventional instruments, the system identifies whether the instrument is close to the highlighted area of ​​the border marker. If the instrument enters the highlighted area, a threat alarm is triggered to the inspector.

6. The multimodal real-time imaging fusion method for interventional magnetic resonance imaging therapy according to claim 5, characterized in that, The steps of acquiring the imaging features of the interventional device in the MRI and ultrasound images, using preset markers on the interventional device as a common spatial reference, and spatially registering the multimodal image data stream to obtain spatially aligned fused image data are as follows: The first imaging feature and the second imaging feature of the interventional device are acquired from the MRI image and the ultrasound image, respectively. Based on the first imaging feature and the second imaging feature, the position of the first marker point, the direction of the first marker point, the position of the second marker point, and the direction of the second marker point on the interventional device are respectively identified. A common layer reference is generated based on the positions of the first and second marker points, and a common direction reference is generated based on the directions of the first and second marker points. A common spatial reference is obtained based on the common layer reference and the common direction reference, and spatial registration is performed on the multimodal image data stream based on the common spatial reference to obtain fused image data.

7. The multimodal real-time imaging fusion method for interventional magnetic resonance imaging therapy according to claim 6, characterized in that, The steps of identifying lesions in the fused image data to obtain target lesion regions, overlaying the ultrasound image and the MRI image of the target lesion region, and setting a virtual knob to control the display transparency of the ultrasound image and the MRI image are as follows: The fused image data is used to identify lesions to obtain target lesion regions, and the target ultrasound image region and target MRI image region of the target lesion region are extracted. The target ultrasound image region and the target MRI image region are made semi-transparent and stained, and then superimposed for display. A virtual ultrasound knob and a virtual MRI knob are respectively provided for the target ultrasound image region and the target MRI image region; The ultrasound virtual knob and the MRI virtual knob respectively control the stepless adjustment of the transparency of the target ultrasound image region and the target MRI image region; The image region outside the target ultrasound image and the target MRI image region is marked as the background image region. The background image region includes the ultrasound image or the MRI image, and a virtual change button is provided for rotating the ultrasound image and the MRI image.

8. A multimodal real-time imaging fusion system for interventional magnetic resonance imaging (MRI) therapy, wherein the system uses the multimodal real-time imaging fusion method for interventional magnetic resonance imaging (MRI) therapy as described in any one of claims 1-7, characterized in that, The system includes: Image acquisition module: used to acquire the patient's real-time respiratory gating signal, and based on the respiratory gating signal, trigger the acquisition process of the magnetically compatible ultrasound probe and MRI scan sequence hardware in a preset respiratory phase to obtain ultrasound images and MRI images; Time fusion module: used to extract the first timestamp of the ultrasound image and the second timestamp of the MRI image, synchronize and align the first timestamp and the second timestamp, and then fuse the ultrasound image and the MRI image to obtain a multimodal image data stream; Danger alarm module: used to delineate vascular-nerve structures based on the ultrasound image and the MRI image, and mark them with a highlighted border. When the interventional device approaches the area marked with the highlighted border, an alarm is triggered. Image fusion module: used to acquire the imaging features of interventional instruments in the MRI image and the ultrasound image, and to perform spatial registration of the multimodal image data stream using preset marker points on the interventional instrument as a common spatial reference to obtain spatially aligned fused image data; Fusion display module: used to identify lesions in the fused image data, obtain the target lesion area, overlay the ultrasound image and the MRI image of the target lesion area, and set a virtual knob to control the display transparency of the ultrasound image and the MRI image.

Images