System for determining positioning of an interventional device in an ultrasound image based on positioning of the interventional device in an x-ray image

Abstract

A medical imaging system can receive an x-ray image of a region of interest of a subject, the region of interest including an interventional device disposed in the region of interest of the subject during an interventional procedure. The medical imaging system can determine a coarse positioning of the interventional device in the x-ray image. The medical imaging system can receive ultrasound data of the region of interest of the subject, the region of interest including the interventional device disposed in the region of interest of the subject. The medical imaging system can determine a refined positioning of the interventional device in the ultrasound data based on the coarse positioning of the interventional device in the x-ray image. The medical imaging system can display an ultrasound image of the region of interest of the subject and the interventional device based on determining the refined positioning of the interventional device in the ultrasound data.

Description

Technical Field

[0001] This disclosure generally relates to a medical imaging system for determining the location of an interventional device in an ultrasound image based on its location in an X-ray image. More specifically, this disclosure relates to a medical imaging system that determines the location of an interventional device in ultrasound data based on its location in an X-ray image to display the patient's region of interest and the ultrasound image of the interventional device positioned within the patient's region of interest during the interventional procedure. Background Technology

[0002] A catheterization lab, or "catheterization room," can refer to an examination room in a hospital or clinical setting that includes various medical imaging systems configured to visualize the structure of the heart. For example, a catheterization lab may include an ultrasound system configured to acquire ultrasound data and generate ultrasound images, or an X-ray system configured to acquire X-ray data and generate X-ray images. During interventional procedures, clinicians can guide interventional devices through the patient's region of interest. For example, during cardiac surgery, clinicians can guide interventional devices through the patient's heart to ablate tissue, insert mitral valve clips, close the left atrial appendage, deliver stents, remove thrombi, and analyze cardiac function. Compared to X-ray images, ultrasound images of the heart provide a clearer and more comprehensive visualization of the heart's structure. However, ultrasound images may not provide a clearer and more comprehensive visualization of interventional devices compared to X-ray images. Summary of the Invention

[0003] This invention describes concepts in more detail in specific embodiments. It should not be used to identify essential features of the claimed subject matter, nor should it be used to limit the scope of the claimed subject matter.

[0004] In one aspect, a medical imaging system may include a memory configured to store instructions; and one or more processors configured to execute the instructions to: receive an X-ray image of a region of interest (ROI) of a subject, the ROI including an interventional device positioned in the ROI during an interventional procedure; determine a coarse location of the interventional device in the X-ray image; receive ultrasound data of the ROI of the subject, the ROI including the interventional device positioned in the ROI of the subject; determine a refined location of the interventional device in the ultrasound data based on the coarse location of the interventional device in the X-ray image; and display the ROI of the subject and the ultrasound image of the interventional device based on the determined refined location of the interventional device in the ultrasound data.

[0005] In another aspect, a method may include: receiving an X-ray image of a region of interest (ROI) of a subject, the ROI including an interventional device positioned in the ROI during an interventional procedure; determining a coarse location of the interventional device in the X-ray image; receiving ultrasound data of the ROI of the subject, the ROI including the interventional device positioned in the ROI of the subject; determining a refined location of the interventional device in the ultrasound data based on the coarse location of the interventional device in the X-ray image; and displaying the ROI of the subject and the ultrasound image of the interventional device based on the determined refined location of the interventional device in the ultrasound data.

[0006] In another aspect, a non-transitory computer-readable medium stores instructions that, when executed by one or more processors, cause the one or more processors to: receive an X-ray image of a region of interest of a subject, the region of interest including an interventional device positioned in the region of interest of the subject during an interventional procedure; determine a coarse location of the interventional device in the X-ray image; acquire ultrasound data of the region of interest of the subject, the region of interest including the interventional device positioned in the region of interest of the subject; determine a refined location of the interventional device in the ultrasound data based on the coarse location of the interventional device in the X-ray image; and display the region of interest of the subject and the ultrasound image of the interventional device based on the determined refined location of the interventional device in the ultrasound data. Attached Figure Description

[0007] Figure 1A This is a diagram of an example system for determining the location of an interventional device in an ultrasound image based on its location in an X-ray image.

[0008] Figure 1B This is an illustration of another example system used to determine the location of an interventional device in an ultrasound image based on its location in an X-ray image.

[0009] Figure 2 In the form of an ultrasound system Figure 1A and / or Figure 1B An illustration of an example component of a medical imaging system.

[0010] Figure 3 In the form of an X-ray system Figure 1A and / or Figure 1B An illustration of an example component of a medical imaging system.

[0011] Figure 4 yes Figure 1B A diagram showing an example component of the tracking system.

[0012] Figure 5 This is a flowchart of an example process for determining the location of an interventional device in an ultrasound image based on its location in an X-ray image.

[0013] Figure 6 This is a diagram illustrating an example procedure for determining the location of an interventional device in an ultrasound image based on its location in an X-ray image.

[0014] Figure 7A and Figure 7B This is a diagram illustrating an example procedure for determining the location of an interventional device in ultrasound data based on its location in an X-ray image to display the patient's region of interest and an ultrasound image of the interventional device positioned within the patient's region of interest during the interventional procedure.

[0015] Figure 8A and Figure 8B This is a diagram illustrating an example procedure for determining the location of an interventional device in ultrasound data based on its location in an X-ray image to display the patient's region of interest and an ultrasound image of the interventional device positioned within the patient's region of interest during the interventional procedure. Detailed Implementation

[0016] During interventional procedures, clinicians can use catheters inserted into one or more medical imaging systems in the laboratory to visualize the location and orientation of the interventional device within the patient's region of interest as the clinician navigates the device. As stated above, echocardiography provides a clearer and more comprehensive visualization of the heart's structure compared to X-ray images. However, echocardiography may not provide a clearer and more comprehensive visualization of interventional devices compared to X-ray images.

[0017] Some embodiments of this document provide a medical imaging system that can: receive an X-ray image of a region of interest (ROI) of a subject, the ROI including an interventional device positioned in the ROI during an interventional procedure; determine a coarse location of the interventional device in the X-ray image; receive ultrasound data of the ROI of the subject, the ROI including the interventional device positioned in the ROI; determine a refined location of the interventional device in the ultrasound data based on the coarse location of the interventional device in the X-ray image; and display the ROI of the subject and the ultrasound image of the interventional device based on the determined refined location of the interventional device in the ultrasound data.

[0018] In this way, the medical imaging system can determine the location of the interventional device in ultrasound data within the patient's region of interest based on its location in X-ray images. The system can then display ultrasound images of the patient's region of interest and the interventional device based on the determined location in the ultrasound data, such as by displaying ultrasound images that clearly and comprehensively depict a specific view of the interventional device, by displaying ultrasound images with adjusted image parameters including the level of the interventional device to more clearly depict it, or by displaying ultrasound images generated based on ultrasound data obtained by controlling the ultrasound probe to correspond to the location of the interventional device.

[0019] In this way, the embodiments of this paper provide an improvement in the field of medical imaging for interventional procedures by utilizing one or more medical imaging systems to clearly and comprehensively visualize the interventional device during the interventional procedure. Furthermore, in this way, the embodiments of this paper provide an improvement to the medical imaging system by allowing the medical imaging system to determine the location of the interventional device in the acquired ultrasound data by utilizing information from X-ray images that more clearly depict the interventional device. By determining the location of the interventional device in the ultrasound data, the medical imaging system can display ultrasound images that clearly and comprehensively depict a specific view of the interventional device, display ultrasound images including adjusted image parameters of the interventional device to more clearly depict the interventional device, display ultrasound images generated based on controlling the ultrasound probe to acquire ultrasound data corresponding to the location of the interventional device, etc.

[0020] Figure 1A This is an illustration of an example system 100 used to determine the location of an interventional device in an ultrasound image based on its location in an X-ray image. (See diagram 100 for example system 100.) Figure 1A As shown, system 100 may include medical imaging system 110 and interventional device 120.

[0021] The medical imaging system 110 may be configured to: receive an X-ray image of a region of interest (ROI) of a patient, the ROI including an interventional device positioned in the ROI during an interventional procedure; determine a coarse location of the interventional device in the X-ray image; receive ultrasound data of the ROI of the patient, the ROI including the interventional device positioned in the ROI; determine a refined location of the interventional device in the ultrasound data based on the coarse location of the interventional device in the X-ray image; and display the ROI and the ultrasound image of the interventional device based on the determined refined location of the interventional device in the ultrasound data. For example, the medical imaging system 110 may be an ultrasound system, an X-ray system, a standalone device, a server, etc.

[0022] System 100 may include one or more medical imaging systems 110. For example, system 100 may include a first medical imaging system 110 (e.g., an ultrasound system), a second medical imaging system 110 (e.g., an X-ray system), etc. Alternatively, system 100 may include a first medical imaging system 110 (e.g., an ultrasound system), a second medical imaging system 110 (e.g., an X-ray system), a third medical imaging system 110 (e.g., a stand-alone device), etc. One or more medical imaging systems 110 may be configured to perform the operations described herein, either jointly or individually.

[0023] Interventional device 120 can be any interventional device capable of being navigated through the patient's region of interest. For example, interventional device 120 can be a catheter, needle, cannula, or end-catheter. Interventional device 120 can be used for a variety of interventional procedures involving the region of interest. For example, a catheter can be used to deliver a stent to an occluded blood vessel, ablate tissue, analyze cardiac function, or remove a thrombus from an occluded blood vessel.

[0024] The subject can be a human, animal, or mannequin. The region of interest can be any anatomical region of the subject. For example, the region of interest could be the heart, brain, organs, blood vessels, etc.

[0025] Figure 1B This is a diagram of an example system 100 for determining the location of an interventional device in an ultrasound image based on its location in an X-ray image. As shown in Figure 1, system 100 may include a medical imaging system 110, an interventional device 120, a tracking system 130, and a network 140.

[0026] Medical imaging system 110 may be configured to: receive an X-ray image of a region of interest (ROI) of a patient, the ROI including an interventional device positioned in the ROI during an interventional procedure; determine a coarse location of the interventional device in the X-ray image; receive ultrasound data of the ROI of the patient, the ROI including the interventional device positioned in the ROI; determine a refined location of the interventional device in the ultrasound data based on the coarse location of the interventional device in the X-ray image; and display the ROI and the ultrasound image of the interventional device based on the determined refined location of the interventional device in the ultrasound data. For example, medical imaging system 110 may be an ultrasound system, an X-ray system, a standalone device, a server, etc. System 100 may include one or more medical imaging systems 110.

[0027] Interventional device 120 can be any interventional device capable of being navigated through the patient's region of interest. For example, interventional device 120 can be a catheter, needle, cannula, or end-catheter. Interventional device 120 can be used for a variety of interventional procedures involving the region of interest. For example, a catheter can be used to deliver a stent to an occluded blood vessel, ablate tissue, analyze cardiac function, or remove a thrombus from an occluded blood vessel.

[0028] The tracking system 130 can be configured to acquire tracking data of the interventional device 120 located in the region of interest of the subject, to acquire tracking data of the medical imaging system 110, to acquire tracking data of the X-ray system 120, etc. For example, the tracking system 130 can be an electromagnetic tracking system, an optical tracking system, an acoustic tracking system, an inertial tracking system, an ultrasound tracking system, etc.

[0029] Network 140 allows communication between medical imaging system 110 and tracking system 130. For example, network 140 can be a local area network (LAN), wide area network (WAN), metropolitan area network (MAN), cellular network, private network, ad hoc network, intranet, Internet, fiber-optic network, wired network, wireless network, and / or a combination of these or other types of networks.

[0030] The number and arrangement of systems 100 are provided as examples. In implementation, systems 100 may include... Figure 1A and / or Figure 1B The systems shown are compared to additional systems, fewer systems, different systems, or systems arranged in a different manner. Additionally or alternatively, the system set of system 100 (e.g., one or more systems) may be integrated into a single system and / or perform one or more functions described as being performed by another system or another set of systems of system 100.

[0031] Figure 2 In the form of an ultrasound system Figure 1A and / or Figure 1B An illustration of example components of a medical imaging system. (See diagram for example.) Figure 2 As shown, the medical imaging system 110 may include an ultrasound probe 202, a transmit beamformer 204, a transmitter 206, an element 208, a receiver 210, a receive beamformer 212, a user input device 214, a processor 216, a display 218, a memory 220, and a communication interface 222. The aforementioned components may be connected via wired or wireless connections.

[0032] When the medical imaging system 110 is an ultrasound system, the medical imaging system 110 can be configured to acquire ultrasound data of the region of interest of the subject. For example, the medical imaging system 110 can be a two-dimensional (2D) ultrasound system, a three-dimensional (3D) ultrasound system, a four-dimensional (4D) ultrasound system, a Doppler ultrasound system, etc.

[0033] The ultrasound probe 202 can be configured to acquire ultrasound data. For example, the ultrasound probe 202 can be a linear probe, a phased array probe, a curved linear probe coupled to a positioning and tracking system, a mechanically manipulated linear array transducer, a phased array transducer, a curved linear array transducer, an electronically manipulated 2D transducer array, an electronic 3D (e3D) probe, an electronic 4D (e4D) probe, or a low-profile wearable patch version of any of the aforementioned probes. According to one embodiment, the ultrasound probe 202 can be configured to generate ultrasound signals, emit ultrasound signals toward the region of interest of a subject, receive echo ultrasound signals backscattered from the region of interest of the subject, generate ultrasound data based on the echo ultrasound signals, and output ultrasound data. The ultrasound probe 202 may include a lens, an acoustic matching layer, a transducer element 208, an acoustic dematching layer, and a backing layer.

[0034] According to one embodiment, the lens can be configured to direct the ultrasonic signal toward the region of interest of the subject. For example, the lens can be made of silicone, epoxy resin, rubber, etc. According to one embodiment, an acoustic matching layer can be configured to facilitate impedance matching, which can exist between a relatively high-impedance transducer element and a relatively low-impedance subject. For example, the acoustic matching layer can be made of graphite, plastic, resin, etc. According to one embodiment, transducer element 208 can be configured to receive element-specific transmitted signals, convert element-specific transmitted signals into ultrasonic signals, and transmit ultrasonic signals toward the region of interest. Additionally or alternatively, transducer element 208 can be configured to receive echo signals reflected or backscattered from the region of interest, convert the echo signals into electrical signals, and transmit electrical signals. For example, transducer element 308 can be a piezoelectric material, such as Pb(Mg) 1 / 3 Nb 2 / 3 )O3-PbTiO3 ("PMN-PT"), Pb(In 1 / 2 Nb 1 / 2 O3-Pb(Mg) 1 / 3 Nb 2 / 3Examples of suitable acoustic dematching layers include PIN-PMN-PT (Pb(ZrTi)O3) and Pb(ZrTi) (PZT). According to one embodiment, the acoustic dematching layer can be configured to reduce insertion loss and enhance the frequency bandwidth of the transducer element 208. For example, the acoustic dematching layer can be tungsten carbide, silicon carbide, etc. According to one embodiment, the backing layer can be configured to attenuate the ultrasonic signal guided from the transducer element 208 in the opposite direction to the subject, and to attenuate the ultrasonic signal deflected by the housing of the ultrasonic probe 202. For example, the backing layer can be epoxy resin, metal, etc.

[0035] Transmit beamformer 204 can be configured to apply a delay time to the electrical signal provided to element 208 to focus the corresponding ultrasonic signal at the region of interest. Transmitter 206 can be configured to send an electrical signal to element 208 to drive element 208 to emit an ultrasonic signal toward the region of interest. Element 208 can be configured to receive the electrical signal from transmitter 206, convert the electrical signal into an ultrasonic signal, and emit the ultrasonic signal toward the region of interest. Element 208 can be configured to receive the echo ultrasonic signal backscattered from the region of interest, convert the echo ultrasonic signal into an electrical signal, and provide the electrical signal to receiver 210. Receiver 210 can be configured to receive the electrical signal from element 208 and provide the electrical signal to receiver beamformer 212. Receiver beamformer 212 can apply a delay time to the electrical signal received from element 208.

[0036] User input device 214 may be configured to receive user input and provide the user input to processor 216. For example, user input device 214 may be a user interface, touchscreen display, keyboard, keypad, mouse, button, switch, microphone, etc. Additionally or alternatively, user input device 214 may be configured to sense information. For example, user input device 214 may sense information from an electromagnetic positioning system, inertial measurement system, accelerometer, gyroscope, actuator, etc.

[0037] Processor 216 may be configured to perform the operations described herein. For example, processor 216 may be a central processing unit (CPU), graphics processing unit (GPU), accelerated processing unit (APU), microprocessor, microcontroller, digital signal processor (DSP), field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), or another type of processing component. Processor 216 may be implemented in hardware, firmware, or a combination of hardware and software. Processor 216 may include one or more processors 216 configured to perform the operations described herein. For example, a single processor 216 may be configured to perform all the operations described herein. Alternatively, multiple processors 216 may be collectively configured to perform all the operations described herein, and each of the multiple processors 216 may be configured to perform a subset of the operations described herein. For example, a first processor 216 may perform a first subset of the operations described herein, a second processor 216 may be configured to perform a second subset of the operations described herein, and so on.

[0038] Processor 216 can be configured to control ultrasound probe 202 to acquire ultrasound data. Processor 216 can be configured to control which elements in element 208 are active and to control the shape of the beam emitted from ultrasound probe 202. Processor 216 can generate ultrasound images for display. For example, processor 216 can generate B-mode images, color Doppler images, M-mode images, or color M-mode images, etc. Ultrasound images can be 3D images, 2D images, single-plane images, dual-plane images, tri-plane images, or multi-plane images, etc. Ultrasound images can correspond to various anatomical planes (e.g., sagittal, coronal, and transverse) of the region of interest.

[0039] Display 218 can be configured to display information. For example, display 218 can be a monitor, LED display, cathode ray tube, projection display, touch screen, tablet computer, or mobile phone. Display 218 can display ultrasound images based on ultrasound data in real time. For example, display 218 can display ultrasound images within one second, two seconds, five seconds, etc., after ultrasound probe 202 acquires ultrasound data.

[0040] Memory 220 may be configured to store information and / or instructions for use by processor 216. Memory 220 may be a non-transitory computer-readable medium. For example, memory 220 may be random access memory (RAM), read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, and / or optical memory) for storing information and / or instructions for use by processor 216. Memory 220 may be configured to store instructions that, when executed by processor 216, cause processor 216 to perform the operations described herein.

[0041] The communication interface 222 may be configured to enable the processor 216 to communicate with other systems, such as via a wired connection, a wireless connection, or a combination of wired and wireless connections. For example, the communication interface 222 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, or a cellular network interface, etc.

[0042] Figure 2 The number and arrangement of components in the ultrasound system 110 shown are provided as an example. In implementation, the medical imaging system 110 may include components related to… Figure 2 The components shown are compared to additional components, fewer components, different components, or components arranged in a different manner. Additionally or alternatively, the set of components of the medical imaging system 110 (e.g., one or more components) can perform one or more functions described as being performed by another set of components of the medical imaging system 110.

[0043] Figure 3 In the form of an X-ray system Figure 1A and / or Figure 1B An illustration of example components of a medical imaging system. (See diagram for example.) Figure 3 As shown, the X-ray system 120 may include a C-arm 302, an X-ray source 304, an X-ray detector 306, a user input device 308, a processor 310, a display 312, a memory 314, a communication interface 316, and a picture archiving and communication system (PACS) 318.

[0044] When the medical imaging system 110 is an X-ray system, it can be configured to acquire X-ray data of the region of interest of the subject. For example, the medical imaging system 110 can be a direct radiography system, a computational radiography system, a computed tomography system, etc.

[0045] The C-arm 302 may be configured to support the X-ray source 304 and the X-ray detector 306. The X-ray source 304 may be configured to emit X-ray radiation in the form of an X-ray beam toward the subject and the X-ray detector 306. The X-ray detector 306 may be configured to detect the X-ray radiation emitted by the X-ray source 304 and attenuated by the subject.

[0046] User input device 308 may be configured to receive user input and provide the user input to processor 310. For example, user input device 308 may be a user interface, touchscreen display, keyboard, keypad, mouse, button, switch, or microphone, etc. Additionally or alternatively, user input device 308 may be configured to sense information. For example, user input device 308 may sense information from an electromagnetic positioning system, inertial measurement system, accelerometer, gyroscope, or actuator, etc.

[0047] Processor 310 may be configured to control x-ray source 304 and x-ray detector 306. For example, processor 310 may be a CPU, GPU, APU, microprocessor, microcontroller, DSP, FPGA, or ASIC. Processor 310 may be implemented using hardware, firmware, or a combination of hardware and software. Processor 310 may include one or more processors 310 configured to perform the operations described herein. For example, a single processor 310 may be configured to perform all the operations described herein. Alternatively, multiple processors 310 may be collectively configured to perform all the operations described herein, and each of the multiple processors 310 may be configured to perform a subset of the operations described herein. For example, a first processor 310 may perform a first subset of the operations described herein, a second processor 310 may be configured to perform a second subset of the operations described herein, and so on.

[0048] Display 312 may be configured to display information. For example, display 312 may be a monitor, LED display, cathode ray tube, projection display, touch screen, tablet computer, or mobile phone. Memory 314 may be configured to store information and / or instructions for use by processor 310. Memory 314 may be a non-transitory computer-readable medium. For example, memory 314 may be RAM, ROM, flash memory, magnetic memory, or optical memory. Memory 314 may be configured to store instructions that, when executed by processor 310, cause processor 310 to perform the operations described herein.

[0049] Communication interface 316 can be configured to enable processor 310 to communicate with other systems, such as via a wired connection, a wireless connection, or a combination of wired and wireless connections. For example, communication interface 316 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, an RF interface, a USB interface, a Wi-Fi interface, or a cellular network interface. PACS 318 can be configured to communicate with external systems and / or networks to allow users at various locations to access X-ray images.

[0050] Figure 4 yes Figure 1B A diagram illustrating example components of the tracking system 130. (See diagram for example.) Figure 4 As shown, the tracking system 130 may include a transmitter 402, a receiver 404, a user input device 406, a processor 408, a display 410, a memory 412, and a communication interface 414.

[0051] Transmitter 402 can be configured to generate a magnetic field. Receiver 404 can be configured to output a signal in response to the magnetic field generated by transmitter 402. Processor 408 can receive the output signal from receiver 404 and acquire tracking data identifying the location and / or orientation of receiver 404. According to one embodiment, receiver 404 can be attached to medical imaging system 110, interventional device 120, etc. For example, receiver 404 can be attached to ultrasound probe 202 to track the location and / or orientation of ultrasound probe 202. As another example, receiver 404 can be attached to x-ray source 304 or x-ray detector 306 to track the location and / or orientation of x-ray source 304 or x-ray detector 306. As another example, receiver 404 can be attached to interventional device 120 to track the location and / or orientation of interventional device 120. As another example, receiver 404 can be attached to a feature in a region of interest.

[0052] User input device 406 may be configured to receive user input and provide the user input to processor 408. For example, user input device 406 may be a touchscreen display, keyboard, keypad, mouse, button, switch, or microphone. Additionally or alternatively, user input device 406 may be configured to sense information. For example, user input device 406 may sense information from an electromagnetic positioning system, inertial measurement system, accelerometer, gyroscope, or actuator.

[0053] Processor 408 may be configured to perform the operations described herein. For example, processor 408 may be a CPU, GPU, APU, microprocessor, microcontroller, DSP, FPGA, or ASIC, etc. Processor 408 may be implemented using hardware, firmware, or a combination of hardware and software. Processor 408 may include one or more processors 408 configured to perform the operations described herein. For example, a single processor 408 may be configured to perform all the operations described herein. Alternatively, multiple processors 408 may be collectively configured to perform all the operations described herein, and each of the multiple processors 408 may be configured to perform a subset of the operations described herein. For example, a first processor 408 may perform a first subset of the operations described herein, a second processor 408 may be configured to perform a second subset of the operations described herein, and so on.

[0054] Processor 408 can be configured to control transmitter 402 to acquire tracking data. Processor 408 can be configured to control the excitation of transmitter 402 to generate a magnetic field. Processor 408 can acquire tracking data based on controlling transmitter 402.

[0055] Display 410 can be configured to display information. For example, display 410 can be a monitor, LED display, cathode ray tube, projection display, touch screen, tablet computer, or mobile phone. Display 410 can display tracking data in real time. For example, display 410 can display tracking data within one second, two seconds, five seconds, etc., of the currently acquired tracking data.

[0056] Memory 412 may be configured to store information and / or instructions for use by processor 408. Memory 412 may be a non-transitory computer-readable medium. For example, memory 412 may be RAM, ROM, flash memory, magnetic memory, or optical memory, etc. Memory 412 may be configured to store instructions that, when executed by processor 408, cause processor 408 to perform the operations described herein.

[0057] The communication interface 414 may be configured to enable the processor 408 to communicate with other systems, such as via a wired connection, a wireless connection, or a combination of wired and wireless connections. For example, the communication interface 414 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, an RF interface, a USB interface, a Wi-Fi interface, or a cellular network interface.

[0058] Figure 4 The number and arrangement of components in the tracking system 130 shown are provided as an example. In implementation, the tracking system 130 may include components related to... Figure 4 The components shown are compared to additional components, fewer components, different components, or components arranged in a different manner. Additionally or alternatively, the set of components of the tracking system 130 (e.g., one or more components) may perform one or more functions described as being performed by another set of components of the tracking system 130.

[0059] although Figure 4 Tracking system 130 is described as an electromagnetic tracking system, but it should be understood that the embodiments described herein are applicable to other types of tracking systems, such as optical tracking systems, acoustic tracking systems, ultrasonic tracking systems, or AI-based tracking methods.

[0060] Figure 5This is a flowchart of an example procedure 500 for determining the location of an interventional device in an ultrasound image based on its location in an X-ray image. Procedure 500 may be performed by a medical imaging system 110. Furthermore, procedure 500 may be performed by one or more medical imaging systems 110. For example, a first medical imaging system 110 (e.g., an ultrasound system) may perform one or more operations of procedure 500, a second medical imaging system 110 (e.g., an X-ray system) may perform one or more operations of procedure 500, and so on.

[0061] like Figure 5 As shown, process 500 may include receiving an X-ray image of a region of interest (ROI) of a patient, which includes an interventional device positioned in that ROI during an interventional procedure (operation 510). For example, medical imaging system 110 may receive an X-ray image of a ROI of a patient, which includes an interventional device 120 positioned in that ROI during an interventional procedure. Medical imaging system 110 may receive X-ray images during an interventional procedure in which the interventional device 120 is navigated through the patient's ROI. During the interventional procedure, the interventional device 120 may be positioned within and navigated through the ROI. Therefore, the X-ray image can depict the interventional device 120.

[0062] like Figure 5 As further shown, process 500 may include determining the approximate location of the interventional device in the X-ray image (operation 520). For example, medical imaging system 110 may determine the approximate location of the interventional device 120 in the X-ray image.

[0063] According to one implementation, the medical imaging system 110 can use image processing techniques to determine the approximate location of the interventional device 120 in an X-ray image. For example, the image processing techniques may include segmentation techniques, pattern matching techniques, feature extraction techniques, image analysis techniques, edge detection techniques, or image registration techniques. The medical imaging system 110 can use image processing techniques to analyze the X-ray image and determine the approximate location of the interventional device 120 in the X-ray image.

[0064] According to another implementation, the medical imaging system 110 can use an AI model to determine the approximate location of the interventional device 120 in the X-ray image. For example, the medical imaging system 110 can input the X-ray image into the AI ​​model and determine the approximate location of the interventional device 120 in the X-ray image based on the output of the AI ​​model. The AI ​​model can be a convolutional neural network (CNN) model, a residual neural network, a random forest model, a decision tree model, an artificial neural network (ANN), a Naive Bayes model, a decision tree, a recurrent neural network (RNN), a logistic regression model, or a support vector machine, etc.

[0065] The medical imaging system 110 can determine one or more coordinates corresponding to a coarse location. For example, these one or more coordinates can be coordinates of a coordinate system of the medical imaging system 110 and the tracking system 130. The coordinate system can be a coordinate system of a catheter insertion laboratory that includes the medical imaging system 110 and the X-ray system 120. According to one embodiment, these one or more coordinates can be coordinates of the shape of the interventional device 120 in the X-ray image. Additionally or alternatively, these one or more coordinates can be coordinates of a search space corresponding to the shape of the interventional device 120 in the X-ray image. For example, the search space can be a three-dimensional (3D) shape based on the shape of the interventional device 120 in the X-ray image. For example, the shape of the interventional device 120 in the X-ray image can be a two-dimensional (2D) shape because the X-ray image is 2D. Because the X-ray image is 2D, the 3D location and orientation of the interventional device 120 in the region of interest may not be fully determined from the X-ray image. Therefore, the search space can represent a depiction of the potential location and / or orientation of the interventional device 120 in the region of interest.

[0066] According to one embodiment, the medical imaging system 110 may determine a search space based on one or more coordinates of coarse positioning. For example, the medical imaging system 110 may extend one or more coordinates in one or more directions to determine the search space. Additionally or alternatively, the medical imaging system 110 may determine the search space based on one or more coordinates of coarse positioning and a predetermined model. For example, the predetermined model may correspond to the search space and may be a 3D model. The medical imaging system 110 may determine the coordinates of the search space based on one or more coordinates of coarse positioning and the dimensions of the 3D model. The 3D model may be a shape similar to that of the interventional device 120. Alternatively, the 3D model may be a shape different from that of the interventional device 120. In any case, the search space may encompass the shape of the interventional device 120 and may potentially encompass additional space around the interventional device 120 to take into account the potential positioning and orientation of the interventional device 120.

[0067] like Figure 5 As further shown, process 500 may include receiving ultrasound data of a region of interest (ROI) of the patient, which includes an interventional device positioned within the RIO (operation 530). For example, medical imaging system 110 may receive ultrasound data of a RIO of the patient, which includes an interventional device 120 positioned within the RIO. Medical imaging system 110 may receive ultrasound data during interventional procedures based on user manipulation of ultrasound probe 202 relative to the patient.

[0068] like Figure 5As further shown, process 500 may include determining a refined location of the interventional device in ultrasound data based on the coarse location of the interventional device in the X-ray image (operation 540). For example, medical imaging system 110 may determine the refined location of interventional device 120 in ultrasound data based on the coarse location of interventional device 120 in the X-ray image.

[0069] According to one implementation, refined positioning can be based on one or more coordinates of the interventional device 120 in ultrasound data, determined according to one or more coordinates of coarse positioning. These one or more coordinates can be coordinates of the coordinate system of the medical imaging system 110 and the tracking system 130. The coordinate system can be the coordinate system of the catheter insertion laboratory that includes the medical imaging system 110 and the tracking system 130.

[0070] According to one embodiment, refined positioning may be based on the coordinates of the X-ray image to determine one or more coordinates of the interventional device 120 in the ultrasound data. Alternatively, refined positioning may be based on the coordinate system of the tracking system 130 to determine one or more coordinates of the interventional device 120 in the ultrasound data. Alternatively, refined positioning may be based on the coordinate system of the medical imaging system 110 that acquired the X-ray image to determine one or more coordinates of the interventional device 120 in the ultrasound data.

[0071] According to one embodiment, the medical imaging system 110 can determine a refined location based on one or more coordinates of a coarse location. For example, if one or more coordinates of the coarse location depict a search space, the medical imaging system 110 can analyze ultrasound data in the search space to determine the refined location of the interventional device 120 in the search space. In other words, the search space may represent a depiction of the potential location and / or orientation of the interventional device 120 in the region of interest, and the refined location of the interventional device 120 in the ultrasound data may represent the actual location and / or orientation of the interventional device 120 in the region of interest.

[0072] According to one embodiment, the medical imaging system 110 can determine a refined location based on the positioning of the ultrasound probe 202 in an X-ray image. For example, the medical imaging system 110 can analyze the X-ray image to determine the positioning of the ultrasound probe 202 in the X-ray image, and determine a refined location of the interventional device 120 in the ultrasound data based on the positioning of the ultrasound probe 202 in the X-ray image.

[0073] According to one implementation, the medical imaging system 110 can use image processing techniques to determine the refined location of the interventional device 120 within the ultrasound data. For example, the image processing techniques may include segmentation, pattern matching, feature extraction, image analysis, edge detection, or image registration. The medical imaging system 110 can use image processing techniques to analyze the ultrasound data and determine the refined location of the interventional device 120 within the ultrasound data.

[0074] According to another implementation, the medical imaging system 110 can use an AI model to determine the refined location of the interventional device 120 in the X-ray image. For example, the medical imaging system 110 can input ultrasound data into the AI ​​model and determine the refined location of the interventional device 120 in the ultrasound data based on the output of the AI ​​model. The AI ​​model can be a CNN model, residual neural network, random forest model, decision tree model, ANN, Naive Bayes model, decision tree, RNN, logistic regression model, or support vector machine, etc.

[0075] According to one implementation, the medical imaging system 110 can analyze a search space in ultrasound data to determine the refined location of the interventional device 120 within the ultrasound data. As described above, the search space can depict the potential location and / or orientation of the interventional device 120 in the ultrasound data. By analyzing only the search space, rather than the entire dataset of ultrasound data, the medical imaging system 110 can determine the refined location of the interventional device 120 in the ultrasound data more quickly and efficiently.

[0076] like Figure 5 As further shown, process 500 may include displaying ultrasound images of the patient's region of interest and the interventional device based on determining the refined location of the interventional device in the ultrasound data (operation 550). For example, medical imaging system 110 may display the patient's region of interest and ultrasound images of the interventional device 120 within the patient's region of interest based on determining the refined location of the interventional device 120 in the ultrasound data.

[0077] According to one embodiment, the medical imaging system 110 can display ultrasound images based on a view that depicts the interventional device 120 within the patient's region of interest. For example, the medical imaging system 110 can determine a view depicting the interventional device 120 within the patient's region of interest and display an ultrasound image corresponding to the determined view. As an example, the medical imaging system 110 can display one or more ultrasound images corresponding to various axes of the interventional device 120 to help the user assess the location of the interventional device 120 within the patient's region of interest.

[0078] According to one embodiment, the medical imaging system 110 can display an ultrasound image based on adjusting image parameters of the ultrasound data at the level of the interventional device 120. For example, image parameters may include color, opacity, hue, brightness, pattern, etc. The medical imaging system 110 can adjust the image parameters of the ultrasound data at the level of the interventional device 120 to improve the visibility of the interventional device 120 in the ultrasound image and / or enhance the interventional device 120 in the ultrasound image. According to one embodiment, the medical imaging system 110 can adjust image parameters of specific portions of the interventional device 120, such as the tip of the catheter, the tip of the guidewire, the "wings" of the mitral valve clip, etc.

[0079] According to one embodiment, the medical imaging system 110 can display an ultrasound image based on ultrasound data acquired by controlling the ultrasound probe 202 to correspond to a refined location of the interventional device 120. For example, the medical imaging system 110 can determine the refined location of the interventional device 120 in the ultrasound data and control the ultrasound probe 202 to acquire additional ultrasound data corresponding to the refined location.

[0080] although Figure 5 Specific operations and sequences of operations are described, but it should be understood that other implementations may include different operations and / or different sequences of operations.

[0081] Figure 6 This is a diagram illustrating an example procedure 600 for determining the location of an interventional device in an ultrasound image based on its location in an X-ray image. (See diagram 600 for example.) Figure 6 As shown, the medical imaging system 110 can receive an X-ray image 602 of a region of interest (ROI) of a subject, which includes an interventional device 120 disposed in that ROI. Figure 6 As further shown, the medical imaging system 110 can determine the approximate location 604 of the interventional device 120 in the X-ray image 602. Furthermore, as shown, the medical imaging system 110 can determine the shape 606 of the interventional device 120 in the X-ray image 602. The medical imaging system 110 can determine a search space 608 based on the shape 606. The search space can represent a depiction of the potential location and / or orientation of the interventional device 120 in the region of interest. Figure 6 As further shown, the medical imaging system 110 can receive ultrasound data 610 of a region of interest (ROI) of the subject, which includes an interventional device 120 disposed within the ROI. For example... Figure 6As further illustrated, the medical imaging system 110 can determine the refined location of the interventional device 120 presented in the ultrasound data based on the coarse location of the interventional device 120 in the x-ray image 602. For example, as shown, the medical imaging system 110 can determine the refined location of the interventional device 120 in the ultrasound data 610 based on the search space 608 and / or shape 606.

[0082] Figure 7A and Figure 7B This is an illustration of an example procedure 700 for determining the location of an interventional device in ultrasound data based on its location in X-ray images to display the patient's region of interest and the ultrasound image of the interventional device positioned within the patient's region of interest during the interventional procedure. (See diagram 700.) Figure 7A As shown, the medical imaging system 110 can display a 3D ultrasound image 702 corresponding to the region of interest (ROI) of the patient, a first 2D ultrasound image 704 corresponding to a first view of the ROI, a second 2D ultrasound image 706 corresponding to a second view of the ROI, and a third 2D ultrasound image 708 corresponding to a third view of the ROI. An interventional device 120 (e.g., a catheter) may be positioned within the ROI of the patient. However, the interventional device 120 may not be readily visible in the 3D ultrasound image 702, the first 2D ultrasound image 704, the second 2D ultrasound image 706, and / or the third 2D ultrasound image 708. Figure 7B As shown, the medical imaging system 110 can display a 3D ultrasound image 710 corresponding to the region of interest of the subject, a first 2D ultrasound image 712 corresponding to a first view of the region of interest of the subject, a second 2D ultrasound image 714 corresponding to a second view of the region of interest of the subject, and a third 2D ultrasound image 716 corresponding to a third view of the region of interest of the subject. As indicated by reference numerals 718 and 720, the interventional device 120 may be visible in the 3D ultrasound image 710 and the third 2D ultrasound image 716. The medical imaging system 110 can be combined with the above description. Figure 5 The described similar approach is based on determining the refined positioning of the interventional device 120 in the ultrasound data to display the ultrasound images from... Figure 7A The display is updated to Figure 7B The display.

[0083] Figure 8A and Figure 8B This is an illustration of an example procedure 800 for determining the location of an interventional device in ultrasound data based on its location in X-ray images to display the patient's region of interest and the ultrasound image of the interventional device positioned within the patient's region of interest during the interventional procedure. (See diagram 800.) Figure 8AAs shown, the medical imaging system 110 can display a 3D ultrasound image 802 corresponding to the patient's region of interest (ROI), a first 2D ultrasound image 804 corresponding to a first view of the ROI, a second 2D ultrasound image 806 corresponding to a second view of the ROI, and a third 2D ultrasound image 808 corresponding to a third view of the ROI. An interventional device 120 (e.g., a mitral valve clip) can be positioned within the ROI. However, the interventional device 120 may not be readily visible in the 3D ultrasound image 802, the first 2D ultrasound image 804, the second 2D ultrasound image 806, and / or the third 2D ultrasound image 808. Figure 8B As shown, the medical imaging system 110 can display a 3D ultrasound image 810 corresponding to the region of interest of the subject, a first 2D ultrasound image 812 corresponding to a first view of the region of interest of the subject, a second 2D ultrasound image 814 corresponding to a second view of the region of interest of the subject, and a third 2D ultrasound image 816 corresponding to a third view of the region of interest of the subject. As indicated by reference numerals 818 and 820, the interventional device 120 may be visible in the first 2D ultrasound image 812 and the third 2D ultrasound image 816. The medical imaging system 110 can be combined with the above description. Figure 5 The described similar approach is based on determining the refined positioning of the interventional device 120 in the ultrasound data to display the ultrasound images from... Figure 8A The display is updated to Figure 8B The display.

[0084] According to one implementation, the medical imaging system 110 may use AI models. One or more AI models may be associated with training, deployment, and monitoring phases. During the training phase, the medical imaging system 110 may receive and process training data to generate a trained model. The training data may be generated, received, or otherwise obtained from internal and / or external resources.

[0085] Typically, a trained model may comprise a set of variables (e.g., nodes, neurons, or filters) tuned to different values ​​through the application of training data (e.g., weighting or biasing). According to one implementation, the training process may employ supervised, unsupervised, semi-supervised, and / or reinforcement learning processes to train the model. According to one implementation, a portion of the training data may be retained during training and / or used to validate the trained model.

[0086] For supervised learning processes, training data may include labels or scores that can facilitate the training process by providing ground truth values. For example, labels or scores may indicate the model's output. Training can continue by feeding a training dataset including the training data into the model. The model may have variables set to initial values ​​(e.g., randomly, based on Gaussian noise, or based on pre-trained values). The model may generate outputs based on the training dataset being fed into it. The outputs can be compared to corresponding labels or scores (e.g., ground truth values) indicating known outputs, and then backpropagated through the model to adjust the values ​​of the variables. This process can be repeated for multiple samples, at least until a determined loss or error falls below a predefined threshold. According to one implementation, some of the training data may be retained and used for further validation or testing of the trained model.

[0087] For unsupervised learning processes, training data may not include pre-assigned labels or scores to aid the learning process. Instead, unsupervised learning processes can include clustering, classification, etc., to identify patterns naturally present in the training data. For example, training data can be clustered into groups based on identified similarities and / or patterns. K-means clustering or K-nearest neighbors can also be used, and these can be supervised or unsupervised. A combination of K-nearest neighbors and unsupervised clustering techniques can also be used. For semi-supervised learning, a combination of training data with pre-assigned labels or scores and training data without pre-assigned labels or scores can be used to train the model.

[0088] When reinforcement learning is employed, an agent (e.g., an algorithm) can be trained to make decisions based on training data through trial and error. For example, based on the decisions made, the agent can then receive feedback (e.g., a positive reward if the predicted value is above a predetermined threshold), adjust its next decision to maximize the reward, and repeat until the loss function is optimized.

[0089] After being trained, the trained model can be stored and subsequently applied by the medical imaging system 110 during the deployment phase. For example, during the deployment phase, the trained model executed by the medical imaging system 110 can receive input data. During the deployment phase, the trained model can perform actions such as combining... Figure 5 One or more operations as described.

[0090] The embodiments described herein provide an improvement in the field of medical imaging for interventional procedures by utilizing X-ray and ultrasound systems to clearly and comprehensively visualize interventional devices during interventional procedures. Furthermore, in this way, the embodiments described herein provide an improvement to ultrasound systems by allowing the ultrasound system to determine the location of the interventional device in the acquired ultrasound data using information from X-ray images that more clearly depict the interventional device. By determining the location of the interventional device in the ultrasound data, the ultrasound system can display ultrasound images that clearly and comprehensively depict a specific view of the interventional device, display ultrasound images including adjusted image parameters of the interventional device to more clearly depict the interventional device, display ultrasound images generated based on ultrasound data obtained by controlling the ultrasound probe corresponding to the location of the interventional device, etc.

[0091] The embodiments shown in the accompanying drawings and described above are merely illustrative embodiments and are not intended to limit the scope of the appended claims, including any equivalents included within the scope of the claims. Various modifications are possible and will be apparent to those skilled in the art. Any combination of non-mutually exclusive features described herein is intended to be within the scope of the invention. That is, features of the described embodiments may be combined with any suitable aspect described above, and optional features of any aspect may be combined with any other suitable aspect. Similarly, features listed in dependent claims may be combined with non-mutually exclusive features of other dependent claims, particularly where the dependent claims are subordinate to the same independent claim. In some jurisdictions that claim single-claim dependents, such single-claim dependents may have been used in practice, but this should not be construed as meaning that features in dependent claims are mutually exclusive.

Claims

1. A medical imaging method, the medical imaging method comprising: Receive X-ray images of a region of interest in a patient, the region of interest including an interventional device placed in the patient's region of interest during the interventional procedure; Determine the approximate location of the interventional device in the X-ray image; Acquire ultrasound data of the region of interest of the subject, wherein the region of interest includes the interventional device disposed in the region of interest of the subject; The detailed positioning of the interventional device in the ultrasound data is determined based on the coarse positioning of the interventional device in the X-ray image. as well as The region of interest of the subject and the ultrasound image of the interventional device are displayed based on the refined positioning of the interventional device in the ultrasound data.

2. The method according to claim 1, further comprising: A view depicting the interventional device in the ultrasound data is determined based on the refined positioning of the interventional device in the ultrasound data. as well as The ultrasound image corresponding to the view is displayed.

3. The method according to claim 1, further comprising: Adjust the image parameters of the ultrasound data at the horizontal position of the interventional device; as well as The ultrasound image is displayed by adjusting the image parameters.

4. The method according to claim 1, further comprising: Control the ultrasound probe to obtain additional ultrasound data corresponding to the refined positioning of the interventional device; as well as The ultrasound image is displayed by controlling the ultrasound probe to obtain the additional ultrasound data.

5. The method according to claim 1, further comprising: The search space is determined based on the rough positioning of the interventional device in the X-ray image; as well as The refined positioning of the interventional device in the ultrasound data is determined based on the search space.

6. The method according to claim 1, further comprising: The refined positioning of the interventional device in the ultrasound data is determined based on the tracking data acquired by the tracking system.

7. The method according to claim 1, further comprising: The refined positioning of the interventional device in the ultrasound data is determined based on the positioning of the ultrasound probe in the X-ray image.

8. A medical imaging system, the medical imaging system comprising: A memory configured to store instructions; as well as One or more processors, the one or more processors being configured to perform the method according to any one of claims 1 to 7.

9. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the method according to any one of claims 1 to 7.

Images