Interventional medical device tracking

Abstract

A system comprising an interface (193) to an optical shape sensing device (102) having a shape following a shape of an interventional medical device (01). The system further comprises a controller (190) having a memory (191) storing instructions and a processor (192) executing the instructions. The instructions cause the system to identify the shape of the optical shape sensing device (102) using optical shape sensing signals received via the interface (193), identify the interventional medical device (01) in different coordinate spaces of an imaging system imaging the interventional medical device (01) in different imaging modes, and register the coordinate spaces to each other and to the interventional medical device (01). The instructions further cause the system to obtain a segmented representation (401) of the interventional medical device (01), and re-register the interventional medical device (01) to one of the coordinate spaces based on registering the interventional medical device (01) to the one of the coordinate spaces using the segmented representation (401).

Description

Background Technology

[0001] Optical shape sensing (OSS) technology is used to provide real-time, in-process information on the shape and relative position of interventional medical devices during interventional procedures. Information from OSS is used to locate and navigate the interventional medical device during the procedure. During the interventional procedure, OSS uses light along a multi-core optical fiber conforming to the shape of the interventional medical device. The principle involved utilizes distributed strain measurements within the optical fiber using a characteristic Rayleigh backscattering, or controlled grating pattern. The shape along the fiber begins at a specific point, called the launch, or z = 0, and the subsequent shape, position, and orientation are relative to the launch.

[0002] Registration is used to align the coordinate systems of two separate devices and / or systems. For example, it can be achieved via a transformation T from an OSS device to an X-ray imaging system. OX This enables registration from OSS equipment to X-ray imaging equipment. It allows for conversion from ultrasound imaging systems to X-ray imaging systems via T... UX This enables registration from an ultrasound imaging system to an X-ray imaging system. It allows for conversion from an OSS device to an ultrasound imaging system via T... OU This is to achieve registration from OSS equipment to ultrasound imaging system.

[0003] Furthermore, segmentation is used to represent the surface of a structure as a three-dimensional model in medical imaging systems.

[0004] Currently, registration between the OSS device and the X-ray imaging system can accumulate significant errors, for example, when the proximal end of the OSS device (i.e., the end closest to the user) moves a few centimeters. Incorrect corrections require re-registration between the OSS device and the X-ray imaging system, which in turn necessitates two new offset X-ray projections. These additional X-ray projections can disrupt interventional procedures, prolong their duration, and subject patients and clinicians to additional X-ray doses.

[0005] The configuration between the OSS device and the ultrasound imaging system can also accumulate significant errors. While the image analysis software searches for the OSS device in the latest ultrasound images, re-registration between the OSS device and the ultrasound imaging system can disrupt the interventional procedure and prolong its duration. The image analysis software can require the OSS device specified in the ultrasound image, for example, from the tip of the OSS device specified by the user in the ultrasound image, to constrain the search for the OSS device in the ultrasound image. Even with initial constraints, the OSS device must be fully identified and located in the ultrasound image for re-registration between the OSS device and the ultrasound imaging system, which disrupts the interventional procedure and prolongs its duration. Summary of the Invention

[0006] According to one aspect of this disclosure, a system for tracking the position of an interventional medical device during an interventional medical procedure includes an interface and a controller. The interface is connected to an optical shape sensing device having a shape that follows the shape of the interventional medical device during the interventional medical procedure. The controller includes a memory storing instructions and a processor executing the instructions. When executed by the processor, the instructions cause the system to identify the shape of the optical shape sensing device using optical shape sensing signals received from the optical shape sensing device via the interface, and, based on the identified shape of the optical shape sensing device, identify the interventional medical device in a first coordinate space of a first imaging system that images the interventional medical device in a first imaging mode during the interventional medical procedure. The instructions also cause the system to register the interventional medical device to the first coordinate space, identify the interventional medical device in a second coordinate space of a second imaging system that images the interventional medical device in a second imaging mode during the interventional medical procedure, and register the first coordinate space of the first imaging system to the second coordinate space of the second imaging system. The instructions also cause the system to segment the interventional medical device in the second coordinate space to obtain a segmented representation of the interventional medical device in the second coordinate space, register the interventional medical device to the second coordinate space using the segmented representation of the interventional medical device, and re-register the interventional medical device to the first coordinate space based on the registration of the interventional medical device to the second coordinate space.

[0007] According to another aspect of this disclosure, a tangible, non-transitory computer-readable storage medium stores a computer program. When executed by a processor, the computer program causes a system including the tangible, non-transitory computer-readable storage medium to identify the shape of an optical shape sensing device using optical shape sensing signals received via an interface from the optical shape sensing device, the optical shape sensing device having a shape that follows the shape of an interventional medical procedure, and, based on the identified shape of the optical shape sensing device, to identify the interventional medical device in a first coordinate space of a first imaging system, the first imaging system imaging the interventional medical device in a first imaging mode during the interventional medical procedure. The instructions also cause the system to register the interventional medical device to the first coordinate space, identify the interventional medical device in a second coordinate space of a second imaging system, the second imaging system imaging the interventional medical device in a second imaging mode during the interventional medical procedure, and register the first coordinate space of the first imaging system to the second coordinate space of the second imaging system. The instructions also cause the system to segment the interventional medical device in the second coordinate space to obtain a segmented representation of the interventional medical device in the second coordinate space, register the interventional medical device to the second coordinate space using the segmented representation of the interventional medical device, and re-register the interventional medical device to the first coordinate space based on the registration of the interventional medical device to the second coordinate space.

[0008] According to another aspect of this disclosure, a method for tracking the position of an interventional medical device during an interventional medical procedure includes: identifying the shape of an optical shape sensing device using optical shape sensing signals received via an interface from an optical shape sensing device, the optical shape sensing device having a shape that follows the shape of the interventional medical device during the interventional medical procedure; and identifying the interventional medical device in a first coordinate space of a first imaging system based on the identified shape of the optical shape sensing device, the first imaging system imaging the interventional medical device in a first imaging mode during the interventional medical procedure. The method further includes registering the interventional medical device to the first coordinate space, identifying the interventional medical device in a second coordinate space of a second imaging system, the second imaging system imaging the interventional medical device in a second imaging mode during the interventional medical procedure, and registering the first coordinate space of the first imaging system to the second coordinate space of the second imaging system. The method further includes segmenting the interventional medical device in the second coordinate space to obtain a segmented representation of the interventional medical device in the second coordinate space, registering the interventional medical device to the second coordinate space using the segmented representation of the interventional medical device, and re-registering the interventional medical device to the first coordinate space based on the registration of the interventional medical device to the second coordinate space. Attached Figure Description

[0009] The exemplary embodiments can be best understood from the following detailed description when read in conjunction with the accompanying drawings. It should be emphasized that the various features are not necessarily drawn to scale. In fact, dimensions may be arbitrarily increased or decreased for clarity of discussion. Where applicable and practical, the same reference numerals refer to the same elements.

[0010] Figure 1 A system for tracking interventional medical devices according to a representative embodiment is shown;

[0011] Figure 2A The registration of an interventional medical device with an X-ray imaging system in interventional medical device tracking is illustrated according to a representative embodiment.

[0012] Figure 2B The registration of an ultrasound system with an X-ray imaging system in the tracking of interventional medical devices is illustrated according to a representative embodiment.

[0013] Figure 2C The registration of an interventional medical device with an ultrasound system in interventional medical device tracking is illustrated according to a representative embodiment;

[0014] Figure 3 The registration of an interventional medical device with an ultrasound system and an X-ray system in interventional medical device tracking is illustrated according to another representative embodiment;

[0015] Figure 4 The registration of an interventional medical device with an ultrasound system in interventional medical device tracking is illustrated according to a representative embodiment;

[0016] Figure 5 A method for tracking interventional medical devices according to a representative embodiment is shown;

[0017] Figure 6 A method for tracking interventional medical devices according to a representative embodiment is shown;

[0018] Figure 7 A method for tracking interventional medical devices according to a representative embodiment is shown;

[0019] Figure 8 A method for tracking interventional medical devices according to a representative embodiment is shown;

[0020] Figure 9 A computer system according to another representative embodiment is shown, on which a method for tracking interventional medical devices is implemented. Detailed Implementation

[0021] In the following detailed description, representative embodiments with specific details disclosed are set forth for purposes of explanation and not limitation, in order to provide a thorough understanding of embodiments according to this teaching. Descriptions of known systems, apparatuses, materials, methods of operation, and methods of manufacture may be omitted to avoid obscuring the description of the representative embodiments. Nevertheless, systems, apparatuses, materials, and methods within the scope of this teaching are available to those skilled in the art and may be used according to representative embodiments. It should be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. The defined terminology is supplementary to the technical and scientific meaning of the defined terminology as commonly understood and accepted in the art field of this teaching.

[0022] It should be understood that although the terms first, second, third, etc., may be used herein to describe various elements or components, these elements or components should not be limited by these terms. These terms are only used to distinguish one element or component from another. Therefore, without departing from the teachings of the inventive concept, the first element or component discussed below may be referred to as the second element or component.

[0023] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the specification and appended claims, the singular forms of the terms “a,” “an,” and “the” are intended to include both singular and plural forms unless the context clearly specifies otherwise. Furthermore, when the terms “comprising” and / or “including” and / or similar terms are used herein, they specify the presence of stated features, elements, and / or components, but do not exclude the presence or addition of one or more other features, elements, components, and / or combinations thereof. As used herein, the term “and / or” includes any and all combinations of one or more associated listed items.

[0024] Unless otherwise stated, when a factor or component is referred to as being “connected to,” “coupled to,” or “proximity to” another element or component, it should be understood that the element or component can be directly connected to or coupled to the other element or component, or that intervening elements or components may be present. That is, these and similar terms cover situations where one or more intermediate elements or components may be used to connect two elements or components. However, when an element or component is said to be “directly connected” to another element or component, this only covers situations where the two elements or components are connected to each other without any intermediate or intervening elements or components.

[0025] Therefore, this disclosure is intended to provide one or more advantages, particularly noted below, through its various aspects, embodiments, and / or specific features or sub-components. Exemplary embodiments with specific details disclosed are set forth for purposes of explanation and not limitation in order to provide a full understanding of embodiments based on this teaching. However, other embodiments consistent with this disclosure but departing from the specific details disclosed herein remain within the scope of the appended claims. Furthermore, descriptions of well-known apparatuses and methods may be omitted so as not to obscure the description of the exemplary embodiments. Such methods and apparatuses are within the scope of this disclosure.

[0026] As described in this article, interventional medical device tracking can utilize the three-dimensional segmentation of the interventional medical device within the ultrasound volume and the registration of the ultrasound imaging system with the X-ray imaging system. Interventional medical device tracking can achieve precise maintenance of the registration between the OSS device and both the ultrasound and X-ray imaging systems throughout the entire duration of the interventional medical procedure without repeating X-ray imaging.

[0027] As described herein, using the shape of the OSS device can enhance the process of identifying the OSS device in images, for example, from an ultrasound imaging system, and this in turn can enhance the registration process described below.

[0028] Figure 1 A system for tracking interventional medical devices, according to a representative embodiment, is shown.

[0029] Figure 1 A tracking system 100 is shown. The tracking system 100 includes a console having a controller 190, an interface 193, and a touch panel 196. The controller 190 includes at least a memory 191 storing instructions and a processor 192 executing the instructions. The controller 190 controls one or more aspects of the methods described herein. The processor 192 retrieves or otherwise receives instructions from the memory 191 via a bus (not shown). When executed by the processor 192, the instructions cause the controller 190 to perform actions relative to... Figure 5 , Figure 6 , Figure 7 and Figure 8 One or more aspects of the method shown and described herein. Interface 193 provides an interface between a console including controller 190 and optical shape sensing device 102. Interface 193 represents an interface between elements and components of tracking system 100. Touch panel 196 includes buttons, keys, and any other touch surfaces capable of inputting commands from a user to tracking system 100.

[0030] The tracking system 100 also includes a monitor 195, an X-ray imaging system 120, and an ultrasound imaging system 110. The monitor 195 can be used to display images from the X-ray imaging system 120 and the ultrasound imaging system 110. As a non-limiting example, the X-ray imaging system 120 can perform fluoroscopic imaging during interventional medical procedures. Also as a non-limiting example, the ultrasound imaging system 110 can perform transesophageal echocardiography (TEE) or other forms of ultrasound imaging. The X-ray imaging system 120 performs imaging in a three-dimensional coordinate space, which can be centered on the isocenter of the C-arm of the X-ray imaging system 120. The ultrasound imaging system 110 performs imaging in another three-dimensional coordinate space. The three-dimensional coordinate space of the ultrasound imaging system 110 and other three-dimensional coordinate spaces can be registered to the three-dimensional coordinate space of the X-ray imaging system 120, such that the isocenter of the C-arm of the X-ray imaging system 120 becomes the origin of all such registered coordinate spaces.

[0031] The tracking system 100 also includes an interventional medical device 101 integrated with an optical shape sensing device 102. The optical shape sensing device 102 may be flexible and may have a shape that flexibly follows the shape of the interventional medical device 101 during interventional medical procedures. Throughout this description, references to the interventional medical device 101 also refer to the optical shape sensing device 102, provided that the optical shape sensing device 102 is integrated with the interventional medical device 101. Alternatively, references to the optical shape sensing device 102 may be specific to the optical shape sensing device 102, independent of the interventional medical device 101, provided that the optical shape sensing device 102 is independently interfaced with the controller 190 via an interface 193 to provide optical shape sensing signals generated by the optical shape sensing device 102.

[0032] Figure 1 The components and parts of the tracking system 100 can be provided together or distributed. For example, the controller 190, monitor 195, and touchpad 196 can be provided as an integrated computer system separate from the X-ray imaging system 120, ultrasound imaging system 110, and interventional medical device 101. The X-ray imaging system 120, ultrasound imaging system 110, and interventional medical device 101 can be provided separately from each other and can be integrated together via an integrated computer system including the controller 190, monitor 195, and touchpad 196.

[0033] In addition to interface 193, controller 190 may also include one or more input interfaces. Other input interfaces (not shown) of interface 193 and controller 190 may include cables, adapters, ports, disk drives, antennas for wireless communication, and other forms of interfaces specifically designed for connecting elements and components of tracking system 100. Input interfaces may also connect user interfaces such as mice, keyboards, microphones, cameras, touchscreen displays, or other elements or components to controller 190. The interfaces of tracking system 100 can connect controller 190 to monitor 195, X-ray imaging system 120, and ultrasound imaging system 110. For example, controller 190 may be connected to monitor 195 via a local wired interface such as an Ethernet cable or via a local wireless interface such as a Wi-Fi connection.

[0034] Monitor 195 may be a computer monitor, a display on a mobile device, a television, an electronic whiteboard, or other screen configured to display electronic images. Monitor 195 may also include one or more input interfaces, such as those mentioned above, which can connect other elements or components to monitor 195. Monitor 195 may also include a touchscreen capable of direct input via touch.

[0035] In one set of embodiments, the tracking system 100 tracks the interventional medical device 101 during an interventional medical procedure. The X-ray imaging system 120 may be a first imaging system imaging the interventional medical device 101 during the interventional medical procedure, and the ultrasound imaging system 110 may be a second imaging system imaging the interventional medical device during the interventional medical procedure. When executed by the processor 192, instructions stored in the memory 191 cause the tracking system 100 to track the position of the interventional medical device 101 during the interventional medical procedure. The process of tracking the interventional medical device 101 may include identifying the shape of the optical shape sensing device 102 using optical shape sensing signals received from the optical shape sensing device 102 via the interface 193. The process in this set of embodiments may also include identifying the interventional medical device 101 in a first coordinate space of the X-ray imaging system 120 based on the identified shape of the optical shape sensing device 102. The interventional medical device 101 is then registered to the first coordinate space of the X-ray imaging system 120. The process may also include identifying the interventional medical device 101 in a second coordinate space of the ultrasound imaging system 110. The first coordinate space of the X-ray imaging system 120 is registered to the second coordinate space of the ultrasound imaging system 110. In this set of embodiments, the process of tracking the position of the interventional medical device 101 may further include segmenting the interventional medical device 101 in the second coordinate space of the ultrasound imaging system 110 to obtain a segmented representation of the interventional medical device 101 in the second coordinate space. The interventional medical device 101 is then registered to the second coordinate space of the ultrasound imaging system 110 using the segmented representation. Subsequently, based on the registration of the interventional medical device 101 to the second coordinate space using the segmented representation, the interventional medical device 101 is re-registered to the first coordinate space of the X-ray imaging system 120. The re-registration of the interventional medical device 101 to the first coordinate space of the X-ray imaging system 120 is performed without requiring additional X-ray imaging of the patient. The processes performed in this set of operations can be performed on demand, periodically, or once movement of the interventional medical device 101 is detected to exceed a threshold.

[0036] The controller 190 can directly perform some of the operations described herein, and can indirectly perform other operations described herein. For example, the controller 190 can directly control the display of the monitor 195, and indirectly control imaging via the X-ray imaging system 120 and / or imaging via the ultrasound imaging system 110. Therefore, when the processor 192 executes instructions from the memory 191, the process implemented by the tracking system 100 may include steps not directly executed by the controller 190.

[0037] In another set of embodiments using the tracking system 100, registration can be performed using a predetermined shape of the interventional medical device 101. For example, the predetermined shape of the interventional medical device 101 can be stored as a template in memory 191 and can be retrieved from memory 191 to search for the ultrasound space for the interventional medical device 101. The shape of the interventional medical device 101 can also be dynamically obtained from the optical shape sensing device 102. In this set of embodiments, the ultrasound imaging system 110 can be a first imaging system, and the X-ray imaging system 120 can be a second imaging system. The interventional medical device 101 can be registered to the ultrasound space (first coordinate space) based on the shape of the interventional medical device 101 identified using optical shape sensing signals, and based on the shape of the interventional medical device 101 identified in the ultrasound space (first coordinate space). The process for tracking interventional medical device 101 may include identifying the shape of optical shape sensing device 102 using optical shape sensing signals received from optical shape sensing device 102 via interface 193, and identifying the shape of interventional medical device 101 in a first coordinate space of an ultrasound system (first imaging system) that images interventional medical device 101 in a first imaging mode during the interventional medical procedure. Based on the shape of interventional medical device 101 identified using the optical shape sensing signals, and based on the shape of interventional medical device 101 identified in the ultrasound space (first coordinate space), interventional medical device 101 is registered to the ultrasound space (first coordinate space). In this set of embodiments, it is not necessary to pre-register the X-ray imaging system to interventional medical device 101 and ultrasound imaging system 110 so that interventional medical device 101 can be registered to ultrasound imaging system 110 using the known shape of interventional medical device 101.

[0038] In progress Figure 2A Before describing the preceding sections, we will now explain the concepts of registration and segmentation more comprehensively. Registration involves aligning different 3D coordinate systems. Figure 1 In this system, the X-ray imaging system 120, the ultrasound imaging system 110, and the optical shape sensing device 102 can each have their own three-dimensional coordinate system. A common three-dimensional coordinate system is provided by aligning the different three-dimensional coordinate systems, for example, by sharing a common origin and a set of axes. Registration may include first adjusting the origin of one coordinate system to the origin of another, and then aligning the axes of one coordinate system with the axes of the other. Registration typically involves calculating and applying a transformation matrix based on observations of common three-dimensional elements in the two coordinate systems.

[0039] Segmentation produces a surface representation of structures such as anatomical features and interventional medical device 101. The segmentation representation includes, for example, a set of points in three-dimensional (3-D) coordinates on the surface of the structure, and triangular plane segmentation defined by connecting adjacent sets of three points, such that the entire structure is covered by a mesh of non-intersecting triangular planes. A three-dimensional model of interventional medical device 101 is obtained through segmentation. Segmentation can also involve performing segmentation on anatomical structures and / or other structures present in a three-dimensional ultrasound volume.

[0040] Figure 2A The registration of an interventional medical device with an X-ray imaging system in interventional medical device tracking is illustrated according to a representative embodiment.

[0041] exist Figure 2A In this process, optical fibers are integrated into the interventional medical device 201. Figure 2A An example of optical fiber is Figure 1 The optical shape sensing device 102 in the middle. The optical fiber provides the position and orientation of the interventional medical device 201. Figure 2A An example of interventional medical device 201 is a catheter with a guidewire. Figure 2A In this context, the optical fiber can be integrated into the interventional medical device 201 in the right branch of the blood vessel (e.g., integrated into the guidewire of the catheter). The interventional medical device 201 with integrated optical fiber is overlaid on an X-ray (fluorescence) image of the blood vessel phantom generated by the X-ray imaging system 120.

[0042] exist Figure 2A In this system, the shape of the optical fiber can be identified using optical shape sensing signals received from the optical fiber via an interface such as interface 193. The X-ray imaging system 120 can be a first imaging system that generates X-ray images in a first coordinate space specific to the X-ray imaging system 120. Based on the X-ray images, the interventional medical device 201 is registered to the X-ray imaging system 120 by assigning the position of the interventional medical device 201 from the optical shape sensing signals to the coordinates of the interventional medical device in the first coordinate system. The interventional medical device 201 can be registered to the X-ray coordinate space using two X-ray projection images offset by 30 degrees or more. The operator identifies the tip of the interventional medical device 201 in each X-ray image and automatically detects the visible portion of the interventional medical device 201. The transformation from the interventional medical device 201 and the X-ray coordinate space is determined from the reconstruction of the interventional medical device based on the two X-ray projections and the optical shape sensing signals from the optical fiber. Based on the shape of the optical shape sensing device 102 identified using optical shape sensing signals from optical fibers, the interventional medical device 201 is identified in the first coordinate space of the X-ray imaging system 120 during the interventional medical procedure.

[0043] Figure 2BThe registration of an ultrasound system with an X-ray imaging system 120 in interventional medical device tracking is shown according to a representative embodiment.

[0044] exist Figure 2B In this process, registration between the ultrasound imaging system 110 and the X-ray imaging system 120 is achieved through an image fusion platform. An example of such an image fusion platform is EchoNavigator. The registration algorithm provided by EchoNavigator is based on fluorescence fluoroscopy images captured from the X-ray imaging system 120. These fluorescence fluoroscopy images contain the probe of the ultrasound imaging system 110. As an example, the ultrasound imaging system 110 could be a transesophageal echocardiography (TEE) system. Based on the probe's orientation in the X-ray image, a transformation involving the ultrasound space and X-ray space (T0) can be calculated. UX ).

[0045] Figure 2C The registration of an interventional medical device with an ultrasound system in interventional medical device tracking is illustrated according to a representative embodiment.

[0046] like Figure 2C As shown, it can integrate multiple registrations between different coordinate systems, thereby aligning three or more coordinate systems. Figure 2C In this system, the optical shape sensing coordinate system (OSS space) can be registered to the X-ray imaging system coordinate system (X-ray space) using a program executed by the controller 190. The X-ray imaging system coordinate system (X-ray space) can be registered to the local environment (patient space) including the X-ray imaging system 120 using a program such as EchoNavigator executed by the controller 190. Additionally, the ultrasound imaging system coordinate system (US space) can be registered to the X-ray imaging system coordinate system (X-ray space) using a program such as EchoNavigator executed by the controller 190.

[0047] Once both the interventional medical device 101 and the ultrasound imaging system 110 are registered in X-ray space, it is possible to transmit the signal via X-ray imaging. Figure 2C The transformations outlined in the figure are represented as interventional medical device 101 in the ultrasound imaging system coordinate system (US space).

[0048] Figure 3 The registration of an interventional medical device with an ultrasound system and an X-ray system is shown in interventional medical device tracking according to another representative embodiment.

[0049] exist Figure 3 In the image, guidewire 301 is shown in the ultrasound space on the left and the X-ray space on the right. Because the interventional medical device 101 is registered to both the ultrasound space and the X-ray space, and the ultrasound space is registered to the X-ray space, therefore... Figure 3The images shown reflect the same coordinate system, even if the perspectives between the two images are different.

[0050] Figure 4 The registration of an interventional medical device with an ultrasound system in interventional medical device tracking is illustrated according to a representative embodiment.

[0051] exist Figure 4 In this context, the optical shape sensing reconstruction of the interventional medical device 101 using the optical shape sensing device 102 is shown as OSS reconstruction 402. The image-based segmentation of the optical shape sensing device 102 in ultrasound space is shown as segmentation representation 401. As shown, in the registration process described herein, OSS reconstruction 402 can be registered to segmentation representation 401 in ultrasound space. Transform T OU The current position of the OSS reconstruction 402 is associated with the segmented representation 401 of the position of the interventional medical device 101 in the ultrasound space.

[0052] In the first set of embodiments described herein, registration can be accomplished by identifying the location of the interventional medical device 101 in a medical image, for example, by user specification, tracking sensors integrated into the tip of the interventional medical device 101, or otherwise. Three-dimensional segmentation of the interventional medical device 101 in ultrasound images can be used to update the registration in the first set of embodiments. Three-dimensional segmentation of tubular interventional medical devices in ultrasound can be achieved using image processing techniques combined with sensor tracking techniques. Examples of three-dimensional segmentation of interventional medical devices are described in U.S. Provisional Patent Application No. 62 / 855013, filed May 31, 2019, with the U.S. Patent and Trademark Office, the disclosure of which is incorporated herein by reference in its entirety. Examples of tubular interventional medical devices that are easily segmented in three dimensions include guidewires and catheters. Alternative mechanisms for identifying interventional medical devices in ultrasound images include initialization using deep learning of artificial intelligence based on previous identification instances of interventional medical devices in ultrasound images, through user clicking on the location of the tip of the interventional medical device in the ultrasound image.

[0053] The updated registration addresses errors known to occur when optical shape sensing is registered to X-ray and / or ultrasound space. That is, even though optical shape sensing provides a highly accurate reconstruction of the local shape of interventional medical device 101, it can easily introduce errors in registration misalignment due to the accumulation of errors along the length of interventional medical device 101. For example, even immediately after registration of interventional medical device 101 with X-ray space is completed, the accuracy may be very good; however, if the proximal end of interventional medical device 101 moves by several centimeters, significant errors can accumulate in the registration. Using the teachings of the first set of embodiments provided herein, errors can be corrected by re-registration without additional exposure to X-ray projection and therefore without increasing X-ray dose exposure to patients and clinicians. Furthermore, the registration of interventional medical device 101 to X-ray imaging system 120 can be continuously updated as needed by the clinician (e.g., when the clinician notices an error) or automatically (e.g., when tracking system 100 detects misalignment exceeding a predetermined threshold). By utilizing the segmentation of the interventional medical device 101 in the ultrasound space, and the registration as described herein, the registration of the interventional medical device 101 can be continuously and accurately updated throughout the duration of the procedure.

[0054] The interventional medical device 101, which has been coarsely registered to X-ray space and three-dimensional ultrasound space, can maintain automatic fine registration based on the segmented shape of the interventional medical device 101 in three-dimensional ultrasound. The shape of the interventional medical device 101 in three-dimensional ultrasound is determined by image processing or deep learning techniques. Then, a rigid point-to-point transformation is calculated based on the corresponding portion of the interventional medical device 101 to the three-dimensional segmentation of the interventional medical device 101 in the ultrasound coordinate system. By automatically segmenting the interventional medical device 101 in the image and aligning and reconstructing it according to the optical shape sensing device 102, accurate registration of the interventional medical device 101 can be maintained throughout the procedure.

[0055] In the second set of embodiments, the known shape of the interventional medical device 101 can be used to achieve and update the registration of the interventional medical device 101, and this can involve a simplified workflow compared to the first set of embodiments. For example, when the shape of the interventional medical device 101 is known, for example, from a template and / or from an optical shape sensing device 102, the shape can be identified in ultrasound coordinate space, and registration between optical shape sensing and ultrasound space can be achieved without first registering the interventional medical device 101 to the X-ray imaging system 120. The template of the shape of the interventional medical device 101 can be obtained from a template library stored in a memory such as memory 191. The template may include a template of a portion of the shape of the interventional medical device 101, such as a template of the shape of the distal end of the interventional medical device 101. When the template is a template of a portion of the shape, the remaining portion of the shape of the interventional medical device 101 can be identified based on image analysis software that searches for the remaining portion of the shape of the interventional medical device 101 in regions close to the portion of the shape identified in the ultrasound image from the template of that portion of the shape.

[0056] Furthermore, in the second set of embodiments, registration between the ultrasound coordinate space and the X-ray coordinate space can be performed without requiring X-ray images from the probe of the ultrasound imaging system 110. For example, the common shape of the interventional medical device 101 in both coordinate systems can be used as a mechanism for registering the two coordinate systems. When the interventional medical device 101 has been registered to the X-ray space, and a template of the shape of the interventional medical device 101 can be used to register the interventional medical device 101 to the ultrasound space, the ultrasound coordinate system can be registered to the X-ray coordinate system by calculating the transformation from the segmentation of the interventional medical device 101 in the ultrasound space to the corresponding interventional medical device 101 in the X-ray space.

[0057] In the second set of embodiments, this portion of the shape can be used as a constraint during the initial search for the shape of the interventional medical device 101 in ultrasound images. Artificial intelligence can be applied to analyze the ultrasound images in ultrasound coordinate space. The search may initially be constrained by the tip of the interventional medical device 101, and once the tip of the interventional medical device 101 is identified in the search, artificial intelligence can be applied to find the remaining portion of the shape of the interventional medical device 101 based on features and parameters identified from previous instances of the interventional medical device 101 in previous searches of ultrasound images.

[0058] In the first and second sets of embodiments, metrics can be generated to show the correlation between the identification of the interventional medical device 101 in different coordinate spaces. For example, the metric can be generated based on the correlation between the segmented representation of the interventional medical device 101 in the existing location in ultrasound space and the newly identified location of the interventional medical device 101 in ultrasound coordinates. The correlation can be an estimate of the confidence level regarding the accuracy of the identification and can be based on, for example, the amount of difference between the segmented representation and the proposed newly identified location of the interventional medical device 101 in ultrasound coordinates.

[0059] Figure 5 A method for tracking interventional medical devices according to a representative embodiment is shown.

[0060] exist Figure 5 In this process, the method begins at step S510 by identifying the shape of the optical shape sensing device. The optical shape sensing device can be... Figure 1 The optical shape sensing device 102 in the embodiments may include an optical fiber. The optical shape sensing technology described above can be used to identify the optical shape sensing device.

[0061] At S520, Figure 5 The method includes identifying the interventional medical device in a first coordinate space. The first coordinate space may be the coordinate space of a first imaging system operating in a first imaging mode, such as the coordinate space of an X-ray imaging system 120 operating in an X-ray imaging mode. At S520, identification may be performed by a user specifying the interventional medical device 101 in the X-ray image, or by image analysis software identifying the interventional medical device 101 in the X-ray image.

[0062] At S530, Figure 5 The method includes registering the interventional medical device 101 to a first coordinate space. Registration at S530 may be registration of the interventional medical device 101 to the X-ray space of the X-ray imaging system 120. Registration at S530 may be based on the shape of the interventional medical device 101 identified at S510 according to the shape of the following optical shape sensing device 102. Registration at S520 may also be based on the shape of the interventional medical device 101 identified from the X-ray image at S520.

[0063] At S540, Figure 5 The method includes identifying the interventional medical device 101 in a second coordinate space. The second coordinate space can be the coordinate space of a second imaging system operating in a second imaging mode, such as the coordinate space of an ultrasound imaging system 110 operating in an ultrasound imaging mode. Identification at S540 can be achieved by the user specifying the interventional medical device 101 in the ultrasound image, or by image analysis software identifying the interventional medical device 101 in the ultrasound image. Although in Figure 5 As not shown, S540 (described below) can be performed between S550 and S560 to register the first coordinate space to the second coordinate space before the fine-tuning interventional medical device 101 is registered in the second coordinate space (i.e., in the ultrasound space).

[0064] In one embodiment, the user can identify the tip of the interventional medical device 101 in an ultrasound image, and the image analysis software can constrain the search for the rest of the interventional medical device 101 to the area around the specified tip.

[0065] In another embodiment, the sensor on the tip of the interventional medical device 101 may be a passive ultrasound sensor that responds to emissions from the ultrasound imaging system 110. Sensor-based tracking of the interventional medical device 101 is described in U.S. Provisional Patent Application No. 62 / 855013, filed May 31, 2019, with the U.S. Patent and Trademark Office, the disclosure of which is incorporated herein by reference in its entirety.

[0066] In embodiments using sensors on the tip of the interventional medical device 101, image analysis software can constrain the search for the remainder of the interventional medical device 101 to the area around the tip identified based on signals from the sensors. The constraint can be based on the user's identification of the tip of the interventional medical device 101, on the identification of signals from a passive ultrasound sensor, or on identification from image analysis software trained by artificial intelligence to recognize the tip of the interventional medical device.

[0067] In another embodiment, the X-ray can be transmitted via a T-band from the interventional medical device 101 to the X-ray space. OX Transformation, and via T from ultrasound space to X-ray space. UX The interventional medical device 101 is coarsely registered to the ultrasound space using a transformation. Then, throughout the acquisition process, the tip of the interventional medical device 101 is used as a coarse estimate of the search space for the constrained image processing algorithm, and device segmentation based on the ultrasound image is continuously calculated, as described below at S560. The transformation T from the interventional medical device 101 to the ultrasound space can be calculated on each ultrasound frame. OU And throughout the process, the registration of the interventional medical device 101 to the ultrasound space is continuously or at fixed intervals.

[0068] At S550, the first coordinate space is registered to the second coordinate space. Registration at S550 can be performed by imaging the ultrasound probe in the ultrasound imaging system 110 using the X-ray imaging system 120. As described above, in some embodiments, S550 can be performed prior to S540.

[0069] At S560, the interventional medical device is segmented in the second coordinate space to generate a segmented representation of the interventional medical device. In an embodiment, the segmentation at S560 is initialized by the user identifying the interventional medical device 101 in the ultrasound image at S540. The image processing algorithm searches for the interventional medical device 101 in the image within the region identified by the user. The rigid transformation T from the interventional medical device 101 to the ultrasound space is calculated. OU This causes the distal portion of the interventional medical device 101, corresponding to the segmentation length of the ultrasound device, to be rotated / translated to most closely match the segmentation representation in the ultrasound. Then, a rigid transformation T is performed. OU The entire length is applied to the reconstruction from the optical shape sensing device 102.

[0070] At S565, a segmented representation of the interventional medical device 101 is presented, for example in Figure 1 On monitor 195. The segmented representation of interventional medical device 101 can be superimposed on ultrasound image, and subsequently on X-ray image.

[0071] At S570, the interventional medical device is registered to the second coordinate space of the ultrasound imaging system 110. The registration at S570 can be an initial registration of the interventional medical device 101 to the ultrasound space, or it can be a re-registration of the interventional medical device 101 to correct for outdated earlier registrations.

[0072] At S580, the interventional medical device 101 is re-registered to the first coordinate space of the X-ray imaging system 120. The re-registration at S580 can correct outdated earlier registrations and does not require additional imaging from the X-ray imaging system 120.

[0073] In this embodiment, the interventional medical device 101 is segmented within the ultrasound volume based on prior identification of the interventional medical device 101 in the ultrasound image, using artificial intelligence. In this embodiment, user identification or sensor initialization is not necessarily required to constrain the search space. The transformation T from the interventional medical device 101 to the ultrasound space can be calculated on each ultrasound frame. OU And throughout the procedure, the registration from the interventional medical device 101 to the ultrasound space is continuously updated.

[0074] although Figure 5As not shown, the first coordinate space can be re-registered to the second coordinate space by calculating the transformation from the segmented representation of the interventional medical device 101 in the first coordinate space to the shape of the interventional medical device 101 as identified by optical shape sensing in the second coordinate system. According to the second set of embodiments described herein, re-registration can be performed without requiring X-ray images of the probe of the ultrasound imaging system 110. When the interventional medical device 101 has already been registered to the X-ray space and the interventional medical device has been registered to any ultrasound space based on a shape template, the ultrasound space can be registered to the X-ray space by calculating the transformation from the segmented representation of the interventional medical device 101 in the ultrasound space to the corresponding portion of the interventional medical device in the X-ray space.

[0075] Figure 6 A method for tracking interventional medical devices according to a representative embodiment is shown.

[0076] exist Figure 6 In this method, the selection of the interventional medical device 101 is detected in the second coordinate space of the ultrasound imaging system 110. The selection may be a user's choice of the tip of the interventional medical device 101.

[0077] At S640, the interventional medical device 101 is identified in the second coordinate space. The identification at S640 is based on the selection at S635 and may involve using image analysis software to search the area around which the user has selected the remaining portion of the interventional medical device 101.

[0078] At S645, the transformation T from the current position of the interventional medical device 101 to the segmented representation is calculated. OU Using the new T OU The transformation can update the previous position of the interventional medical device 101 in the ultrasound space to a new position represented by the segmentation.

[0079] At S648, based on the segmented representation of the interventional medical device 101 in the second coordinate space of the ultrasound imaging system 110, the second coordinate space of the ultrasound imaging system 110 is registered to the first coordinate space of the X-ray imaging system. Through this registration, the previous position of the interventional medical device 101 in the X-ray space is updated to take into account any errors, such as those caused by movement of the interventional medical device 101.

[0080] Figure 6 The embodiments may be for Figure 5 The embodiments are supplemented, and supplements are included regarding Figure 5 The function described.

[0081] Above Figure 6In the description of the embodiments, the identification of the interventional medical device 101 in the ultrasound space is based on user specification of the tip of the interventional medical device 101. In another embodiment, it is possible to search for the known shape of the interventional medical device 101 in the ultrasound space without needing to know the shape of the distal tip of the interventional medical device. The search for the known shape of the interventional medical device 101 can be based on information from an optical shape sensing signal derived from an optical shape sensing device 102, and eliminates the possibility of errors caused by transformation T. OU The need to roughly register the interventional medical device 101 to the ultrasound space.

[0082] Figure 7 A method for tracking interventional medical devices according to a representative embodiment is shown.

[0083] exist Figure 7 In step S736, the interventional medical device 101 is identified in the second coordinate space of the ultrasound imaging system 110. The identification at S736 can be based on the user-selected position of the tip of the interventional medical device 101, or by a signal from a passive ultrasound sensor on the tip of the interventional medical device 101.

[0084] At S760, the interventional medical device 101 is segmented in the ultrasound image to generate a segmented representation of the interventional medical device.

[0085] At S770, the interventional medical device 101 is registered to the second coordinate space of the ultrasound imaging system 110. Registration at S770 can be based at least on transform T. OU .

[0086] At S780, the interventional medical device is registered to the first coordinate space of the X-ray imaging system 120. The transformation at S780 can be based on transformation T. OX T UX and T OU All three of them.

[0087] Figure 7 The embodiments may be for Figure 5 The embodiments are supplemented, and supplements are included regarding Figure 5 The function described.

[0088] In the above Figure 7In the description of the embodiments, the identification of the interventional medical device 101 in the ultrasound space is based on user or sensor identification of the tip of the interventional medical device 101. In another embodiment, the known shape of the interventional medical device 101 can be retrieved from a template or from information from optical shape sensing signals from the optical shape sensing device 102. Registration based on a template or knowledge of the shape of the interventional medical device 101 can be used to define or maintain registration between the ultrasound space and the X-ray space, especially if the ultrasound probe of the ultrasound imaging system 110 is at an angle that is difficult to detect in the X-ray imaging system 120. Assume the transformation T from the interventional medical device 101 to the ultrasound space... OU It is known, and the transformation T from interventional medical device 101 to X-ray space is... OX If it is known, then the transformation T from ultrasound space to X-ray space UX It can be loosely determined as (T) UX =inv(T OU )*T OX As in other embodiments described herein, embodiments that maintain registration by repeatedly updating the transformation TUX may not require additional exposure to X-rays.

[0089] Figure 8 A method for tracking interventional medical devices according to a representative embodiment is shown.

[0090] exist Figure 8 In this process, the method begins at S836 by identifying the interventional medical device 101 in a second coordinate space. Identification can be based on user-specified information or on signals from sensors on the tip of the interventional medical device 101. Identification at S836 can also be performed on demand, periodically and continuously during the interventional medical procedure, or based on the detection of movement of the interventional medical device 101 compared to a previous registration.

[0091] At S837, it is determined whether the deviation of the position of the interventional medical device from the existing segmentation representation exceeds a threshold. The determination at S837 can be based on the detection of movement of the interventional medical device 101 compared with the previous registration. If no deviation exceeds the threshold (S837 = No), the method returns to S836; otherwise, it proceeds to S860.

[0092] At S860, the interventional medical device 101 is segmented in the ultrasound space.

[0093] At S870, based on the segmentation at S860, the interventional medical device 101 is registered to the second coordinate space.

[0094] At S880, the second coordinate system is re-registered to the first coordinate system. This re-registration at S880 is performed without requiring further exposure to X-rays from the X-ray imaging system 120.

[0095] After S880, the program returns to S836. Therefore, Figure 8 The procedure is recursive and may involve repeatedly checking whether the position of the interventional medical device 101 deviates from the last existing segmentation representation by more than a threshold, and correcting the deviation by updating the registration of the interventional medical device to the X-ray space and the ultrasound space.

[0096] Figure 8 The embodiments may be for Figure 5 The embodiments are supplemented, and supplements are included regarding Figure 5 The function described.

[0097] In the above Figure 8 In the description of the embodiments, the identification of the interventional medical device 101 in the ultrasound space is still based on cutting-edge knowledge of the interventional medical device 101. In another embodiment, an irregularly shaped treatment device equipped with optical shape sensing can be delivered via a delivery sheath or catheter that implements OSS. Known irregularly shaped treatment devices can be detected in the ultrasound space via image-based segmentation or via manual analysis applied to previous identification instances of OSS delivery sheaths or catheters. Segmentation of the irregularly shaped treatment device can be used to locate the distal end of the delivery device implementing OSS in ultrasound to repeatedly update the registration between the interventional medical device 101 and the ultrasound space.

[0098] In an additional embodiment, the registration method described above can be automatically triggered. For example, the interventional medical device 101 can be continuously segmented in ultrasound space during background processing. The existing registration of the interventional medical device 101 can be updated to most closely match the interventional medical device in ultrasound space on each frame, on each nth frame, or only when a metric describing the correlation and / or offset between the location of the interventional medical device 101 and the location of the segmented representation of the interventional medical device in ultrasound space exceeds a predetermined threshold.

[0099] Furthermore, the registration method described above can be triggered on demand. The user interface may include a metric describing the correlation and / or offset between the position of the interventional medical device 101 and its position in ultrasound space. Then, when the offset metric exceeds the expected error limit, or at any time when the user wishes to update the current registration based on visual examination, the user can select the "Update Registration" softkey.

[0100] Furthermore, the user interface can provide metrics describing the success of registration after registration has been performed. These metrics may include information about the correlation between the shape of the interventional medical device 101 in re-registration and the shape of the interventional medical device 101 in its segmented representation in ultrasound space. Optionally, the metrics may include a confidence level that the correct shape of the interventional medical device 101 has been detected.

[0101] Figure 9 A computer system according to some representative embodiments is shown, on which a method for tracking interventional medical devices is implemented.

[0102] Figure 9 The computer system 900 illustrates a complete set of components for communication equipment or computer equipment. However, the "controller" described herein can be used in fewer than [number missing]. Figure 9 The computer system 900 may be implemented using a complete set of components, such as a combination of memory and processor. The computer system 900 may include some or all of the components of one or more of the components described herein for tracking interventional medical devices, although any such device may not necessarily include the one or more components described for the computer system 900, and may include other components not described.

[0103] refer to Figure 9 The computer system 900 includes a set of software instructions that are executable to cause the computer system 900 to perform any of the methods or computer-based functions disclosed herein. The computer system 900 may operate as a standalone device or may be connected to other computer systems or peripheral devices, for example, using a network 901. In an embodiment, the computer system 900 performs logic processing based on digital signals received via an analog-to-digital converter.

[0104] In network deployments, computer system 900 operates as a server, or as a client computer in a server-client network environment, or as a peer-to-peer (or distributed) network system. Computer system 900 can also be implemented as various devices or incorporated into various devices, such as... Figure 1The computer system 900 may be a controller 190, a fixed computer, a mobile computer, a personal computer (PC), a laptop computer, a tablet computer, or any other machine capable of executing a set of software instructions (sequentially or otherwise) specifying the actions to be taken by the machine. The computer system 900 may be incorporated as a device or incorporated in a device, which in turn is incorporated in an integrated system including additional devices. In embodiments, the computer system 900 may be implemented using electronic devices that provide voice, video, or data communications. Furthermore, although the computer system 900 is shown in the singular, the term "system" should also be considered as including any collection of systems or subsystems that individually or jointly execute one or more sets of software instructions to perform one or more computer functions.

[0105] like Figure 9 As shown, the computer system 900 includes a processor 910. The processor 910 can be considered as... Figure 1 The processor 910 is a representative example of the controller 190 and executes instructions to implement some or all aspects of the methods and processes described herein. The processor 910 is tangible and non-transitory. As used herein, the term "non-transitory" should be interpreted as not being a permanent characteristic of a state, but rather a characteristic of a state that will last for a period of time. The term "non-transitory" specifically denies transient characteristics, such as carrier waves or signals, or other forms of characteristics that are only temporarily present at any time and place. The processor 910 is an article of manufacture and / or a machine component. The processor 910 is configured to execute software instructions to perform the functions described in the various embodiments herein. The processor 910 may be a general-purpose processor or may be part of an application-specific integrated circuit (ASIC). The processor 910 may also be a microprocessor, microcomputer, processor chip, controller, microcontroller, digital signal processor (DSP), state machine, or programmable logic device. The processor 910 may also be logic circuitry, including programmable gate arrays (PGAs), such as field-programmable gate arrays (FPGAs), or other types of circuitry including discrete gate and / or transistor logic. Processor 910 may be a central processing unit (CPU), a graphics processing unit (GPU), or both. Furthermore, any processor described herein may include multiple processors, parallel processors, or both. Multiple processors may be included in a single device or multiple devices, or coupled to a single device or multiple devices.

[0106] As used herein, the term "processor" encompasses any electronic component capable of executing programs or machine-executable instructions. References to computing devices including "processor" should be interpreted as including more than one processor or processing core, as in a multi-core processor. A processor can also refer to a collection of processors within a single computer system or distributed across multiple computer systems. The term computing device should also be interpreted as including a collection or network of computing devices, each including one or more processors. A program has software instructions that are executed by one or more processors, which may be within the same computing device or distributed across multiple computing devices.

[0107] The computer system 900 also includes main memory 920 and static memory 930, wherein the memories in the computer system 900 communicate with each other and with the processor 910 via bus 908. Either or both of the main memory 920 and static memory 930 can be considered as... Figure 1 The memory 191 of the controller 190 is a representative example and stores instructions for implementing some or all aspects of the methods and processes described herein. The memory described herein is a tangible storage medium for storing data and executable software instructions, and is non-transitory while the software instructions are stored therein. As used herein, the term "non-transitory" should be interpreted as not a permanent characteristic of a state, but a characteristic of a state that will persist for a period of time. The term "non-transitory" specifically denies transient characteristics, such as carrier waves or signals, or other forms of characteristics that are only temporarily present at any time and place. Main memory 920 and static memory 930 are articles of manufacture and / or machine components. Main memory 920 and static memory 930 are computer-readable media from which a computer (e.g., processor 910) can read data and execute software instructions. Each of the main memory 920 and the static memory 930 may be implemented as one or more of random access memory (RAM), read-only memory (ROM), flash memory, electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, removable disks, magnetic tapes, optical disc read-only memory (CD-ROM), digital multifunction disks, floppy disks, Blu-ray discs, or any other form of storage medium known in the art. The memory may be volatile or non-volatile, secure and / or encrypted, insecure and / or unencrypted.

[0108] "Memory" is a paradigm of computer-readable storage media. Computer memory is any storage that the processor can directly access. Examples of computer memory include, but are not limited to, RAM, registers, and register files. References to "computer memory" or "memory" should be interpreted as potentially referring to multiple types of memory. Memory can be, for example, multiple storage devices within the same computer system. Memory can also be multiple storage devices distributed across multiple computer systems or computing devices.

[0109] As shown in the figure, the computer system 900 also includes a video display unit 950, such as a liquid crystal display (LCD), an organic light-emitting diode (OLED), a flat panel display, a solid-state display, or a cathode ray tube (CRT). Furthermore, the computer system 900 includes input devices 960 (e.g., a keyboard / virtual keyboard, or a touch-sensitive input screen, or voice input with voice recognition) and cursor control devices 970 (e.g., a mouse, a touch-sensitive output screen, or a keyboard). The computer system 900 may also optionally include a disk drive unit 980, a signal generation device 990 (e.g., a speaker or a remote control), and / or a network interface device 940.

[0110] In an embodiment, such as Figure 9 As shown, the disk drive unit 980 includes a computer-readable medium 982 in which one or more sets of software instructions 984 (software) are embedded. The set of software instructions 984 is read from the computer-readable medium 982 for execution by the processor 910. Furthermore, when executed by the processor 910, the software instructions 984 perform one or more steps of the methods and processes described herein. In embodiments, during execution of the computer system 900, all or part of the software instructions 984 reside in main memory 920, static memory 930, and / or the processor 910. Additionally, the computer-readable medium 982 may include the software instructions 984, or receive and execute the software instructions 984 in response to a propagation signal, causing a device connected to the network 901 to transmit voice, video, or data through the network 901. The software instructions 984 may be sent or received on the network 901 via a network interface device 940.

[0111] In the embodiments, dedicated hardware implementations (such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), programmable logic arrays, and other hardware components) are configured to implement one or more methods described herein. One or more embodiments described herein may use two or more specifically interconnected hardware modules or devices to implement the functionality, said hardware modules or devices having associated control and data signals capable of communicating between and through modules. Therefore, this disclosure covers software, firmware, and hardware implementations. Nothing in this application should be construed as being implemented or feasible solely in software without hardware (e.g., tangible, non-transitory processors and / or memory).

[0112] According to various embodiments of this disclosure, the methods described herein can be implemented using a hardware computer system that executes software programs. Furthermore, in exemplary, non-limiting embodiments, implementations can include distributed processing, component / object distributed processing, and parallel processing. Virtual computer system processing can implement one or more of the methods or functions described herein, and the processor described herein can be used to support virtual processing environments.

[0113] Therefore, interventional medical device tracking can complete the updated registration to correct the position of interventional medical device 101. However, interventional medical device tracking is not limited to the specific details described herein, but can be applied to other additional embodiments in which other types of medical imaging systems and interventional medical devices are used.

[0114] Although interventional medical device tracking has been described with reference to several exemplary embodiments, it should be understood that the language used is descriptive and illustrative, not restrictive. Changes may be made in its aspects within the scope and spirit of the appended claims, as stated herein and as amended, without departing from the scope and spirit of interventional medical device tracking. While interventional medical device tracking has been described with reference to specific means, materials, and embodiments, it is not intended to be limited to the specific details disclosed; rather, interventional medical device tracking extends to all functionally equivalent structures, methods, and uses, for example, within the scope of the appended claims.

[0115] The illustrated descriptions of the embodiments described herein are intended to provide a general understanding of the structures of various embodiments. These illustrations are not intended to be used as a complete description of all elements and features of the disclosure described herein. Many other embodiments will be apparent to those skilled in the art upon viewing this disclosure. Other embodiments can be derived from and from this disclosure, allowing structural and logical substitutions and changes to be made without departing from the scope of this disclosure. Furthermore, the illustrations are representative only and may not be drawn to scale. Some scales in the illustrations may be exaggerated, while others may be minimized. Therefore, this disclosure and the accompanying drawings should be considered illustrative rather than restrictive.

[0116] For convenience only, one or more embodiments of this disclosure may be individually and / or collectively referred to herein as the “Invention”, and are not intended to voluntarily limit the scope of this disclosure to any specific invention or inventive concept. Furthermore, although specific embodiments have been shown and described herein, it should be understood that any subsequent arrangements designed to achieve the same or similar purpose may replace the specific embodiments shown. This disclosure is intended to cover any and all subsequent changes or variations to the various embodiments. Combinations of the above embodiments and other embodiments not specifically described herein will be apparent to those skilled in the art upon review of the specification.

[0117] This abstract of the disclosure is provided to comply with the requirements of Title 37, Section 1.72(b) of the Federal Code, and it should be understood that the submission of this abstract is not intended to construe as limiting the scope or meaning of the claims. Furthermore, in the foregoing detailed description, various features may be grouped together or described in a single embodiment for the purpose of simplification. This disclosure should not be construed as reflecting an intention to require more features than expressly listed in each claim of the claimed embodiments. Rather, as reflected in the following claims, the subject matter of the invention may refer to fewer than all features of any of the disclosed embodiments. Therefore, the following claims are incorporated into the detailed description, each claim independently defining a separate claimed subject matter.

[0118] The foregoing description, which provides the disclosed embodiments, is intended to enable any person skilled in the art to practice the concepts described in this disclosure. Therefore, the subject matter of the foregoing disclosure is to be considered illustrative rather than restrictive, and the appended claims are intended to cover all such modifications, enhancements, and other embodiments falling within the true spirit and scope of this disclosure. Accordingly, to the fullest extent permitted by law, the scope of this disclosure shall be determined by the broadest permissible interpretation of the following claims and their equivalents, and shall not be limited or bound by the foregoing detailed description.

Claims

1. A system for tracking the location of an interventional medical device (01) during interventional medical procedures, comprising: An interface (193) to an optical shape sensing device (102) having a shape that follows the shape of the interventional medical device (01) during the interventional medical procedure; as well as A controller (190) includes a memory (191) for storing instructions and a processor (192) for executing the instructions, wherein, when executed by the processor (192), the instructions cause the system to: The shape of the optical shape sensing device (102) is identified using the optical shape sensing signal received from the optical shape sensing device (102) via the interface (193); Based on the shape of the optical shape sensing device (102), the interventional medical device (01) is identified in the first coordinate space of the first imaging system, and the first imaging system images the interventional medical device (01) in a first imaging mode during the interventional medical procedure. Register the interventional medical device (01) to the first coordinate space; The interventional medical device (01) is identified in a second coordinate space of a second imaging system, which images the interventional medical device (01) in a second imaging mode during the interventional medical procedure. Register the first coordinate space of the first imaging system to the second coordinate space of the second imaging system; The interventional medical device (01) is segmented in the second coordinate space to obtain a segmented representation (401) of the interventional medical device (01) in the second coordinate space. The interventional medical device (01) is registered to the second coordinate space using the segmentation representation (401) of the interventional medical device (01); and Based on registering the interventional medical device (01) to the second coordinate space, the interventional medical device (01) is re-registered to the first coordinate space.

2. The system according to claim 1, further comprising: The optical shape sensing device (102). X-ray imaging system (120), which includes the first imaging system; as well as An ultrasound imaging system (110) comprising the second imaging system, The instructions also cause the system to: The segmented representation (401) of the interventional medical device (01) is presented in the second coordinate space.

3. The system according to claim 1, wherein, The instructions also cause the system to: Calculate the transformation from the interventional medical device (01) identified in the second coordinate space to the segmented representation (401) of the interventional medical device (01) in the second coordinate space, and The second coordinate space is registered to the first coordinate space based on the transformation.

4. The system according to claim 3, wherein, The instructions also cause the system to: The selection of the interventional medical device (01) is detected in the image from the second imaging system, and the interventional medical device (01) is searched for in the region near the selection in the image. The transformation is calculated by translating the distal portion of the interventional medical device (01) in the image to match the corresponding distal portion of the interventional medical device (01) in the segmentation.

5. The system according to claim 3, wherein, The instructions also cause the system to: In the search of images in the second coordinate space of the system, the tip of the interventional medical device (01) is used as a constraint to repeatedly identify the interventional medical device (01) in the images. The interventional medical device (01) is segmented for each imaging frame in the second imaging mode; and The interventional medical device (01) is repeatedly registered to the second coordinate space.

6. The system according to claim 1, wherein, The instructions also cause the system to: The interventional medical device (01) is repeatedly identified in the image in the second coordinate space of the system. Determine when the position of the interventional medical device (01) deviates from the segmented representation (401) of the interventional medical device (01) in the first coordinate space by more than a predetermined threshold; When the position of the interventional medical device (01) deviates beyond the predetermined threshold, the interventional medical device (01) is re-segmented; and When the position of the interventional medical device (01) deviates beyond the predetermined threshold, the interventional medical device (01) is re-registered to the second coordinate space, and the second coordinate space is re-registered to the first coordinate space.

7. A tangible, non-transitory, computer-readable storage medium storing a computer program, which, when executed by a processor (192), causes a system comprising the tangible, non-transitory, computer-readable storage medium to: The shape of the optical shape sensing device (102) is identified by using optical shape sensing signals received via interface (193), the optical shape sensing device having a shape that follows the shape of the interventional medical device (01) during the interventional medical procedure; Based on the shape of the optical shape sensing device (102), the interventional medical device (01) is identified in the first coordinate space of the first imaging system, and the first imaging system images the interventional medical device (01) in a first imaging mode during the interventional medical procedure. Register the interventional medical device (01) to the first coordinate space; The interventional medical device (01) is identified in a second coordinate space of a second imaging system, which images the interventional medical device (01) in a second imaging mode during the interventional medical procedure. Register the first coordinate space of the first imaging system to the second coordinate space of the second imaging system; The interventional medical device (01) is segmented in the second coordinate space to obtain a segmented representation (401) of the interventional medical device (01) in the second coordinate space. The interventional medical device (01) is registered to the second coordinate space using the segmentation representation (401) of the interventional medical device (01); and Based on registering the interventional medical device (01) to the second coordinate space, the interventional medical device (01) is re-registered to the first coordinate space.

8. The computer-readable storage medium according to claim 7, wherein, The computer program also enables the system to: Identify the interventional medical device (01) in the second coordinate space of the second imaging system; Calculate the transformation from the interventional medical device (01) identified in the second coordinate space to the segmented representation (401) of the interventional medical device (01) in the second coordinate space, and The second coordinate space is registered to the first coordinate space based on the transformation.

9. A computer program product comprising a computer program that, when executed by a processor (192), causes the processor to perform a method comprising: The shape of the optical shape sensing device (102) is identified by using optical shape sensing signals received via interface (193), the optical shape sensing device having a shape that follows the shape of the interventional medical device (01) during the interventional medical procedure; Based on the shape of the optical shape sensing device (102), the interventional medical device (01) is identified in the first coordinate space of the first imaging system, and the first imaging system images the interventional medical device (01) in a first imaging mode during the interventional medical procedure. Register the interventional medical device (01) to the first coordinate space; The interventional medical device (01) is identified in a second coordinate space of a second imaging system, which images the interventional medical device (01) in a second imaging mode during the interventional medical procedure. Register the first coordinate space of the first imaging system to the second coordinate space of the second imaging system; The interventional medical device (01) is segmented in the second coordinate space to obtain a segmented representation (401) of the interventional medical device (01) in the second coordinate space. The interventional medical device (01) is registered to the second coordinate space using the segmentation representation (401) of the interventional medical device (01); and Based on registering the interventional medical device (01) to the second coordinate space, the interventional medical device (01) is re-registered to the first coordinate space.

10. The computer program product according to claim 9, wherein the method further comprises: Presenting the segmented representation (401) of the interventional medical device (01) in the second coordinate space, and Repeat the following operations: segmentation, registration of the interventional medical device (01) to the second coordinate space, and re-registration of the interventional medical device (01) to the first coordinate space.

11. The computer program product according to claim 10, in, Based on the user selection of the interventional medical device (01) in the second coordinate space, perform the following operations: segmentation, registration of the interventional medical device (01) to the second coordinate space, and re-registration of the interventional medical device (01) to the first coordinate space.

12. The computer program product according to claim 10, in, The repetition is performed automatically and periodically.

13. The computer program product according to claim 10, wherein the method further comprises: The position of the interventional medical device (01) in the image in the second coordinate space has been detected to have moved from the position of the segmentation representation (401) of the interventional medical device (01), and The following operations are automatically performed based on the detection: segmentation, registration of the interventional medical device (01) to the second coordinate space, and re-registration of the interventional medical device (01) to the first coordinate space.

14. The computer program product according to claim 9, wherein the method further comprises: Identify the interventional medical device (01) in the second coordinate space of the second imaging system; Calculate the transformation from the interventional medical device (01) identified in the second coordinate space to the segmented representation (401) of the interventional medical device (01) in the second coordinate space, and The second coordinate space is registered to the first coordinate space based on the transformation.

15. The computer program product according to claim 14, wherein the method further comprises: The selection of the interventional medical device (01) is detected in the image from the second imaging system, and the interventional medical device (01) is searched for in the region of the image close to the selection. The transformation is calculated by translating the distal portion of the interventional medical device (01) in the image to match the corresponding distal portion of the interventional medical device (01) in the segmentation.

Images