Device tracking and monitoring out of imaging field of view

Abstract

A system includes a memory, a processor, and a motor. Instructions from memory cause the system to: receive an image; receiving encoded data corresponding to driving of the elongated device by the motor; determining the position of the elongated device; determining when at least a portion of the elongate device is outside of the field of view in the image or may exit the field of view in the image; determining whether to create extended image data to show a position of the elongated device outside the field of view in the image; determining a current position, a past position, and / or a predicted position of the elongated device outside a field of view in the image based on pixel data in the image and the encoded data; and expanding the image to show the position of the elongated device outside the field of view in the image in the expanded image.

Description

Background Art

[0001] Robotic assistance is used during intravascular procedures to improve clinical outcomes and care delivery under fluoroscopic guidance. Potential benefits of using robotic assistance include increased safety and reduced total fluoroscopy acquisition during intravascular procedures performed under fluoroscopic guidance.

[0002] An intravascular robot allows for automatic or guided navigation of intravascular devices within a patient's vasculature using a motorized actuation unit. During an intravascular procedure, under fluoroscopic guidance, an intravascular robot is used to navigate intravascular devices such as catheters and guidewires, which means continuous radiation. Nevertheless, the tip of the guidewire may leave the field of view of the fluoroscopic image and become invisible. To re-acquire the tip of the guidewire, additional X-ray images may be required, and thus additional radiation to the patient and / or surgical staff will be needed. Summary of the Invention

[0003] According to one aspect of the present disclosure, a system for tracking an elongate device in an anatomical structure includes a memory, a processor, and a motor. The memory stores instructions. The processor runs the instructions. When run by the processor, the instructions cause the system to: receive an image; receive encoded data corresponding to the drive of the elongate device by the motor; determine the position of the elongate device; determine when at least a portion of the elongate device is outside the field of view in the image or likely to leave the field of view in the image; determine whether to create extended image data to show the position of the elongate device outside the field of view in the image; determine at least one of the current position, past position, or predicted position of the elongate device outside the field of view in the image based on pixel data in the image and the encoded data; and extend the image to show the position of the elongate device outside the field of view in the extended image.

[0004] Further embodiments of this aspect of the present disclosure are also described based on claims 2-9, considered individually or in any combination.

[0005] According to another aspect of the present disclosure, a method for tracking a slender device in an anatomical structure includes: receiving an image of the slender device at a system including a memory storing instructions and a processor executing the instructions; receiving encoded data corresponding to actuation of the slender device; determining a position of the slender device; determining when at least a portion of the slender device is outside a field of view in the image or likely to leave the field of view in the image; determining whether to create extended image data to show a position of the slender device outside the field of view in the image; determining a current or predicted position of the slender device outside the field of view in the image based on the pixel data and the encoded data; and extending the image to show the position of the slender device outside the field of view in the image in an extended image.

[0006] Further embodiments of the method of the present disclosure are also described in view of claims 11 to 14, considered individually or in any combination. Additional alternative embodiments of the method (considered individually or in combination with any other embodiments) include:

[0007] generating, by an X-ray machine, an image of the slender device as a two-dimensional X-ray image, wherein the slender device includes an intravascular device;

[0008] updating an acquisition of the image based on the encoded data and the pixel data to return the slender device to the field of view in the image; and / or

[0009] determining that creating the extended image data is based on determining that the slender device is outside the field of view in the image.

[0010] According to another aspect of the present disclosure, a controller for tracking a slender device in an anatomical structure includes: a memory storing instructions; and a processor executing the instructions. When executed by the processor, the instructions cause the system to: receive an image including the slender device; receive encoded data corresponding to actuation of the slender device; determine a position of the slender device; determine when at least a portion of the slender device is outside a field of view in the image or likely to leave the field of view in the image; determine whether to create extended image data to show a position of the slender device outside the field of view in the image; determine a current or predicted position of the slender device outside the field of view in the image based on the pixel data and the encoded data; and extend the image to show the position of the slender device outside the field of view in the image in an extended image.

[0011] Further alternative embodiments of the controller of the present disclosure (considered individually or in combination) include:

[0012] When run by the processor, the instructions further cause the controller to provide the display with data regarding the position of the elongate device outside the field of view in the image to supplement the image with additional field of view image data projecting the position of the elongate device outside the field of view in the image; and / or

[0013] A motor, which is configured to drive the elongate device, is included or not included in the controller. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Example embodiments may be best understood from the following detailed description when read in conjunction with the accompanying drawings. It should be emphasized that the various features are not necessarily drawn to scale. In fact, the dimensions may be increased or decreased for the sake of discussion. Where applicable and practical, like reference numerals refer to like elements.

[0015] Figure 1 Illustrated is a system for device tracking and monitoring outside an imaging field of view in accordance with a representative embodiment.

[0016] Figure 2 Illustrated is a method for device tracking and monitoring outside an imaging field of view in accordance with a representative embodiment.

[0017] Figure 3 Illustrated is the visualization of a robot-controlled coaxial guidewire for device tracking and monitoring outside an imaging field of view in accordance with a representative embodiment.

[0018] Figure 4 Illustrated is an overview of a system for device tracking and monitoring outside an imaging field of view in accordance with a representative embodiment.

[0019] FIG. 5 illustrates the visualization of a device when the device is within the imaging field of view and another visualization when the device is outside the imaging field of view in accordance with a representative embodiment.

[0020] Figure 6 Illustrated is a zoom function for device tracking and monitoring outside an imaging field of view in accordance with a representative embodiment.

[0021] Figure 7 Illustrated is a pan function for device tracking and monitoring outside an imaging field of view in accordance with a representative embodiment.

[0022] FIG. 8 illustrates warping for device tracking and monitoring outside an imaging field of view in accordance with a representative embodiment.

[0023] Figure 9Illustrated is a computer system according to another representative embodiment, on which a method for device tracking and monitoring outside the imaging field of view is implemented. Detailed Description

[0024] In the following detailed description, for purposes of explanation and not limitation, exemplary embodiments disclosing specific details are set forth in order to provide a thorough understanding of embodiments according to the teachings of the present invention. However, other embodiments consistent with the present disclosure that depart from the specific details disclosed herein are still within the scope of the claims. Descriptions of known systems, devices, materials, operating methods, and manufacturing methods may be omitted to avoid obscuring the description of the representative embodiments. Nevertheless, systems, devices, materials, and methods within the capabilities of those of ordinary skill in the art are within the scope of this teaching and may be used in accordance with the representative embodiments. It should be understood that the terms used herein are for the purpose of describing particular embodiments only and are not intended to be limiting. The definitions and explanations of the terms herein supplement the technical and scientific meanings of terms commonly understood and accepted in the technical field of the teachings of the present invention.

[0025] It should be understood that although the terms first, second, third, etc. may be used herein to describe various elements or components, these elements or components should not be limited by these terms. These terms are only used to distinguish one element or component from another. Thus, a first element or component discussed below may also be referred to as a second element or component without departing from the teachings of the inventive concept.

[0026] As used in the specification and claims, the singular forms of the terms "a," "an," and "the" are intended to include the singular and plural forms as well, unless the context clearly dictates otherwise. Additionally, when used in this specification, the terms "comprises" and / or "comprising" and / or similar terms specify the presence of the recited features, elements, and / or components, but do not preclude the presence or addition of one or more other features, elements, components, and / or groups thereof. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0027] Unless otherwise stated, when an element or component is said to be "connected to," "coupled to," or "adjacent to" another element or component, it will be understood that the element or component can be directly connected or coupled to the other element or component, or there can be intervening elements or components. That is, these and similar terms include cases where one or more intervening elements or components may be employed to connect two elements or components. However, when an element or component is said to be "directly connected" to another element or component, this only includes the case where the two elements or components are connected to each other without any intervening or intermediate elements or components.

[0028] The present disclosure, through one or more of its aspects, embodiments, and / or specific features or sub-components, is intended to bring one or more of the specifically pointed out advantages as follows.

[0029] As described herein, robotic intelligence and X-ray imaging can be combined to update image acquisition and image visualization based on tracking an intravascular device outside the field of view of an X-ray system. The end of a guidewire outside the field of view can be tracked through information in X-ray images and a robotic encoder so as to provide an overlay for the end of the guidewire on an extended image. The extended image can be obtained by combining current and past X-ray images. Generation of the extended image allows the position of a coaxial intravascular device to be observed without moving the X-ray system or acquiring new X-ray images, thus improving safety and simplifying the workflow.

[0030] Figure 1 System 100 for device tracking and monitoring outside an imaging field of view according to a representative embodiment is illustrated.

[0031] Figure 1 System 100 therein is a system for device tracking and monitoring outside an imaging field of view and includes components that can be provided together or can be provided distributively. System 100 includes an imaging device 101, an elongate device 120, a robot 130, a computer 140, a motor 160, and a display 180.

[0032] The imaging device 101 can include an X-ray machine or an X-ray system. The X-ray machine or the X-ray system can generate two-dimensional X-ray images. The X-ray machine or the X-ray system can be configured to generate an image of the elongate device 120 as a two-dimensional X-ray image.

[0033] The elongate device 120 can include an intravascular device. For example, the elongate device 120 can include a catheter with a guidewire coaxially placed inside it to advance and retract from the front portion of the catheter to the front portion of the catheter. The elongate device 120 can include a distal end that first advances from the front portion of the catheter and a proximal end close to the robot 130. When the guidewire is inside the catheter or outside the field of view of the X-ray machine or the X-ray system, the guidewire may be invisible in X-ray imaging. The elongate device 120 is driven by the robot 130 under the power of the motor 160 and the control of the computer 140.

[0034] The robot 130 can include a controllable device driven by the motor 160 and movable in one or more degrees of freedom. The robot 130 is configured to drive the elongate device 120 at least forward and backward as one degree of freedom. The robot 130 can also be configured to rotate the elongate device 120 as a second degree of freedom.

[0035] The motor 160 may include an electric motor, a linear motor, a precision stepper motor, or a servo motor (“servo”). In some embodiments, the robot 130 and the motor 160 may include an integrated unit, referred to as either or both of the robot and / or the motor. The motor 160 is configured to drive the robot 130 to drive the elongate device 120 in one or more degrees of freedom.

[0036] The motor 160 is also provided with an encoder 165 for tracking and encoding the movement of the robot 130 and thus the movement of the elongate device 120 in one or more degrees of freedom. The encoder 165 outputs encoded data reflecting the amount of movement of the robot 130 and thus the elongate device 120 in each degree of freedom.

[0037] The computer 140 may include a workstation, a laptop computer, a desktop computer, or a dedicated computer. The computer 140 includes at least a first interface 141, a second interface 142, a third interface 143, and a controller 150. Figure 9 Computers that can be used to implement the computer 140 are depicted, but the computer 140 may include more or fewer elements than Figure 1 or Figure 9 shown. The first interface 141 connects the computer 140 to the imaging device 101. The second interface 142 connects the computer 140 to the motor 160 and / or the encoder 165. The third interface 143 connects the computer 140 to the display 180. The first interface 141, the second interface 142, and the third interface 143 may include ports, adapters, and / or other types of suitable hardware configured to accept cable inputs from cables connected to the imaging device 101, the motor 160 and / or the encoder 165, and the display 180.

[0038] The controller 150 includes at least a memory 151 that stores instructions and a processor 152 that runs the instructions. In some embodiments, Figure 1 multiple different elements of the system 100 in

[0039] The controller 150 can directly perform some of the operations described herein and can indirectly implement other operations described herein. For example, the controller 150 can indirectly control operations, such as by generating and transmitting content to be displayed on the display 180. For example, the controller 15 can provide a display signal adapted to show the position of the elongate device 120 outside the field of view in an image on the display 180. The controller 150 can directly control other operations, such as logical operations performed by the processor 152 running instructions from the memory 151 based on inputs received via the interface from electronic components and / or the user. Thus, when the processor 152 runs instructions from the memory 151, the processes implemented by the controller 150 may include steps that the controller 150 does not directly perform.

[0040] The method implemented by the system 100 under the control of the controller 150 can track the elongate device 120 in the anatomical structure. When run by the processor 152, the instructions from the memory 151 cause the system 100 to: receive an image from the imaging device 101; receive encoded data corresponding to the driving of the elongate device 120 by the motor 160 from the encoder 165; determine the position of the elongate device 120; determine when at least a portion of the elongate device 120 is outside the field of view in the image or may leave the field of view in the image; determine whether to create extended image data to show the position of the elongate device outside the field of view in the image on the display 180; determine at least one of the current position, past position, or predicted position of the elongate device 1120 outside the field of view in the image based on the pixel data in the image and the encoded data; and extend the image to show the position of the elongate device outside the field of view in the image in the extended image on the display 180.

[0041] For example, the software program executed by the processor 152 can include a tracking unit, an image extension unit, and other units corresponding to subroutines of the software program described herein. The tracking unit and the image extension unit are used to track and visualize the elongate device 120 in the image data acquired from the imaging device 101 during the process. The image extension unit can be run to create extended image data for an extended image extending from the imaging field of view of the current image data. The extended image data can be created by the image extension unit based on other relevant image data. The tracking unit can be run to receive the current image data, the robotic data, and the extended image data from the image extension unit to determine the position of the elongate device 120 in the extended image when the position of the end of the elongate device 120 is outside the imaging field of view or may leave the field of view in the image.

[0042] The display 180 can be local to the controller 150 or can be remotely connected to the controller 150. The display 180 can be connected to the controller 150 via a local wired interface (such as an Ethernet cable) or via a local wireless interface (such as a Wi-Fi connection). The display 180 can be connected to other user input devices through which a user can input instructions, including a mouse, a keyboard, a trackball, etc. The display 180 can be a monitor, such as a computer monitor, a display on a mobile device, an augmented reality display, a television, an electronic whiteboard, or another screen configured to display an electronic image. The display 180 can also include one or more input interfaces (such as the input interfaces mentioned above that can be connected to other components or parts), and an interactive touch screen configured to display prompts to the user and collect touch inputs from the user. The display 180 can be configured to display the position of the elongate device 120 outside the field of view in the image by supplementing the image with additional image data that projects the position of the elongate device 120 outside the field of view in the image.

[0043] Figure 2 Illustrated is a method for device tracking and monitoring outside an imaging field of view according to a representative embodiment.

[0044] At S210, Figure 2 the method begins with generating and sending a two-dimensional X-ray image. S210 can be performed by the imaging device 101, and the two-dimensional X-ray image can be sent directly to the computer 140 via a connected cable or indirectly to the computer 140 via a local area network (LAN) such as a WiFi network and / or via a wide area network (WAN) such as the Internet.

[0045] At S220, Figure 2 the method includes driving the intravascular device and encoding data. S220 can be performed by the motor 160 and the encoder 165. The encoder 165 can encode data based on the amount by which the motor 160 drives the elongate device 120. The encoded data can include data reflecting the drive amount for each degree of freedom. S220 can be performed simultaneously with S210 during an interventional medical procedure, and both S210 and S220 can be performed intermittently or continuously for the remainder of the time of performing Figure 2 the method.

[0046] At S230, Figure 2 the method includes receiving the two-dimensional X-ray image and the encoded data. S230 can be performed by the computer 140 receiving the X-ray image from the imaging device 101 via the first interface 141.

[0047] At S240, Figure 2The method includes outputting a representation of displacement based on encoded data. The displacement can be an estimate of the movement of the elongate device 120 in one or more degrees of freedom based on the amount of time the motor 160 drives the elongate device 120, based on the speed at which the motor 160 drives the elongate device 120, and / or based on one or more any other factors that the motor 165 may consider. The representation of displacement can be the amount of displacement for each of one or more degrees of freedom.

[0048] At S250, the position of the intravascular device is determined. The intravascular device can be the elongate device 120, and the position can be determined by the controller 150. These positions can be two-dimensional or three-dimensional coordinates of the position of at least a portion of the elongate device 120 within the field of view of the X-ray image or relative to the field of view. The position determined at S250 can include the current position, past position, or future predicted position of the elongate device 120. For example, when the end of the elongate device 120 re-enters the field of view after being outside the field of view, the current position(s) of the end within the field of view and the most recent past position of the end outside the field of view can be determined. Additionally or alternatively, when the end of the elongate device 120 may leave the field of view in the image, such as based on tracking the speed and directionality of the end in past images, the determination of the position at S250 can be made.

[0049] At S260, Figure 2 The method includes determining when the intravascular device is outside the field of view. The determination at S260 can be determining when at least a portion of the elongate device 120 is outside the field of view in the image or may leave the field of view in the image. The likelihood that the elongate device 120 will leave the field of view can be determined based on the speed and directionality of the elongate device 120 shown in the most recent image.

[0050] At S270, Figure 2 The method includes determining whether to create extended image data. The determination at S270 is based on whether the intravascular device is outside the field of view or may move outside the field of view. The extended image data can provide an extended image, such as an extended image with a different color or shading, with a separate border or border type, or otherwise depict to the operator that the extended image is an artificial projection outside the field of view and extends from the actual image within the field of view. If it is not necessary to create extended image data (S270 = no), then Figure 2 The method returns to S250.

[0051] If extended image data is to be created (S270 = yes), then Figure 2The method includes determining the position of the elongate device outside the field of view at S280. The determination at S280 can be based on the pixel data in the image and the encoded data to determine at least one of the current position, past position, or predicted position of the elongate device outside the field of view in the image. The current position or predicted position may include an estimated position of the end of the elongate device 120. For example, the processor 152 can provide a display signal adapted to show the position of the elongate device 120 outside the field of view in the image on the display 180.

[0052] At S290, Figure 2 The method includes extending the X-ray image to the projected position of the elongate device outside the field of view. The end image of the elongate device 120 can be superimposed on the extended image at the position determined at S280 based on the pixel data and the encoded data. Extending the X-ray image at S290 can be based on at least one of the current image data of the current image and the past image data of the past image. The image can be extended at S290 by performing at least one of warping, mosaicking, or stitching on at least one image. A representation of the displacement of the elongate device 120 can be output based on the encoded data, and a representation of the displacement of the elongate device 120 can be generated based on the warping, mosaicking, or stitching of at least one image.

[0053] As described above, Figure 2 The method can include tracking the end of the elongate device 120 when the end is within the field of view of the image and when the end leaves or enters the field of view of the image. The generated extended image shows the current, past, or predicted position of the end, which can help the operator understand the scene and potential hazards presented by any part of the elongate device 120 outside the field of view. In some embodiments, the system 100 can generate an alert based on the movement of the elongate device 120 (such as based on the current, past, or future position, speed, or any other metric available to the system 100).

[0054] Figure 3 Illustrated is the visualization of the use of a robotically controlled coaxial guidewire for device tracking and monitoring outside the field of view of the imaging field according to a representative embodiment.

[0055] Figure 3 The visualization in is provided on the user interface 381, such as Figure 1 the display 180 in. The current image shows the elongate device in the descending aorta. The extended X-ray image is provided below the current image as a previous scene and includes an X marking the end of the guidewire outside the field of view (FOV).

[0056] Figure 3A visualization solution is illustrated, in which the current position of the guide wire end tracked using robotic data is superimposed on a previously acquired X-ray image. The coaxial guide wire can be controlled using the robot even when it is outside the field of view. Figure 3 The end of the middle guide wire is indicated by a cross.

[0057] Figure 4 An overview of a system for device tracking and monitoring outside an imaging field of view according to a representative embodiment is illustrated.

[0058] In Figure 4 the processing unit implemented by the controller 150 in Figure 1 uses robotic data and image data to calculate the position of the elongate device 120 outside the imaging field of view and superimposes the position of the end of the elongate device 120 on the extended view image.

[0059] Figure 4 The system overview in

[0060] includes one or more medical imaging systems 401, an intervention kit 420, a robotic system 430, a relative scale estimator 445, a tracking unit 446, an image expansion unit 447, an action manager 448, and a display 480.

[0061] For example, the medical imaging system 401 can include a C-arm X-ray system. The medical imaging system 401 outputs image data such as X-ray data.

[0062] The robotic system 430 can include any one or both of a robot 130 and a motor 160, and Figure 1 the encoder 165 in

[0063] The relative scale estimator 445, the tracking unit 446, the image expansion unit 447, and the action manager 448 can be implemented by software stored in the memory 151 of the controller 150 in Figure 1 and one or more instances of the processor 152 of the controller 150 in Figure 1

[0064] ​The relative scale estimator 445 receives imaging data and robotic data. The relative scale estimator 445 is configured to: receive (i) current image data in the imaging field of view and (ii) robotic data, and output a scalar parameter representing the displacement of the elongate device in the live image associated with the robotic data. Due to the foreshortening of X-rays and the projection geometry configuration, the scale of each pixel may be different. The extended image data results in warping of the images for stitching, and the relative scale estimator 445 necessarily takes this warping into account to estimate the actual position.

[0065] The relative scale estimator 445 calculates a scalar parameter that is the ratio of robotic actuation to displacement in image coordinates. Given this ratio, the controller 150 can predict how much displacement of the end of the elongate device 120 will occur in image coordinates for a particular robotic actuation command. Due to the foreshortening of X-rays and the projection geometry configuration, the ratio is different for each pixel coordinate in the image, each device type, each robotic configuration, each anatomical structure, etc. The relative scale estimator 445 can be implemented using a trained artificial intelligence model or a look-up table, each of which will be explained later. It should be noted that the images will be warped for stitching and creating an extended view, so the calculated scale factor will change as the images are warped. The warping issue, including how to update the scale factor using interpolation, is discussed in the description of FIG. 8.

[0066] The tracking unit 446 and the image extension unit 447 are used to track and visualize the interventional device controlled by the robotic system 430 during the surgical procedure from the image data acquired by the medical imaging system 401. The tracking unit 446 uses the extended image to calculate the position of the end of the elongate device 120 outside or at the boundary of the imaging field of view. The tracking unit 446 uses the scalar parameter from the relative scale estimator 445 to determine the position of the elongate device.

[0067] When the end is outside the imaging field of view, the tracking unit 446 calculates the displacement of the end of the elongate device 120. The tracking unit 446 receives the extended image canvas, the dynamic scale factor, the robotic data, and calculates the position of the end relative to the image coordinate system, as shown in FIG. 5. The tracking unit 446 continuously calculates the position of the elongate device 120 in the general workflow by combining the robotic and image data. When the elongate device 120 leaves the field of view, the tracking unit 446 will notify the action manager 448, thereby triggering an update on the display 480 or the intervention kit 420.

[0068] The image extension unit 447 is arranged to create extended image data extending from the imaging field of the current image data. The extended image data is created based on other relevant image data. The tracking unit 446 is arranged to receive (i) the current image data and (ii) the robotic data, as well as the extended image data from the image extension unit 447, to determine the position of the guide wire tip in the extended image when the position of the guide wire tip is outside the field of view imaging field. The extended image data from the image extension unit 447 uses the latest image and the previous image. The previous imaging may include both previous X-ray images obtained from the same patient in the same procedure or two-dimensional imaging synthesized from previous three-dimensional imaging (such as digitally reconstructed radiographs).

[0069] The image extension unit 447 uses the latest image and the previous image as Figure 3 shown to create an extended graphical interface. The previous imaging includes both previous X-ray images obtained from the same patient in the same procedure or two-dimensional imaging synthesized from previous three-dimensional imaging (such as digitally reconstructed radiographs).

[0070] Once the position of the end of the elongate device 120 outside the field of view is determined by the tracking unit 446, the action manager 448 will update the display 480 to show that the guide wire tip is outside the imaging field of view, or trigger another event, such as changing the imaging settings to capture the guide wire tip within a new imaging field of view or controlling the movement of the robotic control device accordingly. For example, the change in imaging settings may include zooming or panning. Depending on the workflow, stage, or user preference, the action manager 448 unit will update the display to visualize the position of the device tip (as Figure 3 shown), or will trigger an event in the intervention kit 420, such as changing the imaging settings to capture the device tip within the imaging field of view or controlling the movement of the robotically controlled device accordingly.

[0071] Using Figure 4 the system in the system overview, the tip of the guide wire is tracked outside the field of view by aggregating information in the X-ray images and the robotic encoder. The system provides an overlay for the guide wire tip on the extended image on the display 480. The extended image is obtained by combining the current X-ray image and the past X-ray images.

[0072] FIG. 5 illustrates the visualization of the device when the device is within the imaging field of view according to a representative embodiment, and another visualization when the device is outside the imaging field of view.

[0073] The visualizations in FIG. 5 are shown as a series of user interfaces, such as the display on the display 180. The sequence includes: a first user interface 581A for visualizing the device when it is within the imaging field of view, and a second user interface 581B for visualizing the device when it is outside the imaging field of view.

[0074] In FIG. 5, when the elongate device 120 is within the field of view and is imaged on the first user interface 581A and is active, a live X-ray image is displayed in the graphical interface. When the elongate device 120 leaves the field of view and image acquisition is paused, the controller 150 calculates the position of the elongate device 120 with respect to the image coordinate system for the second user interface 581B.

[0075] Figure 6 Illustrated is a zoom function for device tracking and monitoring outside the imaging field of view according to a representative embodiment.

[0076] Figure 6 The zoom function in is provided on the user interface 681, such as on the display 180 in. In, the elongate device in the shoulder and arm in the upper left corner is reduced in the lower right corner, such as being controlled via the controller 150 in. Figure 1 in Figure 6 in, the elongate device in the shoulder and arm in the upper left corner is reduced in the lower right corner, such as being controlled via the controller 150 in. Figure 1 in Figure 1 the imaging device 101 in.

[0077] The embodiment controls the zoom of the system based on the state of the device end when it is outside the field of view. For example, if the end of the hidden device leaves the field of view, then: 1) zoom out, 2) reacquire one or more X-ray images, 3) superimpose the end position on the one or more new images.

[0078] As Figure 6 shown in, imaging acquisition parameters associated with changing the scale can be triggered and adjusted based on the position of the device outside the current imaging cone.

[0079] Figure 7 Illustrated is a pan function for device tracking and monitoring outside the imaging field of view according to a representative embodiment.

[0080] Figure 7 The pan function in is provided on the user interface 781, such as on the display 180 in. In, the elongate device in the shoulder and arm in the upper left corner is panned to the left in the lower right corner, such as being controlled via the controller 150 in. Figure 1 in Figure 7 in, the elongate device in the shoulder and arm in the upper left corner is panned to the left in the lower right corner, such as being controlled via the controller 150 in. Figure 1 in Figure 1 the imaging device 101 in.

[0081] Figure 7An embodiment of controlling the translation of system 100 based on the state when the end of the elongate device 120 is outside the field of view is shown. For example, if the end of the hidden guidewire (such as the elongate device 120) leaves the field of view, the imaging view can be translated in the direction of the end position, 2) (one or more) X-ray images can be re-acquired, and 3) the end position can be superimposed on the (one or more) new images. For example, when the device reaches the boundary of the image, the C-arm system may translate so that the guidewire device remains visible.

[0082] FIG. 8 illustrates a warp for device tracking and monitoring outside the imaging field of view according to a representative embodiment.

[0083] As shown in FIG. 8, the relative scales in the original image (a) and the warped image (b) are different, d≠dw. FIG. 8 illustrates warping to adapt the relative scale to the situation outside the field of view. The warping in FIG. 8 is displayed via two user interfaces, including a first user interface 881A and a second user interface 881B. This warping reflects the complexity of estimating the depth of the position of the elongate device in the anatomical structure when capturing a two-dimensional image from a two-dimensional imaging system (such as a two-dimensional X-ray system). The two-dimensional image in the first user interface 881A shows the distance d between the end of the guidewire and the designated position. When the elongate device is not perpendicular to the imager of the two-dimensional imaging system, the controller 150 uses warping techniques to transform the position on the first user interface 881A into a warped position on the second user interface 881B.

[0084] Warping the image for stitching will require updating the relative scale to match the scale in the warped image (FIG. 8). This embodiment proposes a method for updating the relative scale based on image warping. To this end, corresponding key points need to be identified in the original image and the warped image. The key points can be handcrafted features (e.g., SIFT) or context-specific markers (e.g., the end of an intravascular device). The cross ratio or the changing distance in the markers is regarded as the update factor (u = dw / d). Finally, for each point in the warped image, a new scale is obtained by multiplying the original scale s by the update factor: su = s * u.

[0085] In some other embodiments, the pause can be implemented by the controller 150. For example, when the end of the elongate device 120 leaves the image boundary, the controller 150 can pause the actuation of the elongate device 120.

[0086] In some embodiments, the gain control can be implemented by the controller 150. For example, the controller 150 can adjust the gain of the robot 130 for actuating the elongate device 120 according to whether the elongate device 120 is entering or leaving the image boundary. For example, when the elongate device 120 leaves the field of view, the gain can be reduced to increase safety.

[0087] In some embodiments, a digitally reconstructed radiograph can be created by controller 150. When the elongate device 120 leaves the field of view, a digitally reconstructed radiographic image can be generated based on previous three-dimensional imaging to visualize the end of the elongate device 120 outside the field of view, rather than stitching together different images.

[0088] In some embodiments, a trained artificial intelligence model can be used to implement a relative scale estimator. The relative calibration scale can be learned by the artificial intelligence model. A neural network having convolutional layers, fully connected layers, and recurrent layers can be used to learn the scale value of the end of the elongate device 120 based on given imaging and robotic data. The neural network can also use segmentations of different devices and anatomical structures as inputs. Ground truth labels can be generated retrospectively from previous robotic navigation or developed in a synthetic environment.

[0089] In some embodiments, a look-up table can be used to implement a relative scale estimator. For example, relative scale factors can be stored in a look-up table at those positions in the image traversed by the end of the catheter device as the catheter advances through the vasculature. Then, when the guidewire is retracted within the catheter, the corresponding scale can be approximated by assigning a scale label from the look-up table to the closest pixel.

[0090] Figure 9 A computer system is illustrated in accordance with another representative embodiment, on which a method for device tracking and monitoring outside an imaging field of view is implemented.

[0091] Reference Figure 9 , computer system 900 includes a set of software instructions that can be run to cause computer system 900 to perform any of the methods or computer-based functions disclosed herein. Computer system 900 can operate as a stand-alone device or can be connected, for example, using network 901 to other computer systems or peripheral devices. In an embodiment, computer system 900 performs logic processing based on digital signals received through an analog-to-digital converter.

[0092] In a networked deployment, the computer system 900 can operate as a server or client user computer in a server-client user network environment, or as a peer computer system in a peer (or distributed) network environment. The computer system 900 can also be implemented as or incorporated into various devices, such as a workstation, a fixed computer, a mobile computer, a personal computer (PC), a laptop, a tablet computer, or any other machine capable of executing a set (sequential or other) of software instructions including a controller, the set of which specifies the operations to be taken by the machine. The computer system 900 can be incorporated into or incorporated into a device as a device, which is then included in an integrated system containing additional devices. In an embodiment, the computer system 900 can be implemented using an electronic device that provides voice, video or data communication. In addition, the computer system 900 is shown as a monomer, but the term "system" should also be considered to include a collection of any system or subsystem that executes one or more sets of software instructions individually or jointly to perform one or more computer functions.

[0093] like Figure 9 As shown in , computer system 900 includes processor 910. Processor 910 can be regarded as a representative example of a processor of a controller and executes instructions to implement some or all aspects of the methods and processes described herein. Processor 910 is tangible and non-transient. As used herein, the term "non-transient" should not be interpreted as a permanent feature of a state, but rather as a feature of a state that will last for a period of time. The term "non-transient" specifically negates fleeting features, such as features of a specific propagation carrier or signal or other forms that exist only transiently anywhere at any time. Processor 910 is an article and / or machine component. Processor 910 is configured to execute software instructions to perform functions as described in various embodiments herein. Processor 910 may be a general-purpose processor, or may be part of an application-specific integrated circuit (ASIC). Processor 910 may also be a microprocessor, a microcomputer, a processor chip, a controller, a microcontroller, a digital signal processor (DSP), a state machine, or a programmable logic device. Processor 910 may also be a logic circuit, including a programmable gate array (PGA) such as a field programmable gate array (FPGA), or another type of circuit including discrete gates and / or transistor logic. Processor 910 may be a central processing unit (CPU), a graphics processing unit (GPU), or both. In addition, any processor described herein may include multiple processors, parallel processors, or both. Multiple processors may be included in or coupled to a single device or multiple devices.

[0094] As used herein, the term "processor" encompasses electronic components capable of running programs or machine-executable instructions. References to a computing device that includes a "processor" should be construed to include more than one processor or processing core, such as a multi-core processor. A processor may also refer to a collection of processors within a single computer system or distributed among multiple computer systems. The term computing device should also be construed to include a collection or network of computing devices, each of which includes one or more processors. A program has software instructions that are executed by one or more processors, which may be within the same computing device or the multiple processors may be distributed across multiple computing devices.

[0095] Computer system 900 also includes main memory 920 and static memory 930, where the memories in computer system 900 communicate with each other and with processor 910 via bus 908. Either or both of main memory 920 and static memory 930 can be regarded as representative examples of the controller's memory and store instructions for implementing some or all aspects of the methods and processes described herein. Memory 307 described herein is a tangible storage medium for storing data and executable software instructions and is non-transitory during the time when the software instructions are stored. As used herein, the term "non-transitory" should not be construed as an eternal characteristic of a state, but rather as a characteristic of a state that will persist for a period of time. The term "non-transitory" specifically negates fleeting characteristics, such as those of a particular propagating carrier or signal or other forms that exist transiently only anywhere at any time. Main memory 920 and static memory 930 are articles of manufacture and / or machine components. Main memory 920 and static memory 930 are computer-readable media from which a computer (e.g., processor 910) can read data and executable software instructions. Memory 920 and static memory 930 can each be implemented as one or more of random access memory (RAM), read-only memory (ROM), flash memory, electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, one or more magnetic disks in a hard disk, removable disks, magnetic tape, compact disc read-only memory (CD-ROM), digital versatile disc (DVD), floppy disk, Blu-ray disc, or any other form of storage medium known in the art. The memory can be volatile or non-volatile, secure and / or encrypted, insecure and / or unencrypted.

[0096] "Memory" is an example of a computer-readable storage medium. Computer memory is any memory that the processor can directly access. Examples of computer memory include, but are not limited to, RAM memory, registers, and register files. References to "computer memory" or "memory" should be construed as potentially being multiple memories. The memory can, for example, be multiple memories within the same computer system. The memory can also be multiple memories distributed among multiple computer systems or computing devices.

[0097] As shown, computer system 900 also includes a video display unit 950, for example, such as a liquid crystal display (LCD), an organic light emitting diode (OLED), a flat panel display, a solid state display, or a cathode ray tube (CRT). Additionally, computer system 900 includes an input device 960, such as a keyboard / virtual keyboard or a touch input screen or voice input with voice recognition, and a cursor control device 970, such as a mouse or a touch input screen or a touchpad. Computer system 900 also optionally includes a disk drive unit 980, a signal generation device 990 (such as a speaker or a remote control), and / or a network interface device 940.

[0098] In one embodiment, as Figure 9 shown, disk drive unit 980 includes a computer-readable medium 982 in which one or more sets 984 of software instructions (software) are embedded. The set 984 of software instructions is read from the computer-readable medium 982 and executed by the processor 910. Further, when the processor 910 executes the software instructions 984, one or more steps of the methods and processes described herein are performed. In one embodiment, the software instructions 984 reside, in whole or in part, within the main memory 920, the static memory 930, and / or the processor 910 during execution by the computer system 900. Additionally, the computer-readable medium 982 can include the software instructions 984 or receive and execute the software instructions 984 in response to a propagated signal such that a device connected to the network 901 transmits voice, video, or data over the network 901. The software instructions 984 can be transmitted or received over the network 901 via the network interface device 940.

[0099] In one embodiment, a dedicated hardware implementation is constructed, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable logic array, and other hardware components, to implement one or more of the methods described herein. One or more embodiments described herein can be implemented using two or more specific interconnected hardware modules or devices having associated control that can communicate between and through the modules. Accordingly, the present disclosure encompasses software, firmware, and hardware implementations. Nothing in this application should be construed as implementing or being implemented using only software rather than hardware such as tangible non-transitory processors and / or memories.

[0100] In accordance with various embodiments of the present disclosure, the methods described herein can be implemented using a hardware computer system that executes software programs. Additionally, in example non-limiting embodiments, the implementation can include distributed processing, component / object distributed processing, and parallel processing. Virtual computer system processing can be implemented to perform one or more methods or functions as described herein, and the processor 205 described herein can be used to support a virtual processing environment.

[0101] Thus, device tracking and monitoring outside the imaging field enables the combination of robotic intelligence and X-ray imaging, and enables updating of image acquisition or image visualization based on tracking intravascular devices outside the field of view in X-ray imaging. The end of the guidewire outside the field of view can be tracked through information in the X-ray image and the robotic encoder to provide an overlay for the end of the guidewire on the extended image. The position of the coaxial intravascular device can be observed without moving the X-ray system or acquiring a new X-ray image, thereby improving safety and simplifying the workflow. Therefore, since the teachings herein allow tracking and visualization of the elongate device even outside the field of view, the safety of the procedure is improved; thus, dangerous manipulations and movements can be viewed and mitigated. Additionally, since additional X-ray imaging is not required in some cases to track and visualize the elongate device, X-ray radiation is reduced. Further, since unnecessary changes to the imaging setup are eliminated, the workflow is also improved.

[0102] Although device tracking and monitoring outside the imaging field of view have been described with reference to several exemplary embodiments, it should be understood that the words used are words of description and illustration, not of limitation. Changes can be made within the scope of the appended claims, as presently stated and modified, provided that the scope and spirit of device tracking and monitoring outside the imaging field of view are not departed from. Although device tracking and monitoring outside the imaging field of view have been described with reference to specific methods, materials, and embodiments, device tracking and monitoring outside the imaging field of view are not intended to be limited to the details disclosed; rather, device tracking and monitoring outside the imaging field of view extend to all functionally equivalent structures, methods, and uses, such as within the scope of the appended claims.

[0103] The illustrations of the embodiments described herein are intended to provide a general understanding of the structures of the various embodiments. These illustrations are not intended to be a complete description of all elements and features of the present disclosure described herein. After reviewing the present disclosure, many other embodiments will be apparent to those skilled in the art. Other embodiments may be utilized and other embodiments may be derived from the present disclosure, such that structural and logical substitutions and changes may be made without departing from the scope of the present disclosure. Additionally, these illustrations are merely representative and may not be drawn to scale. Some scales in the illustrations may be enlarged while other scales may be minimized. Accordingly, the present disclosure and the drawings should be regarded as illustrative rather than restrictive.

[0104] One or more of the embodiments disclosed herein may be referred to independently and / or jointly by the term "invention" solely for convenience, but this does not limit the scope of the present application to any particular invention or inventive concept. Additionally, while particular embodiments have been illustrated and described herein, it should be understood that any subsequent arrangement designed to achieve the same or similar purpose may replace the particular embodiments shown. The present disclosure is intended to cover any and all subsequent modifications or variations of the various embodiments. Combinations of the above embodiments, as well as other embodiments not specifically described herein, will be apparent to those skilled in the art upon review of the specification.

[0105] The abstract of the present disclosure provided complies with 37 C.F.R.§1.72(b), and it should be understood when submitting the abstract that the abstract is not for interpreting or limiting the scope or meaning of the claims. Additionally, in the foregoing detailed description, for the purpose of simplifying the present disclosure, various features may be combined together or described in a single embodiment. The present disclosure should not be construed as reflecting an intention that the claimed embodiments require more features than are expressly recited in each claim. On the contrary, as reflected in the following claims, inventive aspects may be directed to less than all of the features of any one of the disclosed embodiments. Accordingly, the following claims are incorporated into the detailed description, where each claim independently defines the claimed aspect.

[0106] The foregoing description of the disclosed embodiments is provided to enable any person skilled in the art to practice the concepts described in the present disclosure. For this reason, the subject matter disclosed above should be considered illustrative, not restrictive, and the claims are intended to cover all such modifications, enhancements, and other embodiments that fall within the true spirit and scope of the present disclosure. Accordingly, to the maximum extent permitted by law, the scope of the present disclosure will be determined by the broadest permissible interpretation of the following claims and their equivalents, and should not be limited to or restricted by the foregoing detailed description.

Claims

1. A system for tracking an elongate device driven by at least one motor within an anatomical structure, comprising: A controller configured to: Receive an image; Receive encoded data corresponding to the driving of the elongate device by the motor; Determine the position of the elongate device; Determine when at least a portion of the elongate device is outside the field of view in the image or is likely to leave the field of view in the image; Determine whether to create extended image data to show the position of the elongate device outside the field of view in the image; Determine at least one of a current position, a past position, or a predicted position of the elongate device outside the field of view in the image based on pixel data in the image and the encoded data; And Extend the image to show the position of the elongate device outside the field of view in the image in an extended image.

2. The system according to claim 1, further comprising: An X-ray machine that generates an image of the elongate device as a two-dimensional X-ray image, wherein the elongate device includes an intravascular device.

3. The system according to claim 1, wherein The controller is further configured to: Provide a display signal that is adapted to show the position of the elongate device outside the field of view in the image on a display by supplementing the image with additional image data that projects the position of the elongate device outside the field of view in the image.

4. The system according to claim 1, wherein The current position or the predicted position includes an estimated position of an end of the elongate device, and an image of the end of the elongate device is superimposed on the extended image at the position determined based on the pixel data and the encoded data.

5. The system according to claim 1, wherein The controller is further configured to: Update the acquisition of the image based on the encoded data and the pixel data to return the elongate device to the field of view in the image.

6. The system according to claim 1, wherein, The system determines to create the extended image data based on determining that the elongate device is outside the field of view in the image, and wherein the elongate device includes an intravascular device.

7. The system according to claim 1, wherein The controller is further configured to: Receive current image data and current encoded data, and generate and output a representation of the displacement of the elongate device in the current image data based on the encoded data.

8. The system according to claim 7, wherein, The controller is further configured to: Extend the image by performing at least one of warping, mosaicking, or stitching on at least one image, wherein the representation of the displacement is generated based on performing warping, mosaicking, or stitching on the at least one image.

9. The system according to claim 1, wherein, The controller is further configured to: Extend the image based on at least one of current image data of a current image and past image data of a past image.

10. A method for tracking an elongate device within an anatomical structure, comprising: Receiving an image of the elongate device at a system including a memory storing instructions and a processor executing the instructions; Receiving encoded data corresponding to the driving of the elongate device; Determining the position of the elongate device; Determining when at least a portion of the elongate device is outside the field of view in the image or is likely to leave the field of view in the image; Determine whether to create extended image data to show the position of the elongate device outside the field of view in the image; Determine the current position or predicted position of the elongate device outside the field of view in the image based on the pixel data and the encoded data in the image; And Extend the image to show the position of the elongate device outside the field of view in the image in the extended image.

11. The method according to claim 10, further comprising: Display the position of the elongate device outside the field of view in the image on a display by supplementing the image with additional image data that projects the position of the elongate device outside the field of view in the image.

12. The method according to claim 10, wherein The current position or the predicted position includes an estimated position of an end of the elongate device, and an image of the end of the elongate device is superimposed on the extended image at a position determined based on the pixel data and the encoded data.

13. The method according to claim 10, further comprising: Receive current image data and current encoded data, and generate and output a representation of the displacement of the elongate device in the current image data based on the encoded data.

14. The method according to claim 13, further comprising: Extend the image by performing at least one of warping, mosaicking, or stitching on at least one image, wherein the representation of the displacement is generated based on performing warping, mosaicking, or stitching on the at least one image.

15. A controller for tracking an elongate device in an anatomical structure, comprising: A memory that stores instructions; And A processor that runs the instructions, wherein when run by the processor, the instructions cause the system to: Receive an image including the elongate device; Receive encoded data corresponding to the actuation of the elongate device; Determine the position of the elongate device; Determine when at least a portion of the elongate device is outside the field of view in the image or is likely to leave the field of view in the image; Determine whether to create extended image data to show the position of the elongate device outside the field of view in the image; Determine the current position or predicted position of the elongate device outside the field of view in the image based on the pixel data and the encoded data in the image; and Extend the image to show the position of the elongate device outside the field of view in the image in the extended image.

Images