CONTROLLER FOR IMAGING AND CONTROLLING A ROBOTIC DEVICE USING AN IMAGING INTERFACE - Patent application
The controller system integrates imaging and robotic device control through an imaging interface, enabling a single user to seamlessly operate both devices, addressing the need for multiple clinicians and enhancing procedural efficiency in interventional procedures.
Patent Information
- Application Number
- JP2024527119
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2022-02-04
- Filing Date
- 2022-11-03
- Publication Date
- 2024-11-21
AI Technical Summary
In complex interventional procedures, the need arises for integrated control of both interventional and imaging devices to enable seamless operation by a single user, as current methods require multiple clinicians to coordinate the imaging and operation of robotic devices, leading to inefficiencies and the need for additional skilled personnel.
A controller system that integrates an imaging interface with a robotic device, allowing a single user to switch between imaging and control modes using actuators on the imaging interface device, such as an ultrasound probe, to control the robotic device's movement based on ultrasound image data, utilizing image processing and servo control loops to align the robotic device with the user's intended movements.
Enables a single user to operate both imaging and interventional devices independently, reducing the need for additional personnel and enhancing procedural efficiency by allowing intuitive control of robotic devices within a subject's anatomy.
Smart Images

Figure 2024543355000001_ABST
Abstract
Description
[Background technology]
[0001] Complex interventional procedures, such as peripheral vascular interventions, use ultrasound to provide imaging feedback in controlling the movement of an interventional device manipulated within a subject's (patient's) anatomy. Such procedures require at least two clinicians (users) to coordinate the imaging and operation of a robotic device. One user (e.g., a sonographer), typically under the direction of another user (e.g., a physician), manipulates the ultrasound probe to acquire images of the anatomy of interest, while the other user interprets the images and simultaneously manipulates the interventional device via a hand or robotic interface.
[0002] This is a common problem, for example, in endovascular interventions involving catheters and guidewires where interventional devices are manipulated using both hands and at access sites (such as the groin) that are often remote from the treatment site (such as the lower leg). Again, an additional skilled user is required to manipulate and hold the ultrasound probe to image the anatomical structures and the catheters and guidewires positioned within the anatomical structures. Similarly, additional skilled users are required for percutaneous and minimally invasive interventions such as ablation and atherectomy, where multiple devices must be controlled. Summary of the Invention [Problem to be solved by the invention]
[0003] Therefore, there is a need to enable integrated control of interventional and imaging devices such that a single user can seamlessly operate both interventional and imaging devices during an interventional procedure. [Means for solving the problem]
[0004] According to an aspect of the present disclosure, a controller for controlling a robotic device for insertion into an anatomical structure of a subject is provided, the controller including at least one processor and a non-transitory memory for storing machine-executable instructions that, when executed by the at least one processor, cause the at least one processor to receive ultrasound image data from an imaging interface, the ultrasound image data corresponding to an ultrasound image illustrating an anatomical structure and a portion of the robotic device within the anatomical structure, the imaging interface being operable to provide the ultrasound image data, detect movement of the imaging interface, preferably derived from content of the ultrasound image data, and output at least one command for controlling movement of the robotic device from a first position to a second position in response to the detected movement of the imaging interface.
[0005] According to another aspect of the present disclosure, a method for controlling a robotic device for insertion into an anatomical structure of a subject is provided, the method including receiving ultrasound image data from an imaging interface, the ultrasound image data corresponding to an ultrasound image illustrating the anatomical structure and a portion of the robotic device within the anatomical structure, the imaging interface being operable to provide the ultrasound image data, detecting movement of the imaging interface, preferably derived from content of the ultrasound image data, and outputting at least one command for controlling movement of the robotic device from a first position to a second position in response to the detected movement of the imaging interface.
[0006] According to another aspect of the present disclosure, a non-transitory computer-readable medium stores instructions for controlling a robotic device for insertion into an anatomical structure of a subject, the instructions, when executed by at least one processor, cause the at least one processor to receive ultrasound image data from an imaging interface, the ultrasound image data corresponding to an ultrasound image illustrating the anatomical structure and a portion of the robotic device within the anatomical structure, the imaging interface device being operable to provide the ultrasound image data, detect movement of the imaging interface device, preferably derived from content of the ultrasound image data, and output at least one command for controlling movement of the robotic device from a first position to a second position in response to the detected movement of the imaging interface. [Brief description of the drawings]
[0007] The example embodiments are best understood when the following detailed description is read in conjunction with the accompanying drawing figures. It is emphasized that the various features are not necessarily drawn to scale. In fact, dimensions have been arbitrarily increased or decreased for clarity of discussion. Where applicable and practical, like reference numerals refer to like elements.
[0008] [Figure 1] FIG. 1 is a simplified block diagram of a system for controlling imaging and a robotic device insertable into a subject's anatomy, according to a representative embodiment. [Figure 2A] FIG. 2A is a perspective view of an ultrasound probe with push button actuators for selecting imaging and control modes, according to a representative embodiment. [Figure 2B] FIG. 2B is a perspective view of an ultrasound probe with slider button actuators for selecting imaging and control modes, according to a representative embodiment. [Figure 2C]FIG. 2C is a perspective view of an ultrasound probe with a combination push button and slider button actuator for selecting imaging and control modes, according to a representative embodiment. [Diagram 3] FIG. 3 illustrates a perspective view of a translated imaging interface during imaging of an anatomical structure, according to a representative embodiment. [Figure 4] FIG. 4 illustrates a perspective view of an imaging interface rotated while imaging an anatomical structure, according to a representative embodiment. [Diagram 5] FIG. 5 is a simplified block diagram of a system controller for controlling imaging and a robotic device insertable into a subject's anatomy, according to a representative embodiment. [Figure 6] FIG. 6 is a flow diagram of a method for controlling imaging and a robot insertable into a subject's anatomy, according to a representative embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0009] In the following detailed description, for purposes of explanation and not limitation, representative embodiments disclosing specific details are described to provide a thorough understanding of the embodiments according to the present teachings. Descriptions of known systems, devices, materials, methods of operation, and methods of manufacture may be omitted so as not to obscure the description of the representative embodiments. However, systems, devices, materials, and methods within the purview of one of ordinary skill in the art are within the scope of the present teachings and may be used in accordance with the representative embodiments. It is to be understood that the terms used herein are for the purpose of describing particular embodiments only and are not intended to be limiting. Defined terms are to be given the technical and scientific meaning of the defined terms as commonly understood and accepted in the art of the present teachings.
[0010] In this specification, terms such as "first," "second," and "third" are used to describe various elements or components, but it should be understood that these elements or components are not limited by these terms. These terms are used only to distinguish one element or component from another. Thus, a first element or component discussed below may be referred to as a second element or component without departing from the teachings of the inventive concept.
[0011] The terms used herein are for the purpose of describing particular embodiments only and are not intended to be limiting. As used in this specification and the appended claims, singular terms are intended to include both the singular and the plural, unless the context clearly dictates otherwise. Furthermore, the term "comprises" and / or similar terms specify the presence of stated features, elements, and / or components, but do not exclude the presence or addition of one or more other features, elements, components, and / or groups thereof. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.
[0012] Unless otherwise noted, when an element or component is said to be "connected," "coupled," or "adjacent" to another element or component, it is understood that the element or component may be directly connected or coupled to the other element or component, or there may be intervening elements or components present. That is, these and similar terms encompass the cases where one or more intermediate elements or components may be used to connect the two elements or components. However, when an element or component is said to be "directly connected" to another element or component, this only encompasses the cases where the two elements or components are connected to each other without any intermediate or intervening elements or components.
[0013] The present disclosure is intended to provide one or more of the advantages specifically set forth below through one or more of its various aspects, embodiments, and / or specific features, subcomponents. For purposes of explanation and not limitation, example embodiments disclosing specific details are described to provide a thorough understanding of the embodiments according to the present teachings. However, other embodiments consistent with the present disclosure that depart from the specific details disclosed herein remain within the scope of the appended claims. Furthermore, descriptions of well-known devices and methods may be omitted so as not to obscure the description of the example embodiments. Such methods and devices are within the scope of the present disclosure.
[0014] Various embodiments described herein streamline the control of imaging and interventional devices using an imaging interface device operable by a single user, that is, the imaging interface device allows a single user to directly control the imaging modality during an interventional procedure while controlling parameters of the imaging modality and other devices in the system, such as robotic devices within a subject, by operating the imaging interface device. The imaging interface device reduces redundant communication, is intuitive, and minimizes or potentially eliminates the need for assisting imaging modality technicians.
[0015] Typically, the system controller controls at least one control parameter of a robotic device having at least one degree of freedom (DOF) of control in response to the movement of an imaging interface device (e.g., an ultrasound probe) held by a user. Manipulation of the imaging interface device allows commands of the robotic controller to follow the movement of the imaging interface device. This movement is derived from the content of the image provided through the imaging interface device. Thus, the imaging interface device itself is effectively utilized as a joystick, allowing the user to control the robotic device and any interventional device that can be manipulated through the robotic device (e.g., an intravascular device that is in the subject and tracked in the image (e.g., ultrasound volume)). Thus, the user can image the anatomical structure of interest, and can enter a control (joystick) mode through manipulation of an actuator of the imaging interface device when movement of the robotic device (e.g., advancing a guidewire in a blood vessel) is required. Alternatively or additionally, the system controller can also control at least one control parameter of the imaging modality itself, such as, for example, frequency, pulse wave, color, and power, in response to the movement of the imaging interface device.
[0016] With respect to controlling the robotic device, the system controller records the position of a trackable feature of the robotic device (such as the distal tip) within the imaging volume and creates a reference plane relative to the imaging interface device (and essentially the imaging volume) that is perpendicular to the longitudinal axis of the robotic device. For example, as the user translates the imaging interface device along the orientation of the robotic device, the distance between the trackable feature of the robotic device and the reference plane changes. The system controller uses this distance as an error input to servo the robotic device (e.g., using a PID servo control loop) relative to the reference plane so that the robotic device appears to move with the image. The servo control movement continues until a tolerance error is met. As the user rotates the imaging interface device, during translation or non-translation, the amount and direction of the rotation is detected by direct image processing or position data of the imaging interface device, for example, using an on-board inertial measurement unit (IMU), and the system controller rotates the robotic device in a corresponding manner. The user can then exit the control mode and return to the imaging mode when the actuator of the imaging interface device is deactivated.
[0017] FIG. 1 is a simplified block diagram of a system for controlling imaging and a robotic device insertable into a subject's anatomy, according to a representative embodiment.
[0018] 1, system 100 controls the placement and manipulation of a robotic device 125 inserted into an anatomical structure 107 of a subject (patient) 105 during an interventional procedure under the direction of a user (clinician) 140. For example, anatomical structure 107 is a vasculature (such as an artery), and robotic device 125 can include or hold, for example, a catheter and / or a guidewire that is inserted into the vasculature. Robotic device 125 can include or hold a variety of other interventional devices, such as surgical devices (loop cutters, forceps, etc.), ablation devices, imaging and sensing devices, etc., without departing from the scope of the present teachings.
[0019] In the illustrated embodiment, the system 100 includes an ultrasound system 110 and a system controller 120, which includes a robotic controller 122 and an image processing / device tracking system 124. The ultrasound system 110 receives ultrasound image data of the subject 105 via an imaging interface device 115. The imaging interface device 115 is, for example, a handheld ultrasound probe that is manipulated by a user 140 while viewing a graphical user interface (GUI) 145 to obtain a desired view of the anatomical structure 107 and / or the robotic device 125 while within the anatomical structure 107. An example of the imaging interface device 115 as an ultrasound probe is shown in Figures 2A, 2B, and 2C, described below. In various embodiments, the imaging interface device 115 may be realized by any other type of compatible interface, such as, for example, a foot pedal, a wheel, a knob, or the like, in addition to or instead of an ultrasound probe.
[0020] When the imaging interface device 115 is implemented as an ultrasound probe, it may include multiple ultrasound transducers in a transducer array that emits sound waves into the subject 105 and detector elements in a detector array that receive corresponding echoes under the control of the ultrasound system 110, thereby providing ultrasound image data including images of the anatomical structures and the robotic device 125 within the anatomical structures. The ultrasound system 110 provides the ultrasound image data to the GUI 145 for review by the user 140.
[0021] The system controller 120 can operate in two selectable modes, which can be selected by the user 140 using the imaging interface device 115, for example, during an interventional procedure. As described in more detail below, the two selectable modes include an imaging (first) mode and a control (second) mode. In the imaging mode, the system controller 120 receives ultrasound image data of the anatomical structure 107 and the robotic device 125 within the anatomical structure 106 from the imaging interface device 115. In the control mode, the system controller 120 receives movement information from the imaging interface device 115 for controlling the movement of the robotic device 125 within the anatomical structure 107. The movement information can include translational movement of the imaging interface device 115 across the surface of the subject 105 in a direction in which the user wants to guide the robotic device 125, and rotational movement of the imaging interface device 115 on the surface of the subject 105 at an angle in which the user wants to rotate the robotic device 125. Imaging interface device 115 is configured to allow a user to easily select between imaging and control modes when operating imaging interface device 115, as described below.
[0022] The robot controller 122 controls the movement of the robotic device 125 within the anatomy 107 during an intervention procedure. The movement of the robotic device 125 includes translational movement to move the robotic device 125 in and out of the anatomy 107, rotational movement to rotate the robotic device 125 within the anatomy 106, for example according to the available degrees of freedom (DOF) of the robotic device 125, and articulation movement to bend and articulate the robotic device 125. The robotic device 125 may also control the movement and operation of active components such as lasers. As is well known in the art, the robot controller 122 may control the movement of the robotic device 125 through the operation of servo motors (servos) integrated into the robotic device 125. This movement may therefore be referred to as servoing the robotic device 125. The robot controller 122 is implemented, for example, as a proportional-integral-derivative (PID) controller.
[0023] The robot controller 122 and the robotic device 125 may have autonomous capabilities, in which case the user 140 may control a subset of the degrees of freedom of actuation. For example, the user 140 may control the articulation and directional movement of the robotic device 125 by manipulating the imaging interface device 115, while the robot controller 122 moves the robotic device 125 forward (translate) at a constant speed. Similarly, the user 140 may control the amount of advancement or retreat of the robotic device 125 within the anatomy 107 using translation input, while the robot controller 122 simultaneously centers the robotic device 125 in the cross-section of the anatomy 107 (e.g., in a robotic device 125 that may be magnetically actuated).
[0024] The image processing / device tracking system 124 receives ultrasound image data from the ultrasound system 110, determines movement of the imaging interface device 115 from the ultrasound image data relative to tracked anatomical features (such as blemishes, blood vessels, etc.) in the image data, determines an image of the robotic device 125 from the ultrasound image data, and calculates movement of the robotic device 125 in response to movement of the imaging interface device 115.
[0025] As previously mentioned, the imaging interface device 115 operates in an imaging mode and a control mode based on a selection by the user during operation of the imaging interface. In the imaging mode, the imaging interface device 115 outputs ultrasound image data of the anatomical structure 107 and the robotic device 125 within the anatomical structure 107 to the ultrasound system 110. The ultrasound image data is acquired in a known manner, for example by the user 140 manually manipulating the pressure and direction of the imaging interface device 115 (such as an ultrasound probe) on the surface of the subject 105 while viewing a corresponding ultrasound image in the GUI 145 to capture a desired view. In an alternative configuration, the imaging interface device 115 can be operated automatically or partially automatically using robotic controls without departing from the scope of the present teachings. In the control mode, the imaging interface device 115 controls the movement of the robotic device 125; that is, translating the imaging interface device 115 across the surface of the subject 105 results in a corresponding translation of the robotic device 125. Similarly, rotational movement of the imaging interface device 115 produces corresponding rotational movement of the robotic device 125. Specifically with respect to rotational movement, the imaging interface device 115, in the control mode, effectively functions as a joystick that the user 140 can manipulate.
[0026] In particular, the user 140 can use the imaging interface device 115 to conveniently select between imaging and control modes while holding the imaging interface device 115 during an interventional procedure. For example, if the imaging interface device 115 is an ultrasound probe, the imaging interface device 115 includes a probe handle that the user 140 can grasp to perform both ultrasound imaging and robotic device control functions without requiring additional personnel to perform one or the other of the functions. The imaging interface device 115 includes actuators, such as push buttons or slider buttons, that are manually operated by the user 140 while holding the imaging interface device 115. Again, if the imaging interface device 115 is an ultrasound probe, integrating the actuators into the probe handle allows the user to easily access and operate the actuators while holding the probe handle to selectively activate the imaging and control modes.
[0027] In an embodiment, a virtual probe representing the imaging interface device 115 is rendered on the GUI 145 and / or an augmented or virtual reality display (not shown). The GUI 145 allows multiple simultaneous inputs and can display different scales and values for imaging and control mode selection and other functions. In another embodiment, the imaging interface device 115 is controlled by the user in a telerobotic manner rather than handheld. In this case, the user 140 can select the imaging and control modes by remote control.
[0028] 2A, 2B, and 2C show perspective views of an ultrasound probe with integrated actuators for selecting imaging and control modes, according to a representative embodiment.
[0029] 2A, the ultrasound probe 215A includes a push button 216 as an integrated actuator for selecting between imaging and control modes. The ultrasound probe 215A includes a probe handle 221 for a user 140 to manipulate the ultrasound probe 215A, and a transducer / detector array 222 for transmitting sound waves into the subject 105 and receiving corresponding echoes to provide ultrasound image data. The push button 216 is configured such that when the probe handle 221 is held in a hand, the user 140 can simply press the push button 216 to switch between imaging and control modes with the same hand while holding the probe handle 221. The push button 216 can be actuated using one or more fingers, a thumb, or other parts of the user's hand.
[0030] In an embodiment, the ultrasound probe 215A enters imaging mode when the push button 216 is pressed and enters control mode when the push button 216 is released, or vice versa. Alternatively, the push button 216 may have a single position and simply toggle between imaging and control modes each time the push button 216 is pressed. In this configuration, the push button 216 may simply be a touch sensitive button rather than a physically depressible push button.
[0031] 2B, the ultrasound probe 215B includes a slider button 217 on the probe handle 221 as an integrated actuator for selecting between imaging and control modes. The slider button 217 is configured such that, when gripping the probe handle 221, a user can easily slide the slider button 217 between an up position UP and a down position LP to switch between imaging and control modes with the same hand while gripping the probe handle 221. In an embodiment, the ultrasound probe 215A enters the imaging mode when the slider button 217 is in the up position UP and enters the control mode when the slider button 217 is in the down position LP, or vice versa. The slider button 217 can be actuated by a sliding movement or a scrolling movement like a scroll wheel using one or more fingers, thumb, or other parts of the user's hand.
[0032] 2C, the ultrasound probe 215C includes a combination controller 218 on the probe handle 221 as an integrated actuator for selecting imaging and control modes. In the illustrated embodiment, the combination controller includes an upper push button 216U and a lower push button 216L that function as described above with reference to the push button 216, and a slider button 217, such that a user can simply operate any or all of the upper push button 216U, the lower push button 216L, and the slider button 217 as described above when gripping the probe handle 221. The presence of both the upper push button 216U and the lower push button 216L, alone or simultaneously, allows the selection of various parameters that are controlled by the movement of the ultrasound probe 215C. Examples include translation of the robotic device, rotation of the robotic device, ultrasound imaging parameters, and actuation of a treatment device such as laser output in laser atherectomy. The combination controller 218 can be actuated using one or more fingers, thumb, or other parts of the user's hand.
[0033] FIG. 3 illustrates a perspective view of a translated imaging interface during imaging of an anatomical structure, according to a representative embodiment.
[0034] 3, an imaging interface device 115 (e.g., ultrasound probe 215A) is shown imaging the anatomical structure 107 and the robotic device 125 by emitting acoustic waves 300 into the subject 105 and receiving corresponding echo signals (not shown). The imaging interface device 115 defines an image plane B in ultrasound space that for purposes of illustration extends from the transducer / detector array 222 and includes the longitudinal axis of the robotic device 125 and / or a longitudinal section of the anatomical structure. That is, the portion of the robotic device 125 that is within the anatomical structure 107 along the longitudinal axis is in the image plane B in both the imaging and control modes of the imaging interface device 115. In the case of a matrix ultrasound probe, such an imaging plane can be constructed using beamforming techniques or slicing of a 3D volumetric image, both of which are known in the art. In some cases, the image plane B may be elsewhere relative to the imaging interface device 115. For example, it may be parallel to the longitudinal axis 330 and extend from a line on the transducer / detector array, or may be specified by the user 140 without departing from the scope of the present teachings.
[0035] Further, whenever the push button 126 is actuated to, for example, cause the imaging interface device 115 to enter a control mode, a virtual reference plane R is defined in ultrasound space relative to the transducer / detector array 222. The reference plane R intersects with the image plane B at a first location of the trackable feature of the robotic device 125 being tracked (e.g., at an angle of about 90 degrees). The reference plane R is substantially perpendicular to the longitudinal axis of the robotic device 125. In the illustrated embodiment, the trackable feature of the robotic device 125 being tracked is the distal tip 126 of the robotic device 125. However, the trackable feature is not limited to the distal tip 126, but may be another predetermined section of the robotic device 125, a marker on the robotic device 125, a curved portion of the robotic device 125, or a predetermined distance from one of the distal tip 126, another predetermined section, marker, or curved portion, etc.
[0036] The control mode is selected to translate a trackable feature of the robotic device from a first position to a second position. In the control mode, the translation of the imaging interface device 115 is detected by the image processing / device tracking system 124 as the imaging interface device 115 is translated on the surface of the subject 105 relative to the anatomical structure 107. The direction, distance, and / or speed of the translation of the imaging interface may be supplemented by data provided by an IMU of the imaging interface device 115. Other compatible translation detection technologies, such as optical or electromagnetic tracking of the imaging interface and subject, may be incorporated without departing from the scope of the present teachings.
[0037] Because the reference plane R is defined relative to the imaging interface device 115, moving the imaging interface device 115 moves the reference plane R relative to the anatomy. Specifically, when the imaging interface device 115 translates in the x-direction, as indicated by arrow 331, the reference plane R implicitly follows this translation, moving in the same direction and the same distance along the image plane B. In an embodiment, the translation of the imaging interface device 115 controls the force input applied to the robotic device 125 towards the reference plane R.
[0038] Thus, to manipulate the imaging interface device 115 to move the robotic device 125, the user 140 first presses the push button 216 to enter the control mode. In the control mode, the image processing / device tracking system 124 locates a trackable feature (e.g., distal tip 126) of the robotic device 125 in an ultrasound image provided by the ultrasound system 110 in the imaging mode. The image processing / device tracking system 124 can locate the robotic device 125, its distal tip 126, and the anatomical structure 107, for example, using artificial intelligence (AI) segmentation. In this case, an image plane B is created from the anatomical structure 107. The image processing / device tracking system 124 creates the image plane B extending from the imaging interface device 115, and further creates a reference plane R that intersects the image plane B at a first location of the trackable feature. This first reference plane R provides a reference datum for the imaging interface device 115 based on the first location of the trackable feature in the image plane B. The fiducial datum is displayed in the GUI 145 along with the ultrasound images of the anatomical structure 107 and the robotic device 125 .
[0039] The user 140 can then reactivate (or keep active) the push button 216 to enter a control mode and control the movement of the robotic device 125 using the imaging interface device 115 as a joystick or the like. In FIG. 3, the user 140 translates the imaging interface device 115 from a first location to a new second location on the surface of the subject 105 in the x-direction indicated by the arrow 331. Correspondingly, information from the image processing / device tracking system 124 is used by the robot controller 122 to set a new image plane B in the ultrasound system 110 in conjunction with the movement of the imaging interface device 115 to the second location. The image plane B is defined such that the tip and distal longitudinal section of the robotic device 125 are included in the image plane B (if it has been accidentally moved from the original image plane B). The imaging interface device 115 continues to collect images within a reference plane R defined relative to the probe of the imaging interface device 115 so as to image the same location relative to the imaging interface device 115, regardless of image content or tracking information.
[0040] The robot controller 122 receives information from the imaging process / device tracking system 124 identifying the position of the robotic device 125 within the ultrasound image. The robot controller 122 determines the position of the trackable feature of the robotic device 125 within image plane B. The robot controller 122 determines the distance in image plane B between the current trackable feature of the robotic device 125 and the intersection point of the reference plane R and the image plane B, and servos the robotic device 125 to direct the trackable feature of the robotic device 125 toward this intersection point. The robot controller 122 then receives additional information from the imaging process / device tracking system 124 identifying the next position of the robotic device 125 within the ultrasound image, and repeats the subsequent steps in a loop until a threshold is met.
[0041] In an embodiment, the robot controller 122 uses the determined distance as a position error and translates the robot device 125 towards the location of the reference plane R by correcting the position error. That is, as the user 140 translates the imaging interface device 115 in the x-direction, the distance of the trackable feature of the robot device 125 from the first position relative to the anatomical structure to the reference plane R increases. Thus, the robot controller 122 uses this distance as an error input to servo the robot device 125 towards the reference plane R until the associated second position is aligned with the reference plane R, for example, using a PID servo control loop. That is, the robot controller 122 continues the servo control movement until the tolerance error is satisfied. The imaging interface device 115 continues to provide an ultrasound image during the translation. In this case, the movement of the robot device 125 is captured and displayed in real time on the GUI 145. Thus, the robot device 125 moving through the anatomical structure 107 appears to move along with the ultrasound image.
[0042] The user 140 exits the control mode and re-enters the imaging mode when the push button 216 becomes inactive (e.g., released or pressed again). The user 140 then again manipulates the imaging interface device 115 at a second location to acquire ultrasound images of the anatomical structure 107 and the robotic device 125, which are displayed in the GUI 145, assuming imaging is the default mode. This mode is repeated whenever the push button 216 is not pressed. Alternatively, pressing the button may function as a toggle, in which case the push button 216 must be pressed again to exit the mode.
[0043] FIG. 4 shows a perspective view of an imaging interface rotated during imaging of an anatomical structure, according to a representative embodiment.
[0044] 4, for example, imaging interface device 115 (e.g., ultrasound probe 215A) is shown emitting sound waves 300 into subject 105 and receiving corresponding echo signals (not shown) to image anatomical structure 107 and robotic device 125. For rotation of imaging interface device 115, user 140 first presses push button 216, which in response places imaging interface device 115 in control mode. In control mode, image processing / device tracking system 124 locates trackable features (e.g., distal tip 126) of robotic device 125 in ultrasound images provided by ultrasound system 110 in imaging mode. Image processing / device tracking system 124 can locate robotic device 125, its distal tip 126, and anatomical structure 107, for example, using AI segmentation. Image processing / device tracking system 124 creates an image plane B extending from imaging interface device 115 and further creates a reference plane R that intersects longitudinal axis 330 at an initial angle (e.g., 90 degrees relative to plane B). Plane R is substantially attached to one or more tracked anatomical features. The orientation of reference plane R provides a first reference image for imaging interface device 115 and image plane B, which is displayed in GUI 145.
[0045] When the user 140 rotates the imaging interface device 115 while holding down the push button 216, the imaging interface device 115 can be used, for example, as a joystick to control the rotation of the robotic device 125, as indicated by arrow 332. In some embodiments, successive presses of the push button 216 can be used for clutching in and clutching out. In the illustrated example, the purpose of rotating the imaging interface device 115 is to control the directional movement of the robotic device 125 (e.g., rotation or articulation about the longitudinal axis of the distal tip 126), as indicated by arrow 333 in FIG. 4, to steer the robotic device 125 towards the branch 107′ of the anatomical structure 107. The magnitude of the device rotation or articulation is proportional to the rotational deviation of the probe relative to the reference frame R.
[0046] Alternatively, in a similar control manner as above, during a particular control mode, the user 140 rotates the imaging interface device 115 to define an orientation of the reference plane R that is substantially fixed to an anatomical feature in the image. When the control mode is terminated, the robot controller 122 uses a distance metric (from the image processing) from the distal tip 126 of the robot device 125 to the reference plane R to actuate the robot device 125 to move (e.g., rotate or articulate) toward the location of the reference plane R relative to the anatomical structure. Meanwhile, the robot controller 122 receives information from the imaging processing / device tracking system 124 during or after the rotation operation is completed that identifies the new orientation reference plane R and the corresponding rotation angle. The robot controller 122 servos the robot device 125 to move (e.g., articulate) the rotation angle relative to the image plane B. For example, the distal tip 126 of the robot device 125 is steered from the main part of the anatomical structure 107 into the branch 107'. The imaging interface device 115 continues to provide ultrasound images during translation so that the movements of the robotic device 125 along with the rotation of the ultrasound probe 215A are captured and displayed in the GUI 145.
[0047] The GUI 145 can display a basic 3D volume, or in the case of 2D multi-plane imaging, the GUI 145 can display image plane B in a static position and a reference plane R dynamically relative to plane B in a 3D perspective, as shown in Figure 4. That is, this is essentially beam steering of image plane B, so the GUI 145 display can remain in the target region.
[0048] The amount and direction of rotation can be detected by direct image observation by an image processing / device tracking system 124 that performs image processing such as speckle tracking. The amount and direction of rotation can also be detected or supplemented from data provided by an IMU of the imaging interface device 115. The amount and direction of rotation relative to the first orientation of the reference plane R as determined by the image processing / device tracking system 124 is input to the robot controller 122. Other compatible rotation detection techniques may be incorporated without departing from the scope of the present teachings.
[0049] The robot controller 122 may also apply one or more boundary conditions that limit translational or rotational movements linked to physical attributes of the anatomical structure 107, such as a blood vessel wall. For example, the one or more boundary conditions may be defined by an anatomical model or a preoperative image of the anatomical structure. In this case, the physical boundaries are identified automatically, for example, using AI segmentation of ultrasound images, or manually through the user 140's interaction with the anatomical model or preoperative image via the GUI 145. The boundary conditions are then implemented to avoid contact between the robotic device 125 and the anatomical structure 107. For example, the boundary conditions stop the movement of the robotic device 125 associated with the translation or rotation of the imaging interface device 115 when a tracking feature or other part of the robotic device 125 encounters or is about to encounter an inner wall of the anatomical structure 107.
[0050] During rotation, it is desirable for at least a portion of the ultrasound image of the robotic device 125 to appear stationary from the perspective of the user 140, either in a three-dimensional volume or a two-dimensional plane, relative to the original reference image in the display on the GUI 145. This provides a stable visual reference for rotation and for the region of interest in the ultrasound image.
[0051] For three-dimensional volumes, the image processing / device tracking system 124 maintains the static ultrasound image orientation by changing the viewpoint of the three-dimensional model and adjusting the virtual cross-sectional plane during rotation.
[0052] For two-dimensional images (x-plane), image acquisition by the imaging interface device 115 can be controlled to vary with rotation. For example, when an IMU is used, the two-dimensional image plane B is virtually steered by electronic beam steering about the longitudinal axis 330 in the same and opposite direction of the corresponding rotation of the imaging interface device 115 measured by the IMU. For example, when the IMU detects a 10 degree counterclockwise rotation, the image processing / device tracking system 124 rotates the image plane B of the imaging interface device 115 10 degrees clockwise. This keeps the image plane B in the same orientation relative to the anatomical structure 107 during the rotation. Alternatively, the image plane B can be fixed to the location of the robotic device 125 and / or anatomical feature in the image while the imaging interface device 115 is being manipulated, with the constraint that the image plane B rotates about the longitudinal axis 330 and intersects the transducer elements. The image can then be formed using beamforming techniques.
[0053] Additionally, for two-dimensional images, any x-plane can be created by selectively activating and deactivating sets of different ultrasound transducers / detector elements in the transducer / detector array 222 by controlling the imaging interface device 115. Thus, as the imaging interface device 115 is rotated, the ultrasound system 110 activates the transducers and / or detector elements of the imaging interface device 115 that are aligned with the initial position of image plane B and deactivates the remaining transducers and / or detector elements to maintain the same orientation of image plane B, such that image plane B appears static relative to the anatomy 107.
[0054] The user 140 exits the control mode and re-enters the imaging mode when the push button 216 becomes inactive (e.g., released or pressed again). The user 140 then manipulates the imaging interface device 115 again to acquire ultrasound images of the anatomical structure 107 and the robotic device 125 from the new rotated orientation, which are displayed in the GUI 145. In an embodiment, the imaging interface device 115 continues to provide ultrasound images during translation, regardless of the position of the push button 216, such that the movement of the robotic device 125 is captured and displayed in the GUI 145 along with the rotation of the ultrasound probe 215A.
[0055] Thus, the reference plane R may be used differently in different rotation modes. In one rotation mode, the reference plane R is defined relative to any anatomical structure of the subject 105 for the purpose of rotation tracking, rotation control articulation, or other robotic or imaging parameters (in a continuous manner). In this scenario, the reference plane R is just a reference and is replaced only by the IMU. In another rotation mode, the rotation of the imaging interface device 115 is used to position the reference plane R by rotation (e.g., around the distal tip 126 of the robotic device 125) in a continuous manner. Once positioned relative to the anatomical structure of the subject 105, the new reference plane R can be used as a reference datum (goes with the anatomical structure) for subsequent servo loops controlled by the robotic controller 122. For example, the robotic controller 122 can articulate the robotic device 125 toward the new reference plane R. The image plane B in both rotation modes is generally used for user feedback. That is, it is not necessary to provide a representation of the image plane B, in which case the image plane B rotates in the opposite direction to the reference plane R. Alternatively, image plane B remains in the same plane relative to the anatomy 107 or robotic device 125 by direct tracking.
[0056] In addition to, or instead of, directional movement of the distal tip 126, rotational input using the imaging interface device 115 can be detected and used to control other parameters associated with the imaging interface device 115 or the robotic device 125. For example, rotation of the imaging interface device 115 can be used to control rotation about the longitudinal axis of the robotic device 125. That is, the imaging processing / device tracking system 124 interprets rotation of the imaging interface device 115 about the longitudinal axis 330 as a corresponding rotation of the robotic device 125 about its longitudinal axis and matches the angle (amount) and direction of rotation.
[0057] Rotation of the imaging interface device 115 can also be used to control various operations of the robotic device 125. For example, if the robotic device 125 includes or controls a laser device used for ablation, atherectomy, or the like, the imaging process / device tracking system 124 interprets clockwise rotation of the imaging interface device 115 as an increase in laser power and counterclockwise rotation as a decrease in laser power, and vice versa. The angle of rotation (clockwise or counterclockwise) is interpreted as proportional to the magnitude of the corresponding increase / decrease in laser power. Alternatively, translation of the imaging interface device 115 may control the desired speed setting of the laser when the imaging interface device 115 is not rotating. If the robotic device 125 includes or controls a cutting tool, the imaging process / device tracking system 124 interprets clockwise rotation of the imaging interface device 115 as an increase in cutting speed or mechanical vibration, and counterclockwise rotation as a decrease in cutting speed or mechanical vibration, and vice versa. The angle of rotation is interpreted as proportional to the magnitude of the corresponding increase / decrease in cutting speed or mechanical vibration. If the robotic device 125 includes or controls a balloon catheter, the imaging processing / device tracking system 124 interprets clockwise rotation of the imaging interface device 115 as an increase in the balloon inflation rate and counterclockwise rotation as a decrease in the balloon inflation rate, and vice versa.
[0058] With respect to atherectomy in particular, the position of the imaging interface device 115 controls the orientation of the atherectomy cutter / laser that faces laterally of the robotic device 125 within the anatomy 107. This can be done, for example, by positioning the imaging interface device 115 transversely relative to the leg of the subject 105. The robot controller 122 uses this orientation of the imaging interface device 115 to actuate the robotic device 125 to rotate / position the atherectomy body or cutting direction to align with the imaging interface device 115 as it moves across the surface of the leg, thus pointing the cutter toward the imaging interface device 115.
[0059] Rotation of the imaging interface device 115 can also be used to control the displacement of the secondary device relative to the robotic device 125. For example, the robotic device 125 may be a catheter and the secondary device may be a guidewire within the catheter. In this case, the imaging processing / device tracking system 124 interprets clockwise rotation of the imaging interface device 115 as an extension of the secondary device and counterclockwise rotation as a retraction of the secondary device, and vice versa. The angle of rotation is interpreted as being proportional to the corresponding extension / retraction distance. Of course, the detected rotation can be used to adjust other parameters associated with the imaging interface device 115 and / or the robotic device 125 without departing from the scope of the present teachings.
[0060] For example, rotation of the imaging interface device 115 can be used to control the operation of the ultrasound system 110, the imaging interface device 115 itself, and / or the GUI 145. For example, clockwise and counterclockwise rotation can be used to adjust image acquisition settings of the ultrasound system 110 or the imaging interface device 115, such as frequency, pulse wave, color, and power. Similarly, clockwise and counterclockwise rotation can be used to adjust display settings of the GUI 145. For example, rotation can be used to control the contrast of an image as it is displayed in the GUI 145. Or, in an image review using the GUI 145, rotation can be used to control sequential scrolling through acquired ultrasound images.
[0061] In embodiments, external tracking of the imaging interface device 115 and the robotic device 125 can be used to provide feedback, particularly when the robotic device 125 is not within the field of view of the ultrasound image or volume. For example, the imaging interface device 115 and / or the robotic device 125 can be provided with one or more sensors and their respective positions can be determined using electromagnetic (EM) tracking, regardless of whether the robotic device 125 is within the field of view of the imaging interface device 115. Other examples of external tracking include optical tracking, acoustic tracking, electromagnetic tracking, ultrasonic sensing device tracking (e.g., Onvision™ Needle Tip Tracking), and IMU-based tracking.
[0062] The imaging interface device 115 may be implemented using imaging modalities other than ultrasound without departing from the scope of the present teachings. For example, the imaging interface device 115 may provide single photon emission computed tomography (SPECT) imaging, x-ray imaging, or endoscopic / laparoscopic imaging. In this case, the ultrasound system 110 would be replaced with an imaging system compatible with the alternative imaging modality, as would be apparent to one of ordinary skill in the art.
[0063] FIG. 5 is a simplified block diagram of a system controller for controlling imaging and a robotic device insertable into a subject's anatomy, according to a representative embodiment.
[0064] 5, system controller 120 includes software instructions executable to implement any of the methods or computer-based functions disclosed herein. System controller 120 can operate as a stand-alone device or can be connected to other computer systems and peripheral devices, for example, using network 501.
[0065] The system controller 120 includes a processor 510 that executes software instructions to perform the functions described in various embodiments herein. The processor 510 is representative of one or more processing devices that execute software instructions to perform the functions described in various embodiments herein. The processor 510 may be implemented using any combination of hardware, software, firmware, hardwired logic circuitry, or combinations thereof, such as a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a general purpose computer, a central processing unit, a computer processor, a microprocessor, a microcontroller, a microcomputer, a state machine, a programmable logic device, a processor chip, a digital signal processor (DSP), a graphics processing unit (GPU), or a combination thereof. Additionally, any processor described herein may include multiple processors, parallel processors, or both. Multiple processors may be included or combined in a single device or multiple devices.
[0066] The term "processor" as used herein encompasses an electronic component capable of executing a program or machine-executable instructions. References to a computing device that includes a "processor" should be interpreted to include multiple processors or processing cores, such as in a multi-core processor. A processor may also refer to a collection of processors within a single computer system or a collection of processors distributed across multiple computer systems. The term "computing device" should also be interpreted to include a collection or network of computing devices, each of which includes one or more processors. A program comprises software instructions that are executed by one or more processors, either within the same computing device or distributed among multiple computing devices.
[0067] The system controller 120 further includes a main memory 520 and a static memory 530, which communicate with each other and with the processor 510 via a bus 505. A memory as described herein is a tangible storage medium for storing data and executable software instructions, and is non-transient while the software instructions are stored therein. As used herein, the term "non-transient" is to be interpreted as a property of a state that persists for a period of time, rather than a permanent property of a state. The term "non-transient" specifically negates momentary properties, such as carrier waves or signals or other types of properties that exist only temporarily at any time and place.
[0068] The main memory 520 and the static memory 530 are computer-readable media from which a computer (such as the processor 510) can read data and executable software instructions. Each of the main memory 520 and the static memory 530 may be implemented as one or more of a random access memory (RAM), a read-only memory (ROM), a flash memory, an electrically programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a register, a hard disk, a removable disk, a tape, a compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a floppy disk, a Blu-ray disk, or any other form of storage medium known in the art. The memory may be volatile or non-volatile, secure and / or encrypted, non-secure and / or non-encrypted. The main memory 520 and static memory 530 can store various types of information, such as software algorithms, AI models including RNNs and other neural network-based models, and computer programs, all of which are executable by the processor 510.
[0069] "Memory" is an example of a computer-readable storage medium. Computer memory is any memory that is directly accessible to a processor. Examples of computer memory include, but are not limited to, RAM memory, registers, and register files. References to "computer memory" or "memory" should be interpreted as referring to multiple memories. For example, a memory may be multiple memories within the same computer system. A memory may also be multiple memories distributed among multiple computer systems or devices.
[0070] As shown, the system controller 120 further includes a display 550, such as a liquid crystal display (LCD), an organic light emitting diode (OLED), a flat panel display, a solid state display, or a cathode ray tube (CRT). The GUI 145 can be incorporated into the display 550. The system controller 120 further includes an input device 560, such as a keyboard / virtual keyboard, a touch input screen, or voice input with voice recognition, and a cursor control device 570, such as a mouse or a touch input screen or pad. The system controller also includes a robot interface 580 that controls the movement of the robotic device 125 according to the embodiments described herein.
[0071] 6 is a flow diagram of a method for controlling imaging and a robot insertable into a subject's anatomy, according to a representative embodiment. Various steps of the flow diagram may be performed, for example, by the system controller 120 described above.
[0072] Referring to FIG. 6, in step S611, ultrasound image data is received from an imaging interface device (e.g., imaging interface device 115). The ultrasound image data corresponds to an ultrasound image showing an anatomical structure within a subject (e.g., anatomy 107) and a portion of a robotic device (e.g., robotic device 125) within the anatomical structure. The imaging interface device is operable to provide the ultrasound image data. For example, the imaging interface device is an ultrasound probe that a user can manipulate to obtain various images of the anatomical structure and the robotic device during an interventional procedure, which images can be displayed on a display (e.g., GUI 145). A user can manipulate the ultrasound probe directly by holding it and making adjustments, or remotely via a robotic interface to obtain the desired ultrasound images.
[0073] In an embodiment, the imaging interface device has an on-board actuator that allows a user to select between an imaging mode and a control mode of operation. In the imaging mode, image data received from the imaging interface device is processed to provide an image of the anatomical structure and a portion of the robotic device within the anatomical structure. In the control mode, information indicative of the translational and / or rotational movement of the imaging interface device on the surface of the subject is processed to control the movement of the robotic device within the anatomical structure in conjunction with the translational and / or rotational movement of the imaging interface device. If the imaging interface device is an ultrasound probe, the actuator may include one or more push buttons, slider buttons, and / or touch-sensitive buttons conveniently located on the handle of the ultrasound probe, as described above. This allows a user to select between the imaging mode and the control mode with one hand while holding the ultrasound probe, avoiding interruptions to the intervention procedure and the need for another person to operate the robotic device (or imaging device) separately.
[0074] In step S612, movement of the imaging interface device is detected. As previously described, the movement of the imaging interface device can be translational or rotational, which is detectable in a control mode of operation. Translational movement is detected based on direction, distance, and / or velocity data received from an IMU of the imaging interface device. Rotational movement is detected based on position data received from an IMU of the imaging interface device, or by direct image observation using image processing, such as, for example, speckle tracking processing.
[0075] In block S613, at least one command is output in response to the detected movement of the imaging interface device to control movement of the robotic device from a first position to a second position. The at least one command is provided to a robotic controller (e.g., a PID controller), which then servos to move the robotic device in relation to the detected movement of the imaging interface device. Alternatively, the commands may be stored in a non-transitory memory for later use or additional processing (e.g., formatting, additional checks for debugging purposes) prior to use by the robotic controller.
[0076] If the movement of the imaging interface device is a translational movement, the method includes defining an image plane B and a reference plane R in the ultrasound image relative to the imaging interface device. The image plane B extends from the imaging interface device into the subject and includes a longitudinal axis of the robotic device and / or a longitudinal axis or other feature of an anatomical structure. The reference plane R intersects the image plane B at a first position of the robotic device. More specifically, the robotic device includes a tracking feature, such as a distal tip, a curved portion, or a marker of the robotic device, that defines the first position, and the reference plane R intersects the image plane B at the tracking feature. The reference plane R is translated in relation to the translational movement of the imaging interface device, and at least one command is output to the robot controller to control the robotic device based on a distance between a trackable feature of the robotic device and the translated reference plane R. In an embodiment, the robotic device is controlled based on a distance between a trackable feature of the robotic device and the translated reference plane R, while the robotic device is servo-controlled toward the translated reference plane R using the distance as a position error. That is, the robot controller attempts to compensate for the position error by decreasing the distance between the trackable feature and the reference plane R.
[0077] If the movement of the imaging interface device is a rotational movement, the method includes defining an image plane B in the ultrasound image relative to the imaging interface device, an image plane B in the ultrasound image relative to the imaging interface device, and a reference plane R. The image plane B extends from the imaging interface device into the subject and includes a longitudinal axis of the robotic device. The reference plane R is defined in the ultrasound image relative to the imaging interface device and intersects with the image plane B at a first position of the robotic device. For example, from direct image observation using image processing or position data provided by an IMU of the imaging interface device, a direction and an angle of the rotational movement of the imaging interface device are detected. The reference plane R is rotated in the same direction and angle as the rotational movement of the imaging interface device. In response to the rotated reference plane R, at least one command for controlling the robotic device is output to a robot controller.
[0078] With respect to rotational movement, it is desirable to maintain image plane B in a constant image plane as viewed by the display. In a three-dimensional volume, image plane B remains visually static during rotation by changing the viewpoint of the three-dimensional model and adjusting the virtual section plane. In a two-dimensional image, image plane B remains visually static during rotation by virtually steering image plane B about a longitudinal axis of the imaging interface device in a direction opposite to the corresponding rotation using position data measured by one or more IMUs. In a two-dimensional image, image plane B can also remain visually static during rotation by the imaging interface device selectively activating ultrasound transducers and / or detector elements within the imaging interface device to maintain the ultrasound image in a constant image plane.
[0079] In accordance with various embodiments of the present disclosure, the methods described herein may be implemented using a hardware computer system executing a software program. Furthermore, in non-limiting exemplary embodiments, implementations such as distributed processing, component / object distributed processing, and parallel processing are possible. A virtual computer system process may implement one or more of the methods or functions described herein. The processors described herein may also be used to support virtual processing environments.
[0080] Although the development of adaptive predictive analytics has been described with reference to several exemplary embodiments, it is understood that the words used herein are words of description and illustration, rather than words of limitation. Changes may be made in its aspects within the scope of the appended claims, as currently presented and as amended, without departing from the scope and spirit of the optimization of interventional procedures. Although the development of adaptive predictive analytics has been described with reference to specific means, materials, and embodiments, the development of adaptive predictive analytics is not intended to be limited to the details disclosed. Rather, the development of adaptive predictive analytics extends to all functionally equivalent structures, methods, and uses as set forth in the appended claims.
[0081] The illustrations of the embodiments described herein are intended to provide a general understanding of the structure of various embodiments. The illustrations do not completely describe all elements and features of the disclosure described herein. Many other embodiments will be apparent to those skilled in the art upon review of the present disclosure. Other embodiments can be utilized and derived from the present disclosure, such as structural and logical substitutions and changes can be made without departing from the scope of the present disclosure. Moreover, the illustrations are merely representative and may not be to scale. Certain proportions in the illustrations may be exaggerated and other proportions may be minimized. Thus, the present disclosure and the figures should be considered illustrative and not restrictive.
[0082] One or more embodiments of the present disclosure may be referred to herein, individually and / or collectively, by the term "invention" for convenience only and without any intention to spontaneously limit the scope of the present application to any particular invention or inventive concept. Also, although specific embodiments are illustrated and described herein, it should be understood that subsequent arrangements designed to achieve the same or similar purpose may be substituted for the specific embodiment shown. The present disclosure is intended to cover any and all subsequent adaptations or variations of the various embodiments. Combinations of the above embodiments with other embodiments not specifically described herein will be apparent to those of skill in the art upon review of the description.
[0083] The Abstract of the Disclosure is provided to comply with 37 CFR Rule 1.72(b) and is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, the above Detailed Description may group or describe various features in a single embodiment for the purpose of streamlining the disclosure. This disclosure is not to be interpreted as reflecting an intention that the claimed embodiments require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter may be directed to less than all features of any of the disclosed embodiments. Accordingly, the following claims are incorporated into the Detailed Description, with each claim standing on its own as defining separate claimed subject matter.
[0084] The foregoing description of the disclosed embodiments is provided to enable any person skilled in the art to practice the concepts described in the present disclosure. Therefore, the above disclosed subject matter is considered to be illustrative and not restrictive. Moreover, the appended claims are intended to cover all such modifications, enhancements, and other implementations that fall within the true spirit and scope of the present disclosure. Thus, to the maximum extent permitted by law, the scope of the present disclosure shall be determined by the broadest permissible interpretation of the following claims and their equivalents, and shall not be limited or restricted by the foregoing detailed description.
Claims
1. 1. A controller for controlling a robotic device for insertion into an anatomical structure of a subject, comprising: At least one processor; a non-transitory memory for storing machine-executable instructions that, when executed by the at least one processor, cause the at least one processor to: receiving ultrasound image data from an imaging interface device, the ultrasound image data corresponding to an ultrasound image illustrating the anatomical structure and a portion of the robotic device within the anatomical structure, the imaging interface device being operable to provide the ultrasound image data; detecting movement of the imaging interface device or receiving information representative of the detected movement derived from content of the ultrasound image data; outputting at least one command for controlling movement of the robotic device from a first position to a second position in response to the detected movement of the imaging interface device; a non-transient memory for causing Including, the controller.
2. The controller of claim 1 , wherein the movement of the imaging interface device includes at least one of a translational movement and a rotational movement.
3. The movement of the imaging interface device is a translational movement, and outputting the at least one command for controlling the movement of the robotic device includes: defining an image plane in the ultrasound image for the imaging interface device, the image plane including a longitudinal axis of the robotic device; defining a reference plane in the ultrasound image relative to the imaging interface device that intersects the image plane at the first position of the robotic device, the reference plane moving with the imaging interface device; outputting the at least one command for controlling the robotic device based on a distance between a trackable feature of the robotic device and the translated reference surface; The controller of claim 2 , comprising:
4. Controlling the robotic device based on the distance between the trackable feature of the robotic device and the translated reference surface comprises: The controller of claim 3 , further comprising: using the distance as a position error to servo the robotic device toward the translated reference surface.
5. The controller of claim 4 , wherein the trackable feature of the robotic device includes any of a distal end of the robotic device, a marker on a rotational device, a predetermined section of the robotic device, or a curved portion of the robotic device.
6. The movement of the imaging interface device is a rotational movement, and outputting the at least one command for controlling the movement of the robotic device includes: defining a reference plane within the ultrasound image for the imaging interface device that references an anatomical feature; Detecting a direction or receiving information representative of the direction and angle of the rotational movement of the imaging interface device; outputting the at least one command for controlling the robotic device in response to an amount of rotation of the reference plane relative to the imaging interface device; The controller of claim 2 , comprising:
7. The controller of claim 6 , wherein the reference surface is rotated according to one or more boundary conditions that limit a range of rotation of the reference surface.
8. The machine executable instructions further cause the at least one processor to:
7. The controller of claim 6, further comprising: a controller configured to selectively activate ultrasound transducers and / or detector elements of a transducer array in the imaging interface device to maintain the ultrasound image in a constant image plane during the rotational movement of the imaging interface device.
9. 7. The controller of claim 6, wherein outputting the at least one command for controlling the robotic device in response to the rotated reference plane includes moving the robotic device in the direction and angle of the rotational movement relative to the rotated reference plane.
10. 7. The controller of claim 6, wherein outputting at least one command for controlling the robotic device in response to the rotated reference plane includes rotating the robotic device longitudinally in the direction and the angle of the rotational movement relative to the rotated reference plane.
11. 7. The controller of claim 6, wherein outputting the at least one command for controlling the robotic device in response to the rotated reference plane includes triggering deployment or activation of a surgical tool or instrument.
12. 2. The controller of claim 1, wherein the imaging interface device is selectively operable in an imaging mode and one or more control modes, and wherein the at least one processor receives the ultrasound image data from the imaging interface device in the imaging mode and detects or receives information representative of the movement of the imaging interface device in the control mode.
13. The controller of claim 12 , further configured to select the imaging mode and the control mode.
14. 1. A system for controlling a robotic device for insertion into an anatomy of a subject, comprising: A controller according to claim 12; a movable imaging interface device that transmits ultrasound image data to the controller, the ultrasound image data corresponding to an ultrasound image showing the anatomical structure and a portion of the robotic device within the anatomical structure; and Including, A system, wherein the imaging interface device includes an actuator that a user can manipulate while holding the imaging interface device to interactively select the imaging mode and the control mode.
15. 15. The system of claim 14, wherein the imaging interface device includes an ultrasound probe, and the actuator includes at least one of a push button, a slider button, and a touch sensitive button on a probe handle of the ultrasound probe.
16. 1. A method for controlling a robotic device for insertion into an anatomical structure of a subject, the method comprising: receiving ultrasound image data from an imaging interface device, the ultrasound image data corresponding to an ultrasound image showing the anatomical structure and a portion of the robotic device within the anatomical structure, the imaging interface device being operable to provide the ultrasound image data; detecting movement of the imaging interface device or receiving information representative of the detected movement derived from content of the ultrasound image data; outputting at least one command to a robotic controller for controlling movement of the robotic device from a first position to a second position in response to the detected movement of the imaging interface device; A method comprising:
17. 17. The method of claim 16, further comprising receiving a selection of an imaging mode or a control mode from the imaging interface device, wherein in the imaging mode, the ultrasound image data is received from the imaging interface device, and in the control mode, the movement of the imaging interface device is detected.
18. If the movement of the imaging interface device is a translational movement, the step of outputting at least one command for controlling the movement of the robotic device comprises: defining an image plane in the ultrasound image for the imaging interface device, the image plane including a longitudinal axis of the robotic device; defining a reference plane in the ultrasound image for the imaging interface device that intersects the image plane at the first position of the robotic device; translating the reference surface relative to the translation of the imaging interface device; outputting the at least one command for controlling the robotic device based on a distance between a trackable feature of the robotic device and the translated reference surface; 17. The method of claim 16, comprising:
19. If the movement of the imaging interface device is a rotational movement, outputting at least one command for controlling the movement of the robotic device comprises: defining a reference plane in the ultrasound image that intersects an axis of rotation of the imaging interface device; detecting a direction and angle of the rotational movement of the imaging interface device; outputting the at least one command for controlling the robotic device in response to an amount of rotation of the reference plane relative to the imaging interface device; 17. The method of claim 16, comprising:
20. 1. A non-transitory computer-readable medium storing instructions for controlling a robotic device for insertion into an anatomical structure of a subject, the instructions, when executed by at least one processor, causing the at least one processor to: receiving ultrasound image data from an imaging interface device, the ultrasound image data corresponding to an ultrasound image illustrating the anatomical structure and a portion of the robotic device within the anatomical structure, the imaging interface device being operable to provide the ultrasound image data; detecting movement of the imaging interface device or receiving information representative of the detected movement derived from content of the ultrasound image data; outputting at least one command for controlling movement of the robotic device from a first position to a second position in response to the detected movement of the imaging interface device; A non-transitory computer-readable medium for causing
21. The instructions further cause the at least one processor to:
21. The non-transitory computer-readable medium of claim 20, further comprising receiving a selection of an imaging mode or a control mode from the imaging interface device, wherein in the imaging mode, the ultrasound image data is received from the imaging interface device, and in the control mode, the movement of the imaging interface device is detected.
Citation Information
Patent Citations
Probe system, probe device, probe and method of controlling probe system
JP2005279096A
Ultrasonic diagnostic apparatus
JP2010094275A
Medical Workstation
US20120265071A1
Techniques for determining ultrasound probe motion
US20200375571A1
