Visualization adjustment for instrument rotation
The system addresses disorientation issues during scope rotation by adjusting image and indicator orientations within the user interface, ensuring accurate and safe medical procedures by maintaining a consistent frame of reference.
Patent Information
- Application Number
- JP2023531010
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2020-11-24
- Filing Date
- 2021-11-24
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-11-24
AI Technical Summary
Improper use of medical devices during procedures can adversely affect patient health and treatment effectiveness due to disorientation caused by scope rotation, leading to a loss of reference frame in image data displayed to the user.
A communication interface and control circuit that adjusts the image representation and working channel indicator within the user interface to maintain a consistent frame of reference as the scope rotates, using feedback data from sensors or robotic arms to determine the scope's rotation and apply rotational adjustments.
The system ensures that the user maintains a stable frame of reference during scope rotation, enhancing procedural accuracy and reducing the risk of complications by aligning the image representation and working channel indicator with the scope's orientation.
Smart Images

Figure 2022112969000001
Abstract
Description
[Technical field]
[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Provisional Patent Application No. 63 / 117,658, filed November 24, 2020, and entitled "FOR INSTRUMENT MANAGEMENT," the disclosure of which is incorporated by reference in its entirety herein. [Background technology]
[0002] Various medical procedures involve the use of one or more medical devices to examine and / or treat a patient. In some cases, multiple systems / devices are implemented to control the medical devices and perform the procedure on the patient. Improper use of such systems, devices, and / or medical devices can adversely affect the patient's health and / or the effectiveness of the procedure. Summary of the Invention [Means for solving the problem]
[0003] In some embodiments, the present disclosure relates to a control system comprising a communications interface configured to receive image data from an imaging device associated with a scope including a working channel for deploying a medical instrument, and control circuitry communicatively coupled to the communications interface, The control circuitry configured to generate a graphical representation of the image data, display the graphical representation in a user interface, determine an amount the scope has rotated, and rotate the graphical representation within the user interface based at least in part on the amount the scope has rotated.
[0004] In some embodiments, the control circuitry is configured to display a working channel indicator in the user interface and rotate the working channel indicator in the user interface from a first orientation relative to the user interface to a second orientation relative to the user interface based at least in part on the amount the scope has been rotated. The working channel indicator may indicate an angle of rotation of the working channel relative to the graphical representation. Further, in some embodiments, the control circuitry is configured to display the working channel indicator by overlaying the working channel indicator on the graphical representation.
[0005] In some embodiments, the control circuitry is configured to determine when the scope has completed rotation, and in response to determining when the scope has completed rotation, rotate the image representation within the user interface. Further, in some embodiments, the control circuitry is configured to rotate the image representation within the user interface as the scope rotates.
[0006] In some embodiments, the control circuitry is configured to rotate the image representation in the user interface by displaying the image representation in a cropped form in the user interface while the scope is being rotated.
[0007] In some embodiments, the control circuitry is configured to determine the amount that the scope has rotated based on at least one of feedback data from a robotic arm configured to couple to the scope, sensor data generated by a position sensor associated with the scope, or control signals generated to control the robotic arm.
[0008] In some embodiments, the control circuitry is configured to receive an input control signal from the input device for rotating the working channel, and control the scope to rotate based at least in part on the input control signal. The control circuitry can be configured to control the scope to rotate by controlling the scope to rotate a predetermined amount.
[0009] In some embodiments, the disclosure relates to a method including receiving image data from an imaging device associated with a scope, displaying, by a control circuit, a graphical representation of the image data on a user interface, displaying, by the control circuit, a working channel indicator on the user interface, determining an amount the scope has been rotated, rotating the graphical representation within the user interface based at least in part on the amount the scope has been rotated, and rotating the working channel indicator within the user interface from a first orientation to a second orientation, the scope including a working channel. Further, the working channel indicator indicates an angle of rotation of the working channel relative to the graphical representation.
[0010] In some embodiments, the method further includes determining that the scope has completed the rotation. Rotating the image representation and rotating the working channel indicator can occur in response to determining that the scope has completed the rotation.
[0011] In some embodiments, rotating the image representation includes continuously rotating the image representation. Further, in some embodiments, determining the amount the scope has rotated is based on at least one of feedback data from a robotic arm configured to couple to the scope, position data generated by a position sensor associated with the scope, or control signals generated to control the robotic arm. Further, in some embodiments, rotating the image representation includes displaying the image representation in a cropped form in the user interface while rotating the scope.
[0012] In some embodiments, the method further includes receiving a user input from an input device to rotate the scope a predetermined amount and controlling the scope to rotate the predetermined amount. Further, in some embodiments, displaying the working channel indicator includes overlaying the working channel indicator on the graphical representation.
[0013] In some embodiments, the present disclosure relates to a system comprising a scope including a working channel for removably receiving a medical instrument, an imaging device configured to couple to the scope, and control circuitry communicatively coupled to the scope and the imaging device. The control circuitry is configured to receive image data from the imaging device, cause a graphical representation of the image data to be displayed in a user interface, cause a working channel indicator to be displayed in the user interface, determine an amount the scope has rotated based at least in part on the amount the scope has rotated, rotate the working channel indicator within the user interface, and apply a rotational adjustment to the graphical representation within the user interface based at least in part on the amount the scope has rotated. The working channel indicator indicates an angle of rotation of the working channel.
[0014] In some embodiments, the working channel is offset from a longitudinal axis of the scope.Furthermore, in some embodiments, the control circuitry is configured to apply the rotational adjustment to the image representation by controlling the scope to rotate in a rotational direction and rotating the image representation in a rotational direction within the user interface.
[0015] In some embodiments, the control circuitry is further configured to determine that the scope has completed rotation. The rotational adjustments may be applied responsive to determining that the scope has completed rotation. Moreover, in some embodiments, the control circuitry is configured to apply the rotational adjustments by continuously rotating the image representation as the scope rotates.
[0016] In some embodiments, the control circuitry is further configured to determine the amount that the scope has rotated based on at least one of feedback data from a robotic arm configured to couple to the scope, sensor data generated by a position sensor associated with the scope, or control signals generated to control the robotic arm.
[0017] In some embodiments, the control circuitry is configured to apply the rotational adjustments by displaying the image representation in a cropped form on the user interface while rotating the scope.Further, in some embodiments, the control circuitry is configured to receive an input control signal to rotate the working channel and control the scope to rotate based at least in part on the input control signal.
[0018] In some embodiments, the disclosure relates to one or more non-transitory computer-readable media having stored thereon computer-executable instructions that, when executed by a control circuit, cause the control circuit to perform operations including receiving image data from an imaging device associated with a scope, displaying a pictorial representation of the data in a user interface, displaying a working channel indicator in the user interface, determining an amount the scope has rotated based at least in part on an amount the scope has rotated, rotating the pictorial representation within the user interface, and rotating the working channel indicator within the user interface from a first orientation to a second orientation. The scope includes a working channel. The working channel indicator indicates an angle of rotation of the working channel.
[0019] In some embodiments, the operations further include determining that the scope has completed the rotation. In response to determining that the scope has completed the rotation, the image representation and the working channel indicator may be rotated. Further, in some embodiments, rotating the image representation includes continuously rotating the image representation.
[0020] In some embodiments, determining the amount the scope has rotated includes tracking the amount of rotation of the scope based on at least one of feedback data from a robotic arm configured to couple to the scope, sensor data generated by a position sensor associated with the scope, or control signals generated to control the robotic arm. Further, in some embodiments, rotating the image representation includes displaying the image representation in a cropped form in a user interface while rotating the scope.
[0021] In some embodiments, the operations further include receiving a user input from an input device to rotate the scope a predetermined amount and controlling the scope to rotate the predetermined amount. Further, in some embodiments, the working channel indicator is displayed in an overlay manner on the image representation.
[0022] For purposes of summarizing this disclosure, certain aspects, advantages, and features have been described. It is to be understood that not necessarily all such advantages may be achieved in accordance with any particular embodiment. Thus, the disclosed embodiments may be practiced in a manner that achieves or optimizes one advantage or group of advantages taught herein without necessarily achieving other advantages as may be taught or suggested herein. [Brief description of the drawings]
[0023] Various embodiments are illustrated in the accompanying drawings for illustrative purposes and should not be construed as limiting the scope of the present disclosure in any way. In addition, various features of different disclosed embodiments may be combined to form further embodiments that are part of the present disclosure. Throughout the drawings, reference numbers may be reused to indicate correspondence between referenced elements. [Figure 1] FIG. 1 illustrates an example medical system for performing various medical procedures in accordance with one or more embodiments. [Diagram 2] FIG. 2 illustrates example details of the control system and robotic system of FIG. 1 in accordance with one or more embodiments. [Diagram 3] FIG. 1 illustrates an exemplary scope positioned in a portion of a patient's urinary system, in accordance with one or more embodiments. [Figure 4-1] 14 illustrates an example of rotating a scope and applying a rotational adjustment to a scope image in a user interface when the scope completes the rotation, in accordance with one or more embodiments. [Figure 4-2]14 illustrates an example of rotating a scope and applying a rotational adjustment to a scope image in a user interface when the scope completes the rotation, in accordance with one or more embodiments. [Figure 4-3] 14 illustrates an example of rotating a scope and applying a rotational adjustment to a scope image in a user interface when the scope completes the rotation, in accordance with one or more embodiments. [Figure 4-4] 14 illustrates an example of rotating a scope and applying a rotational adjustment to a scope image in a user interface when the scope completes the rotation, in accordance with one or more embodiments. [Figure 4-5] 14 illustrates an example of rotating a scope and applying a rotational adjustment to a scope image in a user interface when the scope completes the rotation, in accordance with one or more embodiments. [Figure 4-6] 14 illustrates an example of rotating a scope and applying a rotational adjustment to a scope image in a user interface when the scope completes the rotation, in accordance with one or more embodiments. [Diagram 4-7] 14 illustrates an example of rotating a scope and applying a rotational adjustment to a scope image in a user interface when the scope completes the rotation, in accordance with one or more embodiments. [Figure 4-8] 14 illustrates an example of rotating a scope and applying a rotational adjustment to a scope image in a user interface when the scope completes the rotation, in accordance with one or more embodiments. [Figure 4-9] 14 illustrates an example of rotating a scope and applying a rotational adjustment to a scope image in a user interface when the scope completes the rotation, in accordance with one or more embodiments. [Figure 5-1] 14 illustrates an example of rotating a scope and applying rotational adjustments to a scope image in a user interface while rotating the scope in accordance with one or more embodiments. [Figure 5-2]14 illustrates an example of rotating a scope and applying rotational adjustments to a scope image in a user interface while rotating the scope in accordance with one or more embodiments. [Figure 5-3] 14 illustrates an example of rotating a scope and applying rotational adjustments to a scope image in a user interface while rotating the scope in accordance with one or more embodiments. [Figure 5-4] 14 illustrates an example of rotating a scope and applying rotational adjustments to a scope image in a user interface while rotating the scope in accordance with one or more embodiments. [Figure 5-5] 14 illustrates an example of rotating a scope and applying rotational adjustments to a scope image in a user interface while rotating the scope in accordance with one or more embodiments. [Figure 6] 14A-14C illustrate an exemplary user interface for assisting a user in controlling the scope and / or instruments deployed through the working channel of the scope, in accordance with one or more embodiments. [Figure 7] 1 illustrates an example flow diagram of a process for applying rotational adjustments to a scope image in a user interface in accordance with one or more embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0024] The headings provided herein are merely for convenience and do not necessarily affect the scope or meaning of the disclosure. Although specific embodiments and examples are disclosed below, the subject matter extends beyond the specifically disclosed embodiments to other alternative embodiments and / or uses, as well as to modifications and equivalents thereof. Thus, the scope of claims that may arise from this specification is not limited by any of the specific embodiments described below. For example, in any method or process disclosed herein, the acts or operations of the method or process may be performed in any suitable order and are not necessarily limited to any particular disclosed order. Various operations may be described in sequence as multiple separate operations in a manner that may be helpful in understanding a particular embodiment. However, the order of description should not be construed to imply that these operations are order-dependent. Furthermore, structures, systems, and / or devices described herein may be embodied as integrated or separate components. For purposes of comparing various embodiments, certain aspects and advantages of these embodiments are described. Not necessarily all such aspects or advantages are realized by any particular embodiment. Thus, for example, various embodiments may be implemented in a manner that achieves or optimizes one advantage or group of advantages as taught herein without necessarily achieving other aspects or advantages that may also be taught or suggested herein.
[0025] Certain standard anatomical terms of location may be used herein to refer to animal (i.e., human) anatomical structures with respect to preferred embodiments. While certain spatially relative terms, such as "outer," "inner," "superior," "lower," "below," "upper," "vertical," "horizontal," "top," "bottom," and similar terms, are used herein to describe the spatial relationship of one device / element or anatomical structure to another device / element or anatomical structure, it should be understood that these terms are used herein for ease of description to describe the positional relationships between elements / structures as illustrated in the drawings. It should be understood that the spatially relative terms are intended to encompass different orientations of the elements / structures in use or operation in addition to the orientation shown in the drawings. For example, when an element / structure is described as being "above" another element / structure, it may also represent a position below or to the side of such other element / structure with respect to the subject patient or alternate orientations of the elements / structures, and vice versa.
[0026] Overview Many medical procedures involve the use of a scope to examine and / or treat a patient. The scope generally includes or can be associated with an imaging device configured to capture image data of a target site, such as a camera at the distal end of the scope. The image data can be displayed to a user to assist the user in controlling the scope within the patient. The scope 40 can include one or more working channels through which additional tools, such as lithotriptors, basket devices, forceps, laser devices, etc., can be deployed to the target site. The working channels are often offset from the imaging device and / or the longitudinal axis of the scope. Additionally, in some cases, the scope can be configured to move based on different joint planes, such that the scope moves with more / fewer degrees of freedom in a direction when oriented in a particular manner.
[0027] To accommodate various scope movements, treatment angles, physician preferences, and / or other circumstances, the scope can be controlled to rotate within the patient, rotating the associated imaging device and working channel. In one example, a user may rotate the scope so that the tools / instruments deployed through the working channel (offset to one side of the scope) have a more optimal position to treat the target site. In another example, a user may rotate the scope to re-orient the scope's articulation plane to facilitate further movement of the scope in a direction. Since the user is often controlling the scope based on image data captured by the scope, rotating the scope can disorient the user within the patient. For example, image data displayed to the user rotates as the scope rotates, causing the user to lose track of the virtual horizon / frame of reference at the treatment site.
[0028] The present disclosure relates to systems, devices, and methods that provide image data and / or other data associated with a medical instrument in an orientation suitable to assist a user in performing a procedure. For example, the system can be configured to communicate with a scope including a working channel for deploying a medical instrument. The system can receive image data from an imaging device associated with the scope and display an image representation of the image data via a user interface. As the scope rotates, the system can adjust the orientation of the image representation, such as by rotating the image representation in the user interface, which may or may not be perceived as occurring within the user interface. The system can also display a working channel indicator via the user interface that indicates the rotation angle of the working channel (e.g., the position of the working channel relative to the image representation). As the scope rotates, the system can adjust the orientation of the working channel indicator in the user interface (e.g., rotate the working channel indicator) to indicate the correct orientation of the working channel relative to the image representation. In some cases, the image representation and / or the working channel indicator are continuously adjusted within the user interface while rotating the scope, giving the perception that a virtual horizon / frame of reference (as displayed in the image representation) is maintained. For example, the user may perceive that only the working channel indicator moves within the user interface (e.g., the working channel indicator rotates while the virtual horizon in the image representation remains the same). In other cases, the image representation and / or the working channel indicator are rotated within the user interface when the scope completes a rotation or otherwise reaches a particular rotation angle and / or when another event occurs. Thus, the present technique can present image data, indicators for the working channel, and / or other data associated with the scope in the proper orientation to assist the user in maintaining a frame of reference within the patient.
[0029] While many of the techniques are discussed in the context of presenting working channel indicators for medical instruments, the techniques may also or alternatively present indicators for other elements, for example, the techniques may present indicators of joint planes associated with the medical instrument, lights associated with the medical instrument, needles / catheters / other instruments in conjunction with the medical instrument, etc.
[0030] In some embodiments, the techniques described herein implement robotic-assisted medical procedures, where the robotic tool may enable a physician to perform endoscopic and / or percutaneous access and / or endoscopic and / or percutaneous procedures for a target anatomical site. For example, the robotic tool may engage and / or control one or more medical instruments, such as a scope, to access and / or perform procedures at a target site within a patient. In some cases, the robotic tool is guided / controlled by a physician. In other cases, the robotic tool operates automatically or semi-automatically. Although many techniques are discussed in the context of robotic-assisted medical procedures, the techniques may also be applicable to other types of medical procedures, such as procedures that do not implement a robotic tool or that implement a robotic tool only for relatively few operations (e.g., less than a threshold number). For example, the techniques may be applicable to procedures in which a manually operated medical instrument, such as a manual scope, that is fully controlled by the physician, is implemented.
[0031] Certain aspects of the present disclosure are described herein in the context of renal, urological, and / or renal procedures, such as kidney stone removal / treatment procedures. However, such context is provided for convenience, and it should be understood that the concepts disclosed herein are applicable to any suitable medical procedure. For example, the following description is also applicable to other surgical / medical procedures or medical procedures involving removal of objects from a patient, including any object that can be removed from a treatment site or a patient's body cavity (e.g., esophagus, ureter, intestine, eye, etc.) via percutaneous and / or endoscopic access, such as gallbladder stone removal, lung (pulmonary / transthoracic) tumor biopsy, or cataract removal. However, as previously mentioned, a description of the renal / urological anatomy and associated medical problems and procedures is provided below to aid in the explanation of the concepts disclosed herein.
[0032] Medical System Example FIG. 1 illustrates an example medical system 100 for performing various medical procedures according to aspects of the present disclosure. The medical system 100 includes a robotic system 110 configured to engage and / or control one or more medical instruments / devices to perform a procedure on a patient 120. In the example of FIG. 1, the robotic system 110 is coupled to a scope 130 and an electromagnetic (EM) field generator 140. The medical system 100 also includes a control system 150 configured to interface with the robotic system 110, provide information regarding the procedure, and / or perform various other operations. For example, the control system 150 can include a display 152 configured to present certain information to assist the physician 160 in performing the procedure. The medical system 100 can include a table 170 (e.g., a bed) configured to hold the patient 120. Various actions are described herein as being performed by the physician 160. These actions can be performed directly by the physician 160, by a user under the direction of the physician 160, by another user (e.g., a technician), a combination thereof, and / or by any other user. The devices / components of the medical system 100 can be arranged in a variety of ways depending on the type of procedure and / or stage of the procedure.
[0033] The control system 150 can operate in cooperation with the robotic system 110 to perform a medical procedure on the patient 120. For example, the control system 150 can communicate with the robotic system 110 via a wireless or wired connection to control a medical instrument connected to the robotic system 110 and receive images captured by the medical instrument or the like. For example, the control system 150 can receive image data from the scope 130 (e.g., an imaging device associated with the scope 130) and display the image data (and / or a representation generated therefrom) to the physician 160 to assist the physician 160 in navigating the scope 130 within the patient 120. The physician 160 can provide input via an input / output (I / O) device, such as the controller 154, and the control system 150 can send control signals to the robotic system 110 to control the movement of the scope 130 connected to the robotic system 110. The scope 130 (and / or another medical instrument) can be configured to move in various manners, such as articulating in a plane, rotating, etc., as discussed in more detail below.
[0034] In some embodiments, the control system 150 can provide fluid to the robotic system 110 via one or more fluid channels, provide power to the robotic system 110 via one or more electrical connections, provide optics to the robotic system 110 via one or more optical fibers or other components, etc. In an example, the control system 150 can communicate with the medical instrument to receive sensor data (via the robotic system 110 and / or directly from the medical instrument). The sensor data can indicate or be used to determine the position and / or orientation of the medical instrument. Additionally, in an example, the control system 150 can communicate with the table 170 to position or otherwise control the table 170 in a particular orientation. Additionally, in an example, the control system 150 can communicate with the EM field generator 140 to control the generation of the EM field around the patient 120.
[0035] The robotic system 110 can include one or more robotic arms 112 configured to engage or control a medical instrument / device. Each robotic arm 112 can include multiple arm segments coupled to joints, thereby providing multiple degrees of movement. A distal end of the robotic arm 112 (e.g., an end effector) can be configured to couple to an instrument / device. In the example of FIG. 1, the robotic arm 112 (A) is coupled to an EM field generator 140, which can be configured to generate an EM field that is detected by a sensor on the medical instrument, such as an EM sensor on the scope 130. A second robotic arm 112 (B) is coupled to a scope-driver instrument coupling 132, which can facilitate robotic control / advancement of the scope 130. Additionally, a third robotic arm 112 (C) is coupled to a handle 134 of the scope 130, which can be configured to facilitate advancement and / or movement of a medical instrument that can be deployed through the scope 130, such as an instrument deployed through a working channel of the scope 130. In this example, the second robotic arm 112(B) and / or the third robotic arm 112(C) can control the movement (e.g., articulation, rotation, etc.) of the scope 130. Although three robotic arms are shown in Figure 1 connected to specific medical instruments, the robotic system 110 can include any number of robotic arms configured to connect to any type of medical instrument / device.
[0036] The robotic system 110 can be communicatively coupled to any component of the medical system 100. For example, in one example, the robotic system 110 is communicatively coupled to the control system 150 to receive control signals from the control system 150 to perform actions such as controlling the robotic arm 112 in a particular manner, manipulating a medical instrument, etc. Additionally, the robotic system 110 is configured to receive images (also referred to as image data) indicative of the internal anatomical structure of the patient 120 from the scope 130 and / or transmit the images to the control system 150, which can then be displayed on the display 152. Additionally, the robotic system 110 is coupled to components of the medical system 100, such as the control system 150, in a manner such that the robotic system 110 can receive fluids, optics, power, etc. from the components.
[0037] The medical instruments can include various types of instruments, such as scopes (sometimes referred to as "endoscopes"), catheters, needles, guidewires, lithotriptors, basket retrieval devices, forceps, vacuums, needles, scalpels, imaging probes, imaging devices, graspers, scissors, graspers, needle holders, micro-dissection instruments, staple appliers, tackers, suction / irrigation tools, clip appliers, etc. The medical instruments can include direct entry instruments, percutaneous entry instruments, and / or other types of instruments. In some embodiments, the medical instruments are steerable devices, while in other embodiments, the medical instruments are non-steerable devices. In some embodiments, a surgical tool refers to a device configured to puncture or be inserted through a body structure, such as a needle, scalpel, guidewire, etc. However, a surgical tool can also refer to other types of medical instruments.
[0038] The term "scope" or "endoscope" may refer to any type of elongated medical instrument having image generating, viewing, and / or capturing capabilities (or configured to provide such capabilities with an imaging device deployed through a working channel) and configured to be introduced into any type of organ, cavity, lumen, chamber, and / or space of the body. For example, a scope or endoscope such as scope 130 may refer to a ureteroscope (e.g., for accessing the urinary tract), a laparoscope, a nephroscope (e.g., for accessing the kidney), a bronchoscope (e.g., for accessing the airways such as the bronchi), a colonoscope (e.g., for accessing the colon), an arthroscope (e.g., for accessing joints), a cystoscope (e.g., for accessing the bladder), a borescope, etc. A scope / endoscope may include a rigid or flexible tube, as the case may be, and may be sized to pass through an outer sheath, catheter, introducer, or other luminal device, or may be used without such a device. In some embodiments, the scope includes one or more working channels through which additional tools / medical instruments, such as lithotriptors, basket devices, forceps, laser devices, imaging devices, etc., can be introduced into the treatment site. In some cases, the working channels are offset from the imaging device and / or the longitudinal axis of the scope. Furthermore, in some cases, the scope can be configured to move based on different joint planes, such that the scope moves with more / fewer degrees of freedom in certain directions when oriented in a particular manner. For example, the scope can be associated with a primary joint plane and a secondary joint plane, and the tip of the scope is configured to move with more degrees of freedom (e.g., more deflection, larger bend radius, etc.) in the primary joint plane compared to the secondary joint plane. The primary joint plane can be, for example, perpendicular to the secondary joint plane.
[0039] The terms "direct entry" or "direct access" can refer to any entry of an instrument through a natural or artificial orifice in a patient's body. For example, because the scope 130 enters the patient's urinary tract via the urethra, the scope 130 may be referred to as a direct access instrument.
[0040] The term "percutaneous entry" or "percutaneous access" may refer to the entry of an instrument, such as by puncture and / or small incision, through a patient's skin and any other body layers necessary to reach a target anatomical site associated with a procedure (e.g., the calyx rete of a kidney). Thus, a percutaneous access instrument may refer to a medical instrument, device, or assembly configured to puncture or be inserted through the skin and / or other tissue / anatomical structure, such as a needle, scalpel, guidewire, sheath, shaft, scope, catheter, etc. However, it should be understood that a percutaneous access instrument may refer to other types of medical instruments in the context of this disclosure. In some embodiments, a percutaneous access instrument refers to an instrument / device inserted or implemented by a device that facilitates puncture and / or small incision through a patient's skin. For example, a catheter may be referred to as a percutaneous access instrument when inserted through a sheath / shaft that has punctured the patient's skin.
[0041] In some embodiments, the medical instrument includes a sensor (also referred to as a position sensor) configured to generate sensor data. In an example, the sensor data can indicate a position and / or orientation of the medical instrument and / or can be used to determine the position and / or orientation of the medical instrument. For example, the sensor data can indicate a position and / or orientation of a scope, which can include a rotation of a distal end of the scope. The position and orientation of the medical instrument can be referred to as the pose of the medical instrument. The sensor can be located at the distal end of the medical instrument and / or at any other location. In some embodiments, the sensor can provide sensor data to the control system 150, the robotic system 110, and / or another system / device to perform one or more localization techniques to determine / track the position and / or orientation of the medical instrument.
[0042] In some embodiments, the sensor may include an electromagnetic (EM) sensor with a coil of conductive material, where an EM field generator, such as EM field generator 140, may provide an EM field that is detected by an EM sensor on the medical instrument. The magnetic field may induce a small current in the coil of the EM sensor, which may be analyzed to determine the distance and / or angle / orientation between the EM sensor and the EM field generator. Additionally, the sensor may include another type of sensor, such as a camera, a distance sensor (e.g., a depth sensor), a radar device, a shape-sensing fiber, an accelerometer, a gyroscope, an accelerometer, a satellite-based positioning sensor (e.g., a global positioning system (GPS)), a radio frequency transceiver, etc.
[0043] In some embodiments, the medical system 100 may also include an imaging device (not illustrated in FIG. 1 ) that may be integrated into the C-arm and / or configured to image during a procedure, such as in the case of a fluoroscopy type procedure. The imaging device may be configured to capture / generate one or more images of the patient 120 during a procedure, such as one or more X-ray or CT images. In an example, images from the imaging device may be provided in real time to view anatomical structures and / or medical instruments within the patient 120 to assist the physician 160 in performing the procedure. The imaging device may be used to perform fluoroscopy (e.g., with contrast agents within the patient 120) or another type of imaging technique.
[0044] The various components of the medical system 100 may be communicatively coupled to one another via a network, which may include wireless and / or wired networks. Exemplary networks include one or more personal area networks (PANs), local area networks (LANs), wide area networks (WANs), Internet area networks (IANs), cellular networks, the Internet, etc. Additionally, in some embodiments, the components of the medical system 100 are connected for data communication, fluid / gas exchange, power exchange, etc., via one or more supporting cables, tubes, etc.
[0045] In some embodiments, the medical system 100 can provide information to assist the physician 160 in controlling the scope 130 and / or tools deployed through the scope 130. For example, the control system 150 can include an interface 156. (A) Image representation 154 (A) You can display the image representation 154 (A) includes or is based on image data captured by an imaging device associated with scope 130. The imaging device can be disposed on the distal end of scope 130, deployed from the distal end of scope 130 through a working channel of scope 130, or otherwise associated with scope 130. In the example of FIG. (A) shows an internal view of a kidney 190 of a patient 120 (i.e., from the perspective of the scope 130 within a calyx 191 of the kidney 190). Additionally, the control system 150 may generate an image representation 154 (A) A working channel indicator 158 may be provided that indicates the angle of rotation of the working channel of the scope 130 relative to the working channel of the scope 130. For example, a working channel indicator 158 may be provided in the interface 156(A). of 8The location indicates that the working channel is located to the right of Fig. 1 relative to image representation 154(A). Although not shown in Fig. 1, control system 150 may also provide other types of indicators, such as indicators for the orientation of the joint plane of scope 130, indicators for the orientation of a light associated with scope 130, indicators for the position / orientation of a needle / catheter / other instrument in conjunction with scope 130, etc.
[0046] In some embodiments, the control system 150 adjusts the image representation 154 to reflect changes in the orientation of the scope 130. (A) and / or the working channel indicator 158. For example, the control system 150 may adjust the image representation 154 as the scope 130 rotates. (A) The control system 150 may rotate the working channel indicator 158 when the control system 150 is in a position to accurately indicate the orientation of the working channel relative to the image representation 154. (A) The orientation of the image representation 154 may be adjusted to provide the same virtual horizon / frame of reference before and after the rotation. For example, the control system 150 may adjust the orientation of the image representation 154 as the scope 130 rotates. (A) can be rotated so that the view of the internal anatomy of the patient 120 has the same orientation before and after the scope 130 is rotated.
[0047] For purposes of explanation, assume that the scope 130 is positioned within the kidney 190 proximate to the kidney stone 192 and includes a laser device deployed through the working channel to break up the kidney stone 192, as shown in FIG. 1. Also assume that the scope 130 is initially positioned in a first orientation such that the laser from the laser device may miss the kidney stone 192 (i.e., extend along a path 193 that contacts the anatomical structure of the patient 120). Here, the control system 150 controls the working channel indicator 15 positioned on the right side to be aligned with the working channel indicator 15. 8To adjust the orientation of the laser device, the physician 160 can rotate the scope 130 to a second orientation in which the laser from the laser device is likely to contact the kidney stone 192 (i.e., extend along a path 194). In this illustration, the scope 130 has been rotated 180 degrees. However, the scope 130 can be rotated by any angle. The control system 150 then , written by Business Channel Indicator 158 Updated location of The control system 150 may also present an updated user interface 156(B) including the image, as shown in the updated user interface 156(B). Representation 154(A) Rotate it 180 degrees, Updated image Expression 154(B) That is, the control system 150 can present the same horizontal line within the user interface 156 such that the frame of reference within the patient 120 is maintained (e.g., such that the kidney stone 192 appears to the left both before and after rotating the scope 130). (B) Image data in Ta's The orientation can be adjusted. Image Representation 154 (A) and / or working channel indicator 158 may be so adjusted to maintain a frame of reference within patient 120. In another solution, such rotation of scope 130 may be adjusted to maintain a frame of reference within image representation 154. (A) 180 degrees from its initial orientation, potentially disorienting the physician 160.
[0048] In some embodiments, the control system 150 may generate the image representation 154 after the scope 130 has completed a rotation. (A) (and / or working channel indicator 158). For example, in response to determining that scope 130 has rotated 180 degrees, control system 150 may apply an orientation adjustment to image representation 154. (A) to the view shown in the user interface 156(B). Here, the physician 160 can rotate the image representation 154 (A)The content within the patient 120 may be perceived as rotating 180 degrees following the rotation of the scope 130, and then rotating 180 degrees back. Examples of such orientation adjustments are described below with reference to FIGS. 4-1 through 4-9. Alternatively, in some embodiments, the control system 150 may adjust the user interface 156 to maintain a frame of reference within the patient 120. (A) For example, the control system 150 may track the rotation of the scope 130 and continuously update the image representation 154. (A) may be continually updated to maintain a virtual horizon within the patient 120 throughout the rotation of the scope 130. Here, the physician 160 may view the working channel indicator 158 as it changes from the image representation 154 (A) While rotating around the image representation 154 (A) can be perceived as remaining the same. Examples of such continuous alignment adjustments are described below with reference to Figures 5-1 to 5-5.
[0049] In some examples, the medical system 100 is implemented to perform medical procedures related to the kidney anatomy, such as treating kidney stones. Kidney stone disease, also known as urolithiasis, is a medical condition involving the formation of solid pieces of material in the urinary tract, referred to as "kidney stones," "urinary stones," "nephrolithiasis," or "nephrolithiasis." Urinary stones may form and / or be found in the kidneys, ureters, and bladder (referred to as "bladder stones"). Such urinary stones may form as a result of the concentration of minerals in the urine, and may cause significant abdominal pain if such stones reach a size sufficient to obstruct urine flow through the ureter or urethra. Urinary stones may be formed from calcium, magnesium, ammonia, uric acid, cysteine, and / or other compounds, or combinations thereof.
[0050] Generally, there are several methods for treating patients with kidney stones, such as observation, medical treatment (such as expulsion therapy), non-invasive treatment (such as extracorporeal shock wave lithotripsy (ESWL)), and surgical treatment (such as ureteroscopy and percutaneous nephrolithotomy (PCNL)). In surgical approaches (e.g., ureteroscopy and PCNL), a physician accesses the lesion (i.e., the object to be removed, e.g., the stone), breaks the stone into smaller pieces or fragments, and mechanically removes the smaller stone fragments / particles from the kidney.
[0051] To remove urinary stones from the bladder and ureters, a physician may insert a ureteroscope into the urinary tract through the urethra. Typically, the ureteroscope includes an endoscope at its distal end configured to allow visualization of the urinary tract. The ureteroscope may also include a lithotripsy device for capturing or breaking up urinary stones. During the procedure, one physician / technologist may control the position of the ureteroscope while another physician / technologist may control the lithotripsy device. To remove relatively large stones (i.e., "kidney stones") from the kidney, a physician may use a percutaneous nephrolithotomy ("PCNL") technique. This involves inserting a nephroscope through the skin (i.e., percutaneously) and intervening tissue to provide access to the treatment site for breaking up and / or removing the stones.
[0052] In some of the examples described herein, robotic-assisted percutaneous procedures can be performed in connection with various medical procedures, such as kidney stone removal procedures, and robotic tools (e.g., one or more components of the medical system 100) can enable a physician / urologist to perform endoscopic (e.g., ureteroscopy) target access as well as percutaneous access / procedures. However, the present disclosure is not limited to kidney stone removal and / or robotic-assisted procedures. In some implementations, robotic medical solutions can provide relatively high precision, better control, and / or better eye-hand coordination for certain instruments compared to strictly manual procedures. For example, robotic-assisted percutaneous access to the kidney through some procedures can advantageously enable a urologist to perform both direct entry endoscopic renal access and percutaneous renal access. Although some embodiments of the present disclosure are presented in the context of catheters, nephroscopes, ureteroscopes, and / or human renal anatomy, it should be understood that the principles disclosed herein can be implemented in any type of endoscopic / percutaneous procedure or another type of procedure.
[0053] In one exemplary and non-limiting procedure, the medical system 100 can be used to remove a kidney stone 192 from a patient 120. During setup for a procedure, a physician 160 can position the robotic arms 112 of the robotic system 110 in a desired configuration and / or attach appropriate medical instruments. For example, as shown in FIG. 1, the physician 160 can position a first robotic arm 112(A) near the treatment site and attach an EM field generator 140, which can assist in tracking the location of the scope 130 and / or other instruments / devices during the procedure. Additionally, the physician 160 can position a second robotic arm 112(B) between the legs of the patient 120 and attach a scope-driver instrument coupling 132, which can facilitate robotic control / advancement of the scope 130. The physician 160 can insert a medical instrument 136 into the urethra 195, through the bladder 196, and / or up the ureter 197 of the patient 120. The physician 160 can connect a medical instrument 136 to the scope-driver instrument coupling 132. The medical instrument 136 can include a luminal device configured to receive the scope 130, thereby aiding in the insertion of the scope 130 (e.g., an access sheath) into the anatomy of the patient 120. Although the medical instrument 136 is shown in FIG. 1, in some embodiments, the medical instrument 136 is not used (e.g., the scope 130 is inserted directly into the urethra 195). The physician 160 can then insert the scope 130 into the medical instrument 136 manually, robotically, or a combination thereof. The physician 160 can attach the handle 134 of the scope 130 to the third robotic arm 112(C), which can be configured to facilitate the advancement and / or operation of a basket device, a laser device, and / or another medical instrument deployed through the scope 130.
[0054] The physician 160 can interact with the control system 150 to cause the robotic system 110 to advance and / or navigate the scope 130 into the kidney 190. For example, the physician 160 can navigate the scope 130 to locate a kidney stone 192. The control system 150 can also display the image representation 154. (A) an interface 156 for viewing the image (e.g., a real-time image captured by the scope 130), a working channel indicator 158 (e.g., showing the current orientation of the working channel), etc. (A) Information regarding the scope 130 may be provided via the display 152 to assist the physician 160 in navigating the scope 130, such as by displaying a position, orientation, or other orientation information about the scope 130. In some embodiments, the control system 150 may use localization techniques to determine the position and / or orientation of the scope 130, which may be viewed by the physician 160 via the display 152. Additionally, other types of information may be presented via the display 152, such as x-ray images of the internal anatomy of the patient 120, to assist the physician 160 in controlling the scope 130.
[0055] When the scope 130 reaches the site of the kidney stone 192 (e.g., in the calyx 191 of the kidney 190), the scope 130 can be used to designate / tag a target location for a catheter to percutaneously access the kidney 190. To minimize damage to the kidney 190 and / or surrounding anatomical structures, the physician 160 can designate the papilla as the target location for percutaneous entry into the kidney 190 with a catheter. However, other target locations can be designated or determined. In some embodiments that designate the papilla 1512, the physician 160 can navigate the scope 130 to contact the papilla, and the control system 150 can use localization techniques to determine the location of the scope 130 (e.g., the location of the distal end of the scope 130), and the control system 150 can associate the location of the scope 130 with the target location. Additionally, in some embodiments, the physician 160 can navigate the scope 130 to be within a particular distance of the nipple (e.g., positioned in front of the nipple) and provide input indicating that the target location is within the field of view of the scope 130. The control system 150 can perform image analysis and / or other localization techniques to determine the location of the target location. Additionally, in some embodiments, the scope 130 can deliver a reference point to mark the nipple as the target location.
[0056] Once a target location has been designated, a catheter (not shown) can be inserted into the patient 120 through the percutaneous access pathway to reach the target site. For example, the catheter can be connected to the first robotic arm 112(A) (upon removal of the EM field generator 140), and the physician 160 can interact with the control system 150 to cause the robotic system 110 to advance and / or navigate the catheter. In some embodiments, a needle or another medical instrument is inserted into the patient 120 to form the percutaneous access pathway. The control system 150 can provide information regarding the catheter via the display 152 to assist the physician 160 in navigating the catheter. For example, the interface can provide image data from the perspective of the scope 130. The image data can show the catheter (e.g., when it is within the field of view of an imaging device of the scope 130).
[0057] Once the scope 130 and / or catheter are positioned at the target location, the physician 160 can use the scope 130 to break up the kidney stone 192 and / or use the catheter to remove pieces of the kidney stone 192 from the patient 120. For example, the scope 130 can deploy a tool (e.g., a laser, cutting instrument, etc.) through the working channel to fragment the kidney stone 192 into pieces, and the catheter can aspirate the pieces from the kidney 190 through the percutaneous access pathway. In an example, the catheter and / or scope 130 can provide irrigation and / or suction to facilitate removal of the kidney stone 192. For example, the catheter (and / or associated medical instrument) can be coupled to an irrigation and / or suction system.
[0058] At any time before, during, or after the procedure, the physician 160 can rotate the scope 130 to provide a more optimal position / orientation of the scope 130. In one example, when positioning the scope 130 to break up / remove a kidney stone 192, the physician 160 can rotate the scope 130 to reposition the working channel of the scope 130, thereby providing a more optimal orientation for deploying instruments through the working channel. In another example, the scope 130 can be associated with different articulation planes (e.g., primary and secondary planes). Here, the physician 160 can control the scope 130 to rotate to reposition the articulation planes, so that the scope 130 can articulate in a more optimal manner (e.g., flex with more deflection). In some cases, the physician 160 can provide input to rotate the scope 130 a predetermined amount, such as flipping the working channel 180 degrees. In other examples, the physician 160 can provide input to rotate the scope 130 by any angle. In either case, as the scope 130 rotates, the control system 150 generates an image representation 154 (A) In order to maintain the correct orientation of the working channel indicator 158 relative to the user interface 156 (A) In addition, the control system 150 may update the working channel indicator 158 in the image representation 154. (A) In order to maintain the frame of reference for the image representation 154 (A) This allows the physician 160 to maintain the orientation within the kidney 190 of the patient 120 and adjust the image representation 154 (A) / User Interface 156(A) The correct orientation of the working channel relative to the implant can be visualized.
[0059] In some embodiments, control system 150 and / or robotic system 110 prevent scope 130 from rotating (and thus the working channel from rotating) when scope 130 is articulated more than a particular number of degrees (e.g., 20 degrees, 30 degrees, etc.) from the longitudinal axis. For example, when a user input to flip the working channel of scope 130 is received, control system 150 and / or robotic system 110 may prevent flipping of the working channel if scope 130 is articulated more than 30 degrees. However, scope 130 may, in some cases, rotate when articulated by any amount.
[0060] Although many examples are described in connection with displaying data associated with a scope (e.g., displaying image data associated with a scope, displaying a working channel indicator associated with a scope, etc.), the techniques can be implemented in connection with other medical instruments. For example, the techniques can be implemented to present information associated with another type of medical instrument that includes / deploys an imaging device.
[0061] Additionally, while many examples present indicators for the working channel, other types of indicators can be presented. For example, the techniques can display a joint plane indicator that indicates the orientation of the joint plane of the medical instrument relative to the image representation in the user interface, a light indicator for a light located at the distal end of the medical instrument (which can indicate the orientation of the light relative to the image representation), a needle / catheter / instrument indicator for a needle / catheter / other instrument associated with the medical instrument (which can indicate the orientation of the needle / catheter / instrument relative to the image representation), etc. Such indicators can be oriented / updated in the user interface as the scope / medical instrument rotates to accurately indicate the orientation of the item that the indicator represents. In some cases, a needle / catheter / instrument associated with the scope may be outside the field of view of the scope, and thus the needle / catheter / instrument is not displayed in the image representation from the perspective of the scope (e.g., the needle / catheter / instrument is off-screen). Here, the needle / catheter / instrument indicator can still be maintained in the proper orientation in the image representation before and after rotating the scope. For example, the needle / catheter / instrument indicator can be positioned to the right side within the user interface (indicating that the needle / catheter / instrument is off the screen to the right). Such positioning of the needle / catheter / instrument indicator can be the same both before and after rotating the scope, assuming the needle / catheter / instrument is not moved. In an example, position / orientation information regarding the needle / catheter / instrument can be obtained based on a position sensor within the needle / catheter / instrument, such as an EM sensor or shape sensing.
[0062] In many examples, rotation of the scope 130 is controlled by the robotic arm 112, while the robotic arm 112 remains in a relatively stationary position (e.g., the end effector of the robotic arm 112 can remain in the same position). For example, the robotic arm 112 can control / drive a rotation feature / mechanism on the scope 130 to rotate the scope 130. However, the scope 130 can in some cases be rotated by rotating the end effector of the robotic arm 112. For example, the end effector can be rotated relative to the longitudinal axis of the scope 130 such that the handle 134 of the scope 130 rotates about the longitudinal axis.
[0063] The medical system 100 provides various benefits, such as, for example, guidance to assist physicians in performing procedures (e.g., instrument tracking, instrument navigation, instrument calibration, etc.), allowing physicians to perform procedures from ergonomic positions without requiring skilled arm movements and / or positions, allowing one physician to perform procedures with one or more medical instruments, avoiding radiation exposure (e.g., associated with fluoroscopy techniques), performing procedures in a single surgical setting, and providing continuous suction for more efficient removal of objects (e.g., removing kidney stones). For example, the medical system 100 provides guidance information to assist physicians in accessing target anatomical features with various medical instruments while minimizing bleeding and / or damage to anatomical structures (e.g., critical organs, blood vessels, etc.). Additionally, the medical system 100 can provide non-radiation-based navigation and / or localization techniques to reduce physician and patient radiation exposure and / or reduce the amount of equipment in the operating room. Additionally, the medical system 100 can provide distributed functionality between at least the control system 150 and the robotic system 110, thereby allowing for independent mobility. Such distribution of functionality and / or mobility can allow the control system 150 and / or robotic system 110 to be placed in a location that is optimal for a particular medical procedure, thereby maximizing the working area around the patient and / or providing an optimized location for the physician to perform the procedure.
[0064] Although the various techniques and systems are discussed as being implemented as robotically-assisted procedures (e.g., procedures that at least partially use the medical system 100), these techniques and systems can be implemented in other procedures, such as fully robotic medical procedures, human-only procedures (e.g., procedures that do not include a robotic system), etc. For example, the medical system 100 can be used to perform procedures (e.g., fully robotic procedures that rely on relatively few inputs to direct the procedure) without a physician holding / manipulating the medical instruments and without a physician controlling the movements of the robotic system / arm. That is, each of the medical instruments used during the procedure can be held / controlled by a component of the medical system 100, such as the robotic arm 112 of the robotic system 110.
[0065] Control system and robot system examples 2 is a diagram illustrating example details of the control system 150 and robotic system 110 of FIG. 1 in accordance with one or more embodiments. Although certain components of the control system 150 and / or robotic system 110 are illustrated in FIG. 2, it should be understood that additional components not illustrated may be included in embodiments according to the present disclosure. Additionally, any of the illustrated components may be omitted, replaced, and / or integrated into other devices / systems, such as a table 170, medical instruments, etc.
[0066] 2, the control system 150 may include one or more of the following components, devices, modules, and / or units (referred to herein as "components"): one or more I / O components 202, one or more communication interfaces 204, one or more power supply units 206, and / or one or more mobilization components 208 (e.g., casters or other types of wheels), either separately / individually and / or in combination / collectively. In some embodiments, the control system 150 may comprise a housing / enclosure configured and / or dimensioned to house or contain at least a portion of one or more of the components of the control system 150. In this example, the control system 150 is illustrated as a cart-based system that is movable with the one or more mobilization components 208. In some cases, after reaching the appropriate position, the one or more mobilization components 208 may be secured using wheel locks to hold the control system 150 in place. However, the control system 150 may be implemented as a stationary system, integrated into another system / device, etc.
[0067] The various components of control system 150 may be electrically and / or communicatively coupled using certain connection circuits / devices / features that may or may not be part of the control circuitry. For example, the connection features may include one or more printed circuit boards configured to facilitate mounting and / or interconnection of at least some of the various components / circuits of control system 150. In some embodiments, two or more of the components of control system 150 may be electrically and / or communicatively coupled to one another.
[0068] The one or more I / O components / devices 202 can include various components for receiving input and / or providing output, such as for interfacing with a user to assist in performing a medical procedure. The one or more I / O components 202 can be configured to receive touch, speech, gestures, or any other type of input. In an example, the one or more I / O components 202 can be used to provide input for device / system control, such as for controlling the robotic system 110, navigating a scope or other medical instrument attached to the robotic system 110 (and / or deployed via the scope), controlling the table 170, controlling a fluoroscopy device, etc. For example, a physician 160 (not shown) can provide input via the I / O components 202, in response to which the control system 150 can send control signals to the robotic system 110 to operate a medical instrument. In an example, the physician 160 can use the same I / O device to control multiple medical instruments (e.g., switch control between instruments).
[0069] As shown, the one or more I / O components 202 may include one or more displays 152 (which may be referred to as "one or more display devices 152") configured to display data. The one or more displays 152 may include one or more liquid crystal displays (LCDs), light emitting diode (LED) displays, organic LED displays, plasma displays, e-paper displays, and / or any other type of technology. In some embodiments, the one or more displays 152 include one or more touch screens configured to receive input and / or display data. Additionally, the one or more I / O components 202 may include one or more I / O devices / controllers that may include a touch pad, a controller (e.g., a handheld controller, a video game type controller, a tactile interface, a haptic interface, a finger-based control / finger clutch that allows finger-like movements, etc.), a mouse, a keyboard, a wearable device (e.g., an optical head mounted display), a virtual reality device or an augmented reality device (e.g., a head mounted display), a foot panel (e.g., buttons under the user's feet), etc. 203 Additionally, the one or more I / O components 202 may include one or more speakers configured to output sound based on the audio signal and / or one or more microphones configured to receive sound and generate an audio signal. In some embodiments, the one or more I / O components 202 include or are implemented as a console.
[0070] In some embodiments, one or more of the I / O components 202 can output information related to the procedure. For example, the control system 150 can receive real-time images captured by the scope and display the real-time images and / or a visual / pictorial representation of the real-time images via the display 152. The display 152 can present an interface, such as any of the interfaces described herein, that can include image data from the scope and / or another medical instrument. Additionally or alternatively, the control system 150 can receive signals (e.g., analog signals, digital signals, electrical signals, acoustic / sonic signals, pneumatic signals, tactile signals, hydraulic signals, etc.) from a medical monitor and / or sensor associated with the patient, and the display 152 can present information related to the patient's health or environment. Such information can include, for example, information displayed via the medical monitor, such as heart rate (e.g., ECG, HRV, etc.), blood pressure / blood velocity, muscle biosignals (e.g., EMG), body temperature, blood oxygen saturation (e.g., SpO2), CO2, brainwaves (e.g., EEG), environmental temperature and / or local or core temperature.
[0071] The one or more communication interfaces 204 can be configured to communicate with one or more devices / sensors / systems. For example, the one or more communication interfaces 204 can transmit / receive data in a wireless and / or wired manner over a network. Networks according to embodiments of the present disclosure can include a local area network (LAN), a wide area network (WAN) (e.g., the Internet), a personal area network (PAN), a body area network (BAN), etc. In some embodiments, the one or more communication interfaces 204 can implement wireless technologies such as Bluetooth, Wi-Fi, Near Field Communication (NFC), etc.
[0072] The one or more power supply units 206 can be configured to manage and / or provide power to the control system 150 (and / or, in some cases, the robotic system 110). In some embodiments, the one or more power supply units 206 include one or more batteries, such as lithium-based batteries, lead acid batteries, alkaline batteries, and / or another type of battery. That is, the one or more power supply units 206 can comprise one or more devices and / or circuits configured to provide a power source and / or provide power management functions. Additionally, in some embodiments, the one or more power supply units 206 include a mains power connector configured to couple to an alternating current (AC) or direct current (DC) mains power source.
[0073] 2, the control system 150 can include and / or control other components, such as one or more pumps, flow meters, valve controls, and / or fluid access components, to provide controlled irrigation and / or aspiration capabilities to a medical instrument (e.g., a scope), devices deployable through the medical instrument, and the like. In some embodiments, irrigation and aspiration capabilities can be delivered directly to the medical instrument via separate cables. Additionally, the control system 150 can include voltage and / or surge protectors designed to provide filtered and / or protected power to another device, such as the robotic system 110, thereby avoiding the placement of power transformers and other auxiliary power components within the robotic system 110, making the robotic system 110 smaller and more mobile.
[0074] In some embodiments, the control system 150 can include support equipment for sensors deployed throughout the medical system 100. For example, the control system 150 can include optoelectronics for detecting, receiving, and / or processing data received from optical sensors and / or cameras. Such optoelectronics can be used to generate real-time images for display on any number of devices / systems, including the control system 150. Similarly, the control system 150 can include electronic subsystems for receiving and / or processing signals received from deployed electromagnetic (EM) sensors. In some embodiments, the control system 150 can also be used to house and / or position EM field generators for detection by EM sensors in or on the medical instrument.
[0075] Additionally, in some embodiments, the control system 150 can be used to couple to the robotic system 110, the table 170, and / or the medical instruments via one or more cables or connections (not shown). In some implementations, support functions from the control system 150 can be provided via a single cable, simplifying and cluttering the operating room. In other implementations, certain functions can be combined in separate cable lines and connections. For example, power can be provided via a single power cable, while support for control, optics, fluidics, and / or navigation can be provided via separate cables.
[0076] The robotic system 110 generally includes an elongated support structure 210 (also referred to as a "column"), a robotic system base 212, and a console 214 at the top of the column 210. The column 210 can include one or more carriages 216 (also referred to as "arm support members 216") for supporting the deployment of one or more robotic arms 112. The carriages 216 can include individually configurable arm mounts that rotate along a vertical axis to adjust the base of the robotic arms 112 for positioning relative to a patient. The carriages 216 also include a carriage interface 218 that allows the carriage 216 to translate vertically along the column 210. The carriage interface 218 can be connected to the column 210 through slots, such as slots 220 positioned on either side of the column 210 to guide the vertical translation of the carriage 216. The slots 220 can include a vertical translation interface for positioning and / or holding the carriage 216 at various vertical heights relative to the base 212. Vertical translation of the carriage 216 allows the robotic system 110 to adjust the reach of the robotic arm 112 to meet different table heights, patient sizes, physician preferences, etc. Similarly, individually configurable arm mounts on the carriage 216 allow the robotic arm base 222 of the robotic arm 112 to be angled in a variety of configurations. The column 210 may include mechanisms therein, such as gears and / or motors, designed to use a vertically aligned lead screw to translate the carriage 216 in a mechanized manner in response to control signals generated in response to user input, such as input from an I / O device.
[0077] The base 212 can balance the weight of the column 210, the carriage 216, and / or the robotic arm 112 on a surface such as a floor. Thus, the base 212 can house heavier components such as one or more electronics, motors, power supplies, and components that allow for movement and / or stabilize the robotic system 110. For example, the base 212 can include rotatable wheels 224 (also referred to as "casters 224" or "mobilization components 224") that allow the robotic system 110 to move around a room for a procedure. After reaching the appropriate position, the casters 224 can be locked using wheel locks to hold the robotic system 110 in place during a procedure. As shown, the robotic system 110 also includes a handle 226 to aid in steering and / or stabilizing the robotic system 110. In this example, the robotic system 110 is shown as a cart-based system that is mobile. However, the robotic system 110 can be implemented as a stationary system, integrated into a table, etc.
[0078] The robotic arm 112 may generally comprise a robotic arm base 222 and an end effector 228 separated by a series of linkages 230 (also referred to as "arm segments 230") connected by a series of joints 232. Each joint 232 may comprise an independent actuator, and each actuator may comprise an independently controllable motor. Each independently controllable joint 232 represents an independent degree of freedom available to the robotic arm 112. For example, each arm 112 may have seven joints, thus providing seven degrees of freedom. However, any number of joints may be implemented with any degrees of freedom. In an example, multiple joints may provide multiple degrees of freedom, allowing for "redundant" degrees of freedom. Redundant degrees of freedom allow the robotic arm 112 to position each end effector 228 at a specific position, orientation, and / or trajectory in space using different linkage positions and / or joint angles. In some embodiments, the end effectors 228 may be configured to engage and / or control medical instruments, devices, objects, and the like. The freedom of movement of the arm 112 allows the robotic system 110 to position and / or orient medical equipment from a desired point in space and / or allows a physician to move the arm 112 to a clinically convenient position away from the patient to allow access while avoiding arm collisions.
[0079] Each end effector 228 of the robotic arms 112 can include an instrument device manipulator (IDM), which can be attached using a Mechanism Exchange Interface (MCI). In some embodiments, the IDM can be removed and replaced with a different type of IDM. For example, a first type of IDM can manipulate an endoscope, a second type of IDM can manipulate a catheter, a third type of IDM can hold an EM field generator, etc. The MCI can include connectors for communicating air pressure, power, electrical signals, and / or optical signals from the robotic arms 112 to the IDM. The IDMs 228 can be configured to manipulate medical instruments (e.g., surgical tools / instruments) using techniques including, for example, direct drives, harmonic drives, gear drives, belts and pulleys, magnetic drives, etc. In some embodiments, the IDMs 228 can be attached to a respective one of the robotic arms 112, which are configured to insert or retract their respective coupled medical instruments into or from a treatment site.
[0080] In some embodiments, the robotic arm 112 can be configured to control the position, orientation, and / or tip articulation of a medical instrument (e.g., a sheath and / or a reader of a scope) attached thereto. For example, the robotic arm 112 can be configured / configurable to manipulate a scope using an elongated movement member. The elongated movement member can include one or more pull wires (e.g., pull or push wires), cables, fibers, and / or flexible shafts. By way of example, the robotic arm 112 can be configured to actuate a number of pull wires coupled to the scope to deflect the tip of the scope. The pull wires can include any suitable or desired material, such as metallic and / or non-metallic materials, such as stainless steel, Kevlar, tungsten, carbon fiber, and the like. In some embodiments, the scope is configured to exhibit non-linear behavior in response to an applied force by the elongated movement member. The non-linear behavior can be based on the stiffness and compressibility of the scope and the variability in sag or stiffness between different elongated movement members.
[0081] As shown, the console 214 is positioned at the top of the column 210 of the robotic system 110. The console 214 may include a display 234 that provides a user interface for receiving user input and / or providing output (e.g., a dual-purpose device such as a touch screen), such as to provide the physician / user with pre-operative and / or intra-operative data, information for configuring the robotic system 110, and the like. Potential pre-operative data may include pre-operative planning, navigation, and / or mapping data derived from a pre-operative computed tomography (CT) scan, and / or notes from a pre-operative patient interview. Intra-operative data may include optical, sensor, and / or coordinate information provided from tools, as well as vital patient statistics such as respiration, heart rate, and / or pulse. The console 214 may be configured to allow the physician to view and / or display the column 210. 210The console 214 can be positioned and tilted to allow access to the console 214 from the opposite side of the arm support member 216. From this position, a physician may view the console 214, the robotic arm 112, and the patient while operating the console 214 from behind the robotic system 110.
[0082] The robotic system 110 may include one or more I / O components / devices 236 to receive input and / or provide output, such as for interfacing with a user. The one or more I / O components 236 may be configured to receive touch, speech, gestures, or any other type of input. In an example, the one or more I / O components 236 may be used to provide input for device / system control, such as to control / configure the robotic system 110. As shown, the one or more I / O components 236 may include one or more displays 234 configured to display data. The one or more displays 234 may include one or more liquid crystal displays (LCDs), light emitting diode (LED) displays, organic LED displays, plasma displays, e-paper displays, and / or any other type of technology. In some embodiments, the one or more displays 234 may include one or more touch screens configured to receive input and / or display data. Further, the one or more I / O components 236 may include one or more I / O devices / controllers 238, which may include a touchpad, a controller, a mouse, a keyboard, a wearable device (e.g., an optical head mounted display), a virtual reality device or an augmented reality device (e.g., a head mounted display), and the like. Further, the one or more I / O components 236 may include one or more speakers configured to output sound based on an audio signal, and / or one or more microphones configured to receive sound and generate an audio signal. In some embodiments, the one or more I / O components 236 include or are implemented as the console 214. Further, the one or more I / O components 236 may include one or more buttons that can be physically pressed, such as a button on a distal end of the robotic arm 112 (which can enable / disable a robotic arm 112 admittance control mode).
[0083] The various components of the robotic system 110 may be electrically and / or communicatively coupled using certain connection circuits / devices / features that may or may not be part of the control circuitry. For example, the connection features may include one or more printed circuit boards configured to facilitate mounting and / or interconnection of at least some of the various components / circuitry of the robotic system 110. In some embodiments, two or more of the components of the robotic system 110 may be electrically and / or communicatively coupled to one another.
[0084] The one or more communication interfaces 240 can be configured to communicate with one or more devices / sensors / systems. For example, the one or more communication interfaces 240 can transmit / receive data in a wireless and / or wired manner over a network. Networks according to embodiments of the present disclosure can include a local area network (LAN), a wide area network (WAN) (e.g., the Internet), a personal area network (PAN), a body area network (BAN), etc. In some embodiments, the one or more communication interfaces 240 can implement wireless technologies such as Bluetooth, Wi-Fi, Near Field Communication (NFC), etc.
[0085] The one or more power supply units 242 can be configured to manage and / or provide power to the robotic system 110. In some embodiments, the one or more power supply units 242 include one or more batteries, such as lithium-based batteries, lead acid batteries, alkaline batteries, and / or another type of battery. That is, the one or more power supply units 242 can comprise one or more devices and / or circuits configured to provide a power source and / or provide power management functions. Further, in some embodiments, the one or more power supply units 242 include a mains power connector configured to couple to an alternating current (AC) or direct current (DC) mains power source. Further, in some embodiments, the one or more power supply units 242 include a connector configured to couple to the control system 150 to receive power from the control system 150.
[0086] The robotic system 110 may also include one or more actuators / hardware 244 to facilitate movement of the robotic arm 112. Each actuator 244 may comprise a motor, which may be mounted at a joint or elsewhere in the robotic arm 112 to facilitate movement of the joint and / or connected arm segment / linkage. Additionally, the robotic system 110 may include various other components, such as pneumatics, light sources, etc.
[0087] The control system 150 and / or the robotic system 110 may include control circuitry 246 and / or data storage / memory 248 configured to perform the functions described herein. For ease of explanation and illustration, the control circuitry 246 and data storage 248 are shown in blocks between the control system 150 and the robotic system 110. It should be understood that in many embodiments, the control system 150 and the robotic system 110 may include separate instances of the control circuitry 246 and data storage 248. That is, the control system 150 may include its own control circuitry and data storage (e.g., for performing processes on the control system 150), while the robotic system 110 may include its own control circuitry and data storage (e.g., for performing processes on the robotic system 110). In many embodiments, references to control circuitry herein may refer to circuitry embodied in the robotic system, the control system, or any other component of a medical system, such as any component of the medical system 100 shown in FIG. 1.
[0088] Although control circuitry 246 is illustrated as a separate component from the other components of control system 150 / robotic system 110, it should be understood that any or all of the other components of control system 150 / robotic system 110 may be at least partially embodied in control circuitry 246. For example, control circuitry 246 may include various devices (active and / or passive), semiconductor materials and / or areas, layers, regions, and / or portions thereof, conductors, leads, vias, connections, etc., and one or more of the other components of control system 150 / robotic system 110 and / or portions thereof may be at least partially formed and / or embodied in such circuit components / devices.
[0089] As illustrated, the data storage device 248 may include a user interface component 250 configured to facilitate one or more of the user interfaces. For example, the user interface component 250 may generate and / or provide user interface data to cause a user interface to be displayed via the display 152. The user interface may include an image representation from the perspective of the medical instrument (e.g., a scope), indicators for elements associated with the medical instrument (e.g., working channel indicators, joint plane indicators, etc.), controls or other interface elements, and / or other information. The user interface component 250 may adjust the orientation of the image representation as the medical instrument rotates / changes orientation, such as by applying a rotational adjustment to the image representation in the user interface. The user interface component 250 may maintain the orientation of the indicator relative to the image representation to indicate the correct orientation of the element (e.g., working channel) relative to the image representation. The user interface component 250 may use orientation / position information regarding the medical instrument to provide the image representation and / or the working channel indicator.
[0090] The data storage device 248 may also include an instrument drive component 252 configured to drive / control the medical instrument. 252can be configured to process input signals from the I / O component 202, process position / orientation data indicative of the position / orientation of the medical instrument, process image orientation data indicative of the orientation of image data / representation in the user interface, generate control signals for controlling the robotic system 110 and / or medical instruments / tools connected to the robotic system 110, and send the control signals to the robotic system 110 to control the robotic system 110 and / or medical instruments / tools, etc. In some embodiments, user control of the medical instrument can be disabled during rotation of the medical instrument to prevent unintended movement of the medical instrument and / or complications in controlling the mechanical components of the robotic system 110. For example, the instrument drive component 252 can ignore received user input once a scope rotation has been initiated until the scope rotation is completed, such that the user cannot further articulate the scope during rotation.
[0091] In some embodiments, the instrument driving component 252 can control the medical instrument to move in a manner that correlates to the view provided through the user interface. For example, assume that the physician 160 is driving the scope from the scope's perspective (e.g., based on image data from the scope provided through the user interface) and the physician 160 provides input through the I / O component 202 to move the scope upward relative to the I / O component 202, such as by selecting a move up control on the I / O component 202. The instrument driving component 252 can determine the orientation / position of the scope and / or the orientation of the image data / image representation displayed in the user interface. The instrument driving component 252 can use such orientation / position information to move the scope upward on the user interface. Thus, the instrument driving component 252 can configure / update the control of the medical instrument based on the orientation of the image data / image representation displayed in the user interface. In an example, the manner in which the medical instrument is controlled can be reconfigured as the medical instrument rotates.
[0092] Additionally, the data storage device 248 may include a localization component 254 configured to determine / track the orientation / position of an object / medical instrument. For example, the localization component 254 may process the input data to generate position / orientation data of a medical instrument coupled to the robotic system 110. In some cases, the position / orientation data may indicate an amount of rotation of the medical instrument. The object / medical instrument position / orientation data may indicate a position / orientation of the object / medical instrument relative to a frame of reference. The frame of reference may be a frame of reference relative to a patient's anatomy, a known object (e.g., an EM field generator), a coordinate system / space, etc. In some implementations, the position / orientation data may indicate a position and / or orientation of a distal end (and / or, in some cases, a proximal end) of a medical instrument. For example, the position / orientation data of a scope may indicate a position and orientation of the distal end of the scope, including an amount of rotation of the distal end of the scope. The position and orientation of an object may be referred to as the pose of the object. In an example, a technique for determining and / or tracking a position and / or orientation of a medical instrument may be referred to as a localization technique.
[0093] Exemplary input data that can be used to generate the object / medical instrument position / orientation data can take a variety of different forms, such as sensor data from sensors associated with the medical instrument (e.g., EM field sensor data, visual / image data captured by an imaging / depth / radar device on the medical instrument, accelerometer data from an accelerometer on the medical instrument, gyroscope data from a gyroscope on the medical instrument, fiber optic based shape sensing, satellite based positioning data from a satellite based sensor (e.g., Global Positioning System (GPS)), etc.), feedback data (also referred to as "kinematic data") from a robotic arm / component (e.g., data indicating how the robotic arm / component moved / actuated), robot command data for the robotic arm / component (e.g., control signals sent to the robotic system 110 / robot arm 112 to control the movement of the robotic arm 112 / medical instrument), model data regarding the patient's anatomy (e.g., a model of the internal / external parts of the patient's anatomy), patient position data (e.g., data indicating how the patient is positioned on a table), pre-operative data, etc.
[0094] In some implementations, the localization component 254 can use electromagnetic tracking to determine the location and / or orientation of an object. For example, the localization component 254 can use real-time EM tracking to determine the real-time location of the medical instrument in a coordinate system / space that can be registered to the patient's anatomy, and the location can be represented by a pre-operative model or other model. In EM tracking, an EM sensor (or tracker) including one or more sensor coils can be embedded in one or more locations and / or orientations in the medical instrument (e.g., scope, needle, etc.). The EM sensor can measure variations in an EM field formed by one or more static EM field generators positioned at known locations. The location information detected by the EM sensor can be stored as EM data. The localization component 254 can process the EM data to determine the location and / or orientation of an object, such as a medical instrument. An EM field generator (or transmitter) can be placed near the patient (e.g., within a predetermined distance) to form a low-intensity magnetic field that can be detected by the EM sensor. The magnetic field can induce small currents (e.g., below a threshold) in the sensor coil of the EM sensor, which can be analyzed to determine the distance and / or angle between the EM sensor and the EM field generator. These distances and / or orientations can be "recorded" intraoperatively on the patient's anatomy (e.g., a pre-operative model) to determine a geometric transformation that aligns a single location in a coordinate system with a location in the pre-operative model of the patient's anatomy. Once recorded, an EM sensor (e.g., an implanted EM tracker) at one or more locations of the medical instrument (e.g., the distal tip of an endoscope, a needle, etc.) can provide a real-time display of the position and / or orientation of the medical instrument through the patient's anatomy.
[0095] Additionally, in some embodiments, the localization component 254 can perform vision-based techniques to determine the location and / or orientation of an object. For example, the medical instrument can be equipped with a camera, a distance sensor (sometimes referred to as a “depth sensor”), a radar device, etc., to provide sensor data in the form of visual / image data. The localization component 254 can process the visual data to facilitate vision-based location tracking of the medical instrument. For example, pre-operative model data can be used in conjunction with the visual data to enable computer vision-based tracking of the medical instrument (e.g., an endoscope). In an example, using the pre-operative model data, the control system 150 can generate a library of predicted endoscopic images based on the expected path of travel of the scope, with each image linked to a location in the model. During surgery, this library can be referenced by the control system 150 to compare real-time images and / or other visual data acquired by the scope (e.g., a camera at the distal end of the endoscope) to images in the image library to aid in localization.
[0096] Additionally, other types of vision-based techniques can be performed to determine the location and / or orientation of the object. For example, the localization component 254 can use feature tracking to determine the movement of the image sensor (e.g., a camera or other sensor) and therefore the medical instrument associated with the image sensor. In some cases, the localization component 254 can identify circular shapes in the preoperative model data that correspond to anatomical lumens and track changes in those shapes to determine which anatomical lumens were selected, as well as the relative rotational and / or translational movement of the medical instrument. The use of a topology map can also further enhance the vision-based algorithms or techniques. Additionally, the localization component 254 can use optical flow, another computer vision-based technique, to analyze the displacement and / or translation of image pixels in a video sequence in the vision data to infer camera movement. Examples of optical flow techniques can include motion detection, object segmentation calculations, luminance, motion compensated coding, stereo disparity measurements, etc. By comparing multiple frames over multiple iterations, the localization component 254 can determine the movement and location of the image sensor (and therefore the endoscope).
[0097] Further, in some embodiments, the localization component 254 can use the robot command and / or kinematic data to determine the position and / or orientation of the object. The robot command and / or kinematic data can indicate the position / orientation (e.g., pitch, yaw, etc.) of the robot arm resulting from the articulation / actuation command. For example, the robot command and / or kinematic data can indicate the amount / direction of actuation of a joint, gear, pulley, belt, or another actuation element of the robot arm. The localization component 254 can use the data indicating the position / orientation of the robot arm to determine the position / orientation of a medical instrument attached to the robot arm. For example, based on the position / orientation of the robot arm attached to a catheter, the command sent to control the catheter, and / or the characteristics of the catheter (e.g., catheter length, catheter capacity, etc.), the localization component 254 can determine / estimate the position / orientation of the catheter. Further, in an example, the localization component 254 can use the robot command data to determine how far the medical instrument has been inserted / retracted into the patient, etc., based on the command to control the medical instrument, markings on the medical instrument indicating distance, etc. In some intraoperative embodiments, the calibration measurements can be used in combination with known insertion depth information to estimate the position and / or orientation of the medical instrument. Alternatively or additionally, these calculations can be analyzed in combination with EM, visual, and / or topological modeling to estimate the position and / or orientation of the medical instrument.
[0098] Additionally, in some embodiments, the localization component 254 can use other types of data to determine the position and / or orientation of the object. For example, the localization component 254 can analyze sensor data from shape-sensing fibers (e.g., which can provide shape data regarding the location / shape of the medical instrument), accelerometers, gyroscopes, satellite-based positioning sensors (e.g., Global Positioning System (GPS)), radio frequency transceivers, etc. embedded in the medical instrument. Such data can indicate the position and / or orientation of the medical instrument.
[0099] In some embodiments, the localization component 254 can process the pre-operative data to determine the position and / or orientation of the object. The pre-operative data (sometimes referred to as "mapping data") can be generated by performing a computed tomography (CT) scan, such as a low-dose CT scan. The pre-operative CT images from the scan can be reconstructed into a three-dimensional image, which is visualized (e.g., as a cutaway "slice" of the patient's internal anatomy). When analyzed as a whole, an image-based model can be generated of the anatomical cavities, spaces, and structures of the patient's anatomy, such as the patient's pulmonary network, renal anatomy, etc. Centerline shapes can be determined and approximated from the CT images to create a two- or three-dimensional volume of the patient's anatomy, referred to as model data (also referred to as "pre-operative model data" when generated using only pre-operative CT scans). A network topology model can also be derived from the CT images.
[0100] In some embodiments, the localization component 254 can use a combination of input data. For example, the localization component 254 can use a probabilistic approach in which confidence weights are assigned to positions / orientations determined from multiple forms of input data. To illustrate, when the EM data is unreliable (such as in the presence of EM interference), the EM data can be associated with a relatively low confidence value and other forms of input data, such as visual data, robot commands, kinematic data, etc., can be relied upon.
[0101] Although many embodiments are discussed in the context of a user interface component 250, an instrument driving component 252, and / or a location determination component 254 being implemented (or including) one or more instructions executable by the control circuitry 246, any of these components may be implemented, at least in part, as control circuitry.
[0102] The term "control circuitry" may refer to any collection of one or more processors, processing circuits, processing modules / units, chips, dies (e.g., semiconductor dies including one or more active and / or passive devices and / or connection circuits), microprocessors, microcontrollers, digital signal processors, microcomputers, central processing units, graphics processing units, field programmable gate arrays, application specific integrated circuits, programmable logic devices, state machines (e.g., hardware state machines), logic circuits, analog circuits, digital circuits, and / or any devices that manipulate signals (analog and / or digital) based on hard-coding of circuitry and / or operational instructions. The control circuitry may further include one or more storage devices, which may be embodied in a single memory device, multiple memory devices, and / or embedded circuits of a device. Such data storage devices may include read-only memory, random access memory, volatile memory, non-volatile memory, static memory, dynamic memory, flash memory, cache memory, data storage registers, and / or any device that stores digital information. It should be noted that in embodiments in which the control circuitry includes a hardware state machine (and / or implements a software state machine) and includes analog, digital, and / or logic circuits, the data storage devices / registers that store any associated operational instructions may be embedded within or external to the circuitry that includes the state machine, analog, digital, and / or logic circuits.
[0103] The term "memory" can refer to any suitable or desirable type of computer-readable medium, including, for example, one or more volatile, non-volatile, removable, and / or non-removable data storage devices implemented using any suitable or desirable technology, layout, and / or data structures / protocols that contain any suitable or desirable computer-readable instructions, data structures, program modules, or other types of data.
[0104] Computer-readable media that may be implemented according to embodiments of the present disclosure may include, but are not limited to, phase-change memory, static random-access memory (SRAM), dynamic random-access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage device, magnetic cassette, magnetic tape, magnetic disk storage device or other magnetic storage device, or any other non-transitory medium that may be used to store information for access by a computing device. As used in certain contexts herein, computer-readable media may generally not include communication media such as modulated data signals and carrier waves. Thus, computer-readable media should generally be understood to refer to non-transitory media.
[0105] Example Scope FIG. 3 is a diagram illustrating a scope 130 positioned in a portion of a patient's urinary system, in accordance with one or more embodiments of the present disclosure. In this example, the scope 130 is implemented as an endoscope / ureteroscope. As mentioned above, ureteroscopic procedures can be implemented to investigate and / or treat abnormalities within a human ureter. For example, ureteroscopic procedures can be implemented to treat and / or remove kidney stones. Such procedures can be performed at least in part manually and / or at least in part using robotic technology, such as the robotic system 110 shown in FIG. 1. For example, the use of robotic devices and / or systems for certain endoscopic procedures can provide relatively greater precision, control, and / or coordination compared to strictly manual procedures.
[0106] The scope 130 may include a rigid or flexible tube and / or may be sized to pass within an outer sheath, catheter, introducer, or other luminal device. However, the scope 130 may be used without such a device in some examples. For example, the scope 130 may be configured to pass through an access sheath 136 to access the target anatomical structure. The access sheath 136 may have a diameter sufficient to allow the scope 130 to be retracted through the access sheath along with the object captured by the instrument (e.g., a basket device) if the size of the object / stone is not too large (e.g., within a threshold size). In some examples, the access sheath 136 may be advanced through the ureter 197 to a position near the renal pelvis 302 and / or the ureteropelvic junction 304. The distal end of the access sheath 136 may be placed at a position within the ureter 197 and / or the renal pelvis 302, and such placement location may depend at least in part on the anatomy. However, in other instances, the access sheath 136 may be positioned only within the urethra or other anatomical structure, without being positioned near the renal pelvis 302. In general, the access sheath 136 may not be articulatable to the extent that the scope 130 can articulate, and therefore it may not be practical to navigate / drive the access sheath 136 into the kidney.
[0107] The scope 130 may be articulatable, such as with respect to at least a distal portion of the scope, such that the scope 130 may be maneuvered within the human anatomy. In some embodiments, the scope 130 is configured to be articulated with 5 degrees of freedom (DOF), including, for example, XYZ coordinate translation, and pitch and yaw. Additionally, in some embodiments, the scope 130 is articulatable with 6-DOF, including XYZ coordinate translation, and pitch, yaw, and rotation. In other embodiments, the scope 130 is articulatable with other DOF. In embodiments in which the scope 130 includes a position sensor, the position sensor may provide position information such as 3-DOF position information (e.g., x, y, and z coordinates), 5-DOF position information (e.g., x, y, and z coordinates and pitch and yaw angles), 6-DOF position information (e.g., x, y, and z coordinates and pitch, yaw, and rotation angles), etc. In some embodiments, the scope 130 may include telescopic portions, such as an inner leader portion and an outer sheath portion, which may be manipulated to telescopically extend the scope 130.
[0108] The scope 130 may include one or more elongated movement members (not shown) configured to control the movement of the scope 130, such as the distal end of the scope 130. The elongated movement members may include one or more wires (e.g., pull wires or push wires), cables, fibers, and / or flexible shafts. The pull wires may include any suitable or desired material, such as metallic and non-metallic materials, such as stainless steel, Kevlar, tungsten, carbon fiber, and the like. In some embodiments, the scope 130 is configured to exhibit non-linear behavior upon a force applied by the elongated movement members. The non-linear behavior may be based on the stiffness and / or compressibility of the scope 130, as well as variability in sag or stiffness between different elongated movement members. In an embodiment of a robotic arm, one or more pull wires coupled to the scope 130 may be configured to be actuated to deflect the tip 306 of the scope 130. In contrast, in a user handheld embodiment, a user can provide manual input via an actuator to actuate one or more pull wires in the scope 130 and deflect the tip 306 of the scope 130.
[0109] The scope 130 may include an imaging device 308 configured to capture image data, such as image data representative of the patient's internal anatomical structures. The imaging device 308 may include a camera, such as an optical camera and / or another imaging device. The imaging device 308 may include an optical fiber, a fiber array, and / or a lens. One or more optical components of the imaging device 308 may move with the tip 306 of the scope 130, and movement of the tip 306 results in a change in the image captured by the imaging device 308. In some embodiments, the scope 130 may house wires and / or optical fibers for transmitting signals between the optical assembly and the distal end 306 of the scope 130. Additionally, the scope 130 may be further configured to house optical fibers to carry light from a proximally disposed light source, such as a light emitting diode, to the distal end 306 of the scope. The distal end 306 of the scope 130 may include a port for a light source 310 to illuminate the anatomical space, which may be useful when using the imaging device 308. Although multiple light sources 310 are shown, the scope 130 may be implemented with any number of light sources.
[0110] The scope 130 may also include a working channel 312 for deploying an instrument 314. An exemplary instrument 314 may be a laser. 35The working channel 312 may include a laser device 314a configured to provide a suction to the scope 130, a basket device 314b configured to capture / retrieve an object (e.g., fragments of a kidney stone), forceps 314c configured to grasp / hold an object, a scalpel 314d configured to cut an object, a lithotriptor (not shown), an irrigation / aspiration device (not shown) configured to provide irrigation / aspiration to a target site, and the like. The working channel 312 may extend longitudinally through the scope 130 from the proximal end to the distal end 306 of the scope 130. In an example, the working channel 312 is offset to one side of the scope 130 (e.g., offset from the longitudinal axis 316 of the scope 130), as shown in FIG. 3. In other examples, the working channel 312 is positioned at the center of the scope 130 or at another location. Although the imaging device 308 is shown as being attached to the distal end 306 of the scope 130 (e.g., integral with the scope 130), in some cases, the imaging device 308 is a separate device that is deployed through a working channel 312. Additionally, although a single working channel 312 is shown, any number of working channels may be implemented.
[0111] In some embodiments, the scope 130 includes a sensor (sometimes referred to as a “position sensor”) configured to generate and / or transmit sensor data to another device. The sensor data (sometimes referred to as “sensor position data”) can indicate a position and / or orientation of the scope 130 (e.g., its distal end 306) and / or can be used to determine / infer the position / or orientation of the scope 130. For example, the sensor can provide sensor data to a control system, which is then used to determine the position and / or orientation of the scope 130. The sensor can be located at the distal end 306 of the scope 130 and / or at another location. In some embodiments, the sensor can include an electromagnetic (EM) sensor having a coil of conductive material, or other forms / embodiments of an antenna. However, the scope 130 can include other types of sensors, such as shape-sensing fibers, accelerometers, gyroscopes, satellite-based positioning sensors (e.g., Global Positioning System (GPS) sensors), radio frequency transceivers, etc.
[0112] In some implementations, the scope 130 can be configured to move in different planes by different amounts such that the scope moves with more / fewer degrees of freedom in certain directions when oriented in a particular manner. For example, the scope 130 can be associated with a primary joint plane and a secondary joint plane, the primary and secondary planes being perpendicular to one another. The tip 306 of the scope 130 can be configured to move with more degrees of freedom (e.g., exhibit more deflection) in the primary joint plane compared to the secondary joint plane.
[0113] The scope 130 and / or instrument 314 may be controllable in any suitable or desirable manner, based on either manual manipulation of handle components, electronic user input, or automated input. For example, image 318 shows an exemplary robotic control configuration for controlling the scope 130 and / or instrument 314 using one or more robotic arms, where the scope-driver instrument coupling 132 and / or the handle 134 can each be coupled to a robotic arm that can manipulate the scope-driver instrument coupling 132 and / or the handle 134 to control the scope 130 / instrument 314. Meanwhile, image 320 shows an exemplary manual control configuration, where the scope 130 includes a handheld handle 322 configured to be held / manipulated by a user to facilitate movement of the scope 130 / instrument 314.
[0114] Exemplary Image / Indicator Adjustments 4-1-4-9 show examples of rotating a scope and applying rotational adjustments to a scope image in a user interface when the scope completes the rotation (sometimes referred to as a "stepped approach"), according to one or more embodiments. Meanwhile, FIG. 5-1-5 show examples of rotating a scope and applying rotational adjustments to a scope image in a user interface as the scope rotates (sometimes referred to as a "continuous approach"), according to one or more embodiments. In these examples, the scope may rotate a predetermined amount to rotate the working channel of the scope by the same amount, thereby providing a better orientation for the scope to treat the target site. In particular, the scope rotates 180 degrees to reorient the working channel. However, any amount of rotation may be performed, which may or may not be predetermined. Scope rotation may be initiated in various manners, such as by receiving a user input via an I / O device / component (via a user interface or otherwise), automatically determining to rotate the scope, etc.
[0115] In these examples, the user interface 402 / 502 provides information to assist a user in controlling the scope 404 / 504 and / or instruments deployed through the scope 404 / 504, such as to perform a medical procedure. The user interface 402 / 502 can provide an image representation 406 / 506 that represents a view from the perspective of the scope 404 / 504 and a working channel indicator 408 / 508 that indicates an angle of rotation of the working channel of the scope 404 / 504 relative to the image representation 406 / 506. The working channel indicator 408 / 508 can be displayed within the user interface 402 / 502 at all times, and / or at other times when working channel rotation is performed. The image representation 406 / 506 can include, be based on, and / or be generated from image data captured by an imaging device associated with the scope 404 / 504. Here, the image representation 406 / 506 shows a kidney stone 410 / 510 and a body cavity 412 / 512 adjacent to the kidney stone 410 / 510, which may be typically viewed when performing a procedure to remove the kidney stone 410 / 510 (although the kidney stone 410 / 510 may be otherwise oriented). As shown, the user interface 402 / 502 may include a user interface element 414 / 514 for rotating the working channel of the scope 404 / 504 (i.e., rotating the scope 404 / 504). Meanwhile, the image 416 / 516 and element 418 / 518 of each figure indicate the orientation of the scope 404 / 504 within the patient's anatomy for the respective figure. The element 418 / 518 represents the orientation of the tip and coordinate frame of the scope 404 / 504 for the respective image 416 / 516.
[0116] In general, the orientation of the image representation in the user interface can be correlated (with or without offset) to the orientation of the tip of the scope / imaging device. In other words, the user's frame of reference (also referred to as the "user's view") through the user interface is correlated to the orientation of the tip of the scope / imaging device. Thus, when the scope 404 rotates, the image representation generally rotates in the user interface unless a rotation adjustment is applied. To illustrate, in the context of Figures 4-1 to 4-9, the coordinate frame of the scope 404 can initially be oriented as shown at 418 in Figure 4-1, with the upward direction of movement (e.g., the direction pointed to by the arrow at 418) relative to the coordinate frame of the scope 404 corresponding to the leftward direction in Figure 4-1. Here, the coordinate frame of the imaging device of the scope 404 is the same, but may be offset in some cases. In this example, the user interface 402 provides an image representation 406 associated with the scope 404 with an offset in Figure 4-1 such that the upward direction in the user interface 402 corresponds to the rightward direction in the coordinate frame of the scope 404. Such offsets can be maintained until a rotational adjustment is applied.
[0117] Furthermore, controls of the scope may generally be correlated to a user interface such that the scope may be controlled in an intuitive manner by the user. For example, assume that a user provides input via an I / O component to move the scope in an upward direction relative to the I / O component, such as by selecting a move up control on the I / O component. Information regarding the orientation / position of the scope and / or the orientation of the image representation in the user interface may be used to move the scope in an upward direction on the user interface, even if such direction is not upward with respect to the coordinate frame of the scope. Thus, the user may view the scope as moving in an upward direction on the user interface.
[0118] As discussed above, FIGS. 4-1-4-9 show examples of applying a rotation adjustment to the image representation 406 after the scope 404 has completed a rotation. In these figures, the arrow next to the image 416 indicates the direction of rotation of the scope 404, and the arrow next to the user interface 402 indicates the direction of rotation of the image representation 406 within the user interface 402. For example, in FIGS. 4-6-4-9, the arrow next to the user interface 402 indicates that the image representation 406 is rotating counterclockwise within the user interface 402, which is due to a rotation adjustment being applied to adjust / correct the orientation of the image representation 406.
[0119] As shown, FIG. 4-1 illustrates an image representation in a user interface 402 when a scope 404 is oriented in an initial position. 406 and working channel indicator 408 4-5. Here, the scope 404 is oriented as shown at 418. In this example, the user rotates the scope 404 a predetermined amount by selecting an interface element 414. As shown in FIGS. 4-2-4-5, the scope 404 rotates 180 degrees counterclockwise. However, the scope 404 can be rotated clockwise to achieve the same amount of rotation. FIGS. 4-5 show the orientation of the user interface 402 and the scope 404 after the scope 404 has completed the rotation. As the scope 404 rotates, the content in the image representation 406 rotates clockwise in the user interface 402, as also shown in FIGS. 4-2-4-5. That is, a virtual horizon associated with the image representation 406 can rotate clockwise due to the rotation of the scope 404 (and the associated imaging device of the scope 404). The virtual horizon can represent a reference line for the content in the image representation 406 (e.g., a reference horizontal line for the patient's anatomy). With reference to Figures 4-1 through 4-9, the virtual horizon may initially be the horizon of Figure 4-1.
[0120] While the scope 404 is rotating, the image representation 406 may be displayed in a cropped form. For example, as shown in FIGS. 4-2 to 4-5, the image representation 406 may be displayed in a circular form within the user interface 402. This may avoid black bands appearing in the user interface 402 due to missing image data from the imaging device. The cropped form may also indicate to the user that the scope 404 is rotating. Although many examples are discussed in the context of presenting the image representation 406 in a cropped manner (e.g., a circle) during and during the rotation of the scope 404, the image representation 406 may be displayed in an uncropped manner and / or in other forms during the rotation, such as the initial / original form of the image representation 406 (based on the field of view of the imaging device), another shape, etc.
[0121] As the scope 404 rotates, the working channel indicator 408 can maintain the rotational angle of the working channel relative to the image representation 406. In particular, as shown in FIGS. 4-2-4-5, the working channel indicator 408 remains stationary in the user interface 402 as a virtual horizontal line in the image representation 406 rotates to accurately indicate the position of the working channel relative to the content of the image representation 406. That is, the working channel indicator 408 can maintain the same rotational angle in the user interface 402 as the scope 404 rotates up to 180 degrees from an initial orientation. In this example, the working channel indicator 408 is presented in an overlay manner on the image representation 406 as a partially transparent interface element. This can allow the image representation 406 to be at least partially viewable through the working channel indicator 408. However, the working channel indicator 408 can be displayed in other manners, such as in a non-overlay manner relative to 406, as an opaque interface element (e.g., solid), a fully transparent interface element, and / or in other forms / manners.
[0122] Once the scope 404 has completed its rotation, resulting in a 180 degree rotation of the working channel, a rotation adjustment may be applied to adjust / correct the orientation of the image representation 406. In an example, the determination that the scope 404 has completed its rotation may be made based on one or more of the techniques (e.g., localization techniques) described herein. As shown in Figures 4-6-4-9, the rotation adjustment may include rotating the image representation 406 counterclockwise within the user interface 402 until a virtual horizontal line associated with the image representation 406 returns to the initial orientation shown in Figure 4-1 (e.g., the content within the image representation 406 returns to the initial orientation). The image representation 406 may be rotated the same amount (e.g., 180 degrees) that the scope 404 has rotated. The image representation 406 may be rotated within the user interface 402 while the scope 404 remains in the same orientation (e.g., rotated / flipped position). 4-9 illustrates the user interface 402 (and working channel indicator 408) when the image representation 406 has been rotated 180 degrees to return the virtual horizon to its initial orientation. Although the rotational adjustment has been described in the context of rotating the image representation 406 counterclockwise, the rotational adjustment may alternatively be implemented by rotating the image representation 406 clockwise.
[0123] While rotating the image representation 406, the image representation 406 may be displayed in a cropped form in a manner similar to that described while rotating the scope 404. Once a rotation adjustment is applied to the image representation 406 and the working channel 408 is rotated, as shown in FIGS. 4-9, the image representation 406 may be returned to an uncropped form. However, the image representation 406 may be displayed in other forms during and / or upon completion of the scope rotation. Thus, the rotation adjustment may provide the user with the same frame of reference (e.g., the same virtual horizon) before and after rotating the scope 404, while allowing the user to accurately view the position of the working channel (e.g., the working channel is inverted).
[0124] 4-6-4-9, during rotational adjustment of the image representation 406, the working channel indicator 408 can be rotated counterclockwise within the user interface 402 to maintain an accurate representation of the angle of the working channel relative to the image representation 406. The working channel indicator 408 can be rotated the same amount that the scope 404 is rotated (e.g., 180 degrees).
[0125] In some embodiments, the rotational adjustment techniques discussed in the context of FIGS. 4-6 through 4-9 are implemented in cases where it may be difficult to accurately track the orientation of the tip of the scope 404 during rotation. For example, rotating the scope 404 may include disengaging / engaging mechanical components of the scope 404 / robot arm, assessing the tightness of the elongated shaft of the scope 404, sending one or more control signals to components of the robotic system, etc. Such operations may take an unknown amount of time and / or be associated with an amount of delay. Furthermore, in examples where operations are performed at the proximal end of the scope 404, such operations may not instantly translate to rotation at the distal end of the scope 404. Furthermore, in some examples, the scope 404 may be rotated when the EM field generator is not in use, causing the control system to rely on data other than EM sensor data to track the rotation of the scope 404, which may be less accurate / reliable.
[0126] 4-1-4-9 are described with the image representation 406 rotating when the scope 404 completes a rotation, the image representation 406 may rotate when other events occur. For example, the image representation 406 may be rotated in an incremental manner such that each time the scope 404 completes a predetermined amount of rotation, the image representation 406 rotates by such amount (e.g., rotates 15 degrees for every 15 degree rotation completed by the scope 404).
[0127] On the other hand, Figs. 5-1 to 5-5 show examples of applying rotation adjustments in a continuous approach while rotating the scope 504. In these figures, an arrow next to the image 516 indicates the direction of rotation of the scope 504. Here, the image representation 506 is continuously updated in the user interface 502 as the scope 504 rotates from an initial orientation to a final orientation of 180 degrees. As shown, the virtual horizon for the image representation 506 remains the same throughout the rotation of the scope 504, but the working channel indicator 508 rotates around the image representation 506. In particular, the scope 504 rotates in a counterclockwise direction relative to the coordinate frame of the scope 504, and the working channel indicator 508 rotates in the same direction (i.e., counterclockwise direction) relative to the user interface 502. Furthermore, the image representation 506 can rotate in a counterclockwise direction in an imperceptible manner. To make such adjustments to the working channel indicator 508 and the image representation 506, the orientation of the scope 504 can be tracked in real-time using one or more of the localization techniques described herein. This allows the user interface 502 to present information in an accurate manner. As also described above, while rotating the scope 504, the image representation 506 can be displayed in a cropped form, as shown in Figs. 5-2-5-4. Once the scope 504 has completed the rotation, the image representation 506 can return to an uncropped form, as shown in Fig. 5-5. By continuously performing such rotational adjustments, the user may perceive the image representation 506 to remain the same, which may further help the user avoid becoming disoriented within the patient's anatomy. That is, in Figs. 5-1-5-5, the orientation of the image representation 506 can be continuously updated to maintain a virtual horizon relative to the patient's anatomy (e.g., to maintain the user's frame of reference).
[0128] Although Figures 4-1-4-9 and 5-1-5-9 show various image representations in cropped or uncropped form during particular stages of scope rotation, the image representations may be displayed in cropped or uncropped form at any time before, during, and / or after scope rotation.
[0129] Exemplary User Interface 6 is a diagram illustrating an example user interface 602 for assisting a user in controlling a scope and / or instruments deployed through a working channel of the scope, according to one or more embodiments. The user interface 602 can provide a pictorial representation 604 representing a view from the perspective of the scope and a working channel indicator 606 indicating a rotational angle of the working channel of the scope relative to the pictorial representation 604. The pictorial representation 604 can include, be based on, and / or be generated from image data captured by an imaging device associated with the scope. Here, the pictorial representation 604 shows a kidney stone 608 and a body cavity 610 adjacent to the kidney stone 608. In this example, the user interface 602 is presented during a medical procedure. However, the user interface 602 could be presented at other times.
[0130] As shown, the user interface 602 can include a working channel element 612 for rotating the working channel of the scope. The working channel indicator 606 can be implemented as a semi-transparent semi-circular element overlaid on the image representation 604. However, the working channel element 612 can take other forms and / or be located elsewhere within the user interface 602. In one example, a user can select the working channel element 612 to rotate the scope 180 degrees (e.g., flip from an initial orientation to an orientation rotated 180 degrees), although rotation is also possible. In another example, a user can provide a first input via an I / O component (e.g., the user interface 602, a controller, etc.) to initiate scope rotation and provide a second input to stop scope rotation. In yet another example, a user can provide an input to rotate the scope by selecting and holding a control element on the I / O component and rotate the scope while the control element is selected. In either event, one or more of the rotation adjustment techniques discussed herein can be implemented to adjust the image representation 604 as / after the scope rotates. Additionally, the working channel indicator 606 can be adjusted to indicate the orientation of the working channel of the scope. The user interface 602 can also include a horizontal rotation element 614 for rotating the scope without applying rotational adjustments to the image representation 604 in the user interface 602 (e.g., without rotationally correcting the scope image). This can be used to align the visual horizon with the main plane of the kidney or other features. This can also be used to align the main / principal plane of articulation of the scope so that the scope can more easily reach / navigate to a location.
[0131] Additionally, the user interface 602 may include articulation bars 616, 618 about the pictorial representation 604 to visualize the amount of articulation associated with a selected instrument (e.g., a scope or a tool in this example). The top / bottom articulation bar 616 may indicate an amount of vertical articulation. For example, the top / bottom articulation bar 616 may be positioned above or below the image data 604 and / or may expand / contract in length to indicate vertical articulation of the medical instrument. In the example of FIG. 6 , the top / bottom articulation bar 616 is positioned above the pictorial representation 604 to indicate that the scope is articulating upward. The right / left articulation bar 618 may indicate an amount of horizontal articulation. For example, the right / left articulation bar 618 may be positioned to the right or left of the pictorial representation 604 and / or may expand / contract in length to indicate horizontal articulation of the medical instrument. In the example of FIG. 6, right / left articulation bar 618 is positioned to the right of image representation 604 to indicate that the scope is articulated to the right.
[0132] The user interface 602 may also include interface elements 620, 622 for selecting a medical instrument to calibrate / control. The interface element 620 allows for control of the scope, and the interface element 622 allows for control of instruments (also referred to as "tools") associated with the scope. The instruments associated with the scope may be deployed through the working channel of the scope. In this example, the scope is selected, which allows the user to navigate the scope and causes the user interface 602 to display information about the scope.
[0133] Flow diagram example FIG. 7 illustrates an example flow diagram of a process 700 for applying rotational adjustments to a scope image in a user interface, according to one or more embodiments. Various actions / acts associated with the process 700 may be performed by control circuitry implemented in any or combination of devices / systems described herein, such as the control system 150, the robotic system 110, the table 170, a medical instrument, and / or another device. Although various blocks are illustrated as being part of the process 700, any such blocks may be omitted. Additionally, additional blocks may be implemented as part of the process 700. The order in which the blocks are illustrated is for illustrative purposes only, and the blocks may be implemented in any order. In some embodiments, one or more of the blocks of the process 700 are implemented as executable instructions that, when performed by the control circuitry, cause the control circuitry to perform the described functionality / actions. However, one or more of the blocks of the process 700 may be implemented in other ways, such as by other devices / systems, a user, etc.
[0134] At block 702, process 700 can include receiving image data from an imaging device associated with a scope. The scope can include a working channel for deploying a medical instrument / tool. The working channel can be offset from a longitudinal axis of the scope. In some cases, the imaging device is located on the end of the scope, while in other cases, the imaging device is deployed through the working channel of the scope. In examples, the image data can be received in real time, such as receiving multiple images from the imaging device.
[0135] At block 704, process 700 may include generating an image representation based at least in part on the image data. For example, the image representation may include / represent image data from a scope. In some cases, the image data is reformatted / converted to generate the image representation.
[0136] At block 706, process 700 may include causing the image representation to be displayed on a user interface. In one example, causing the image data to be displayed may include transmitting the data to a display such that the display can present the image representation via the user interface. In another example, causing the image representation to be displayed may include displaying / presenting the image representation via a display.
[0137] At block 708, the process 700 may include causing a working channel indicator to be displayed. The working channel indicator may indicate a rotation angle of the working channel relative to the pictorial representation. In one example, causing the working channel indicator to be displayed may include transmitting data to a display such that the display may present the pictorial representation via a user interface. In another example, causing the pictorial representation to be displayed may include displaying / presenting the working channel indicator via the display. In some cases, the working channel indicator is overlaid on the pictorial representation and / or positioned at a periphery of the pictorial representation.
[0138] At block 710, process 700 may include receiving an input control signal from an input device to rotate the working channel. For example, a user may provide an input to rotate the scope using an input device, and the input device may generate / send an input signal indicating to rotate the scope. In some cases, the input is to rotate the scope / working channel a predetermined amount.
[0139] At block 712, the process 700 can include controlling the scope to rotate based at least in part on the input control signal. For example, the control signal can be sent to a robotic system / arm to control the robotic arm to rotate the scope, which can be in particular a rotational direction. In some cases, the scope can be controlled to rotate a predetermined amount, such as flipping the working channel from one orientation to another.
[0140] At block 714, process 700 may include determining that the scope has completed rotation. Such a determination may be based on feedback data from a robotic arm configured to couple to the scope, sensor data generated by a position sensor associated with the scope, and / or control signals generated to control the robotic arm. In some cases, the rotation of the scope is tracked over time.
[0141] At block 718, process 700 may include determining an amount that the scope has rotated. Such a determination may be based on feedback data from a robotic arm configured to couple to the scope, sensor data generated by a position sensor associated with the scope, and / or control signals generated to control the robotic arm. Although process 700 shows blocks 714 and 716, in some cases, one or more of such blocks (and / or other blocks of process 700) are not implemented.
[0142] At block 718, the process 700 can include rotating the working channel indicator within the user interface. For example, based at least in part on the amount the scope has been rotated, the working channel indicator can be rotated from a first orientation relative to the user interface to a second orientation relative to the user interface. The first orientation can indicate a first angle of rotation of the working channel, and the second orientation can indicate a second angle of rotation of the working channel.
[0143] At block 720, the process 700 may include applying a rotational adjustment to the image representation within the user interface. For example, the image representation may be rotated within the user interface based at least in part on the amount the scope has rotated. In some cases, the image representation may be rotated within the user interface upon a determination that the scope has completed a rotation. In other examples, the image representation may be rotated as the scope rotates, such as continuously based on the current orientation of the scope. Here, the image representation may be rotated without being perceived by the user via the user interface. In examples, the image representation may be displayed in a cropped form while the scope is rotating and / or while the image representation is rotating. Additionally, in examples, the image representation may be rotated within the user interface in the same rotational direction (e.g., clockwise or counterclockwise) as the scope rotates relative to the frame of reference of the tip of the scope (e.g., the coordinate frame of the scope).
[0144] Further embodiments Depending on the embodiment, certain acts, events, or functions of any of the algorithms or processes described herein may be performed in a different order, added, merged, or omitted entirely, and thus, in some embodiments, not all of the described acts or events are necessary to the performance of a process.
[0145] In particular, conditional language used herein, such as "can," "could," "might," "may," "eg," and the like, unless specifically stated otherwise or understood otherwise within the context in which it is used, is intended to have its ordinary meaning and is generally intended to convey that certain embodiments include certain features, elements, and / or steps, while other embodiments do not. Thus, such conditional language is generally not intended to suggest that features, elements, and / or steps are required in any way for one or more embodiments, or that one or more embodiments necessarily include logic, with or without input or prompting, for determining whether those features, elements, and / or steps are included or performed in any particular embodiment. Terms such as "comprising," "including," "having," and the like, are used in their ordinary sense and are used inclusively in a non-limiting manner and do not exclude additional elements, features, acts, operations, etc. Also, when the term "or" is used, for example, to connect a list of elements, the term "or" is used in its inclusive sense (and not its exclusive sense) to mean one, some, or all of the listed elements. Unless specifically stated otherwise, connective language such as the phrase "at least one of X, Y, and Z" is understood in the context as it is commonly used to convey that an item, term, element, etc. can be either X, Y, or Z. Thus, such connective language is not generally intended to imply that a particular embodiment requires that at least one of X, at least one of Y, and at least one of Z, respectively, be present.
[0146] In the above description of the embodiments, it should be understood that various features are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the method of the disclosure should not be interpreted as reflecting an intention that any claim requires more features than are expressly recited in that claim. Moreover, any component, feature, or step illustrated and / or described in a particular embodiment herein may be applied to or used with any other embodiment. Moreover, no component, feature, step, or group of components, features, or steps is necessary or essential for each embodiment. Thus, it is intended that the scope of the disclosure disclosed herein and claimed below should not be limited by the specific embodiments described above, but should be determined by a fair reading of the following claims.
[0147] It should be understood that certain ordinal terms (e.g., "first" or "second") may be provided for ease of reference and do not necessarily imply physical characteristics or ordering. Thus, as used herein, ordinal terms (e.g., "first," "second," "third," etc.) used to modify elements, such as structures, components, operations, etc., do not necessarily indicate a priority or order of the element relative to any other elements, but rather may generally distinguish the element from other elements having similar or identical names (apart from the use of the ordinal term). Furthermore, as used herein, the indefinite articles ("a" and "an") may indicate "one or more" rather than "one." Furthermore, an operation performed "based on" a condition or event may also be performed based on one or more other conditions or events not expressly recited.
[0148] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the example embodiments belong. It is further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning consistent with their meaning in the context of the relevant art, and should not be interpreted in an idealized or overly formal sense unless expressly so defined herein.
[0149] Spatially relative terms such as "outer", "inner", "upper", "lower", "below", "upper", "vertical", "horizontal", and similar terms may be used herein for ease of description to describe the relationship between one element or component and another element or component as illustrated in the figures. It should be understood that the spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if the device shown in the figures is inverted, a device positioned "below" or "under" another device may be placed "above" the other device. Thus, the illustrative term "lower" may include both lower and upper positions. The device may also be oriented in other directions, and thus the spatially relative terms may be interpreted differently depending on the orientation.
[0150] Unless otherwise specified, comparative and / or quantitative terms such as "less," "more," "greater than," and the like are intended to encompass the notion of equality. For example, "less" can mean not only "less than" in the strict mathematical sense, but also "less than or equal to."
[0151] [Embodiment] (1) A control system comprising: a communications interface configured to receive image data from an imaging device associated with a scope including a working channel for deploying a medical instrument; a control circuit communicatively coupled to the communication interface, generating a pictorial representation of the image data; Displaying the graphical representation in the user interface; determining an amount that the scope has rotated; and control circuitry configured to rotate the image representation within the user interface based at least in part on the amount the scope is rotated. (2) The control circuit displaying a working channel indicator on the user interface, the working channel indicator indicating a rotation angle of the working channel relative to the pictorial representation; 2. The control system of claim 1, further comprising: rotating the working channel indicator within the user interface from a first orientation relative to the user interface to a second orientation relative to the user interface based at least in part on the amount the scope has been rotated. (3) A control system as described in embodiment 2, wherein the control circuit is configured to display the working channel indicator by overlaying the working channel indicator on the image representation. (4) The control circuit determining that the scope has completed a rotation; 2. The control system of claim 1, configured to rotate the image representation within the user interface upon determining that the scope has completed rotation. (5) A control system as described in embodiment 1, wherein the control circuit is configured to rotate the image representation within the user interface as the scope rotates.
[0152] (6) The control system of embodiment 1, wherein the control circuit is configured to determine the amount that the scope has rotated based on at least one of feedback data from a robotic arm configured to couple to the scope, sensor data generated by a position sensor associated with the scope, or control signals generated to control the robotic arm. (7) The control circuit receiving an input control signal from an input device to rotate the working channel; 2. The control system of claim 1, configured to control the scope to rotate based at least in part on the input control signal. (8) A control system as described in embodiment 7, wherein the control circuit is configured to control the scope to rotate by controlling the scope to rotate a predetermined amount. (9) A method comprising the steps of: receiving image data from an imaging device associated with a scope, the scope including a working channel; displaying, by a control circuit, a pictorial representation of the image data on a user interface; displaying, by the control circuitry, a working channel indicator on the user interface, the working channel indicator indicating a rotation angle of the working channel relative to the pictorial representation; determining an amount that the scope has rotated; based at least in part on the amount that the scope has rotated; rotating the image representation within the user interface; rotating the working channel indicator from a first orientation to a second orientation within the user interface. (10) further comprising determining when the scope has completed rotation; 10. The method of claim 9, wherein rotating the image representation and rotating the working channel indicator occur upon a determination that the scope has completed rotation.
[0153] (11) The method of claim 9, wherein rotating the image representation includes rotating the image representation continuously. (12) The method of embodiment 11, wherein determining the amount that the scope has rotated is based on at least one of feedback data from a robotic arm configured to couple to the scope, position data generated by a position sensor associated with the scope, or control signals generated to control the robotic arm. (13) The method of embodiment 11, wherein rotating the image representation includes displaying the image representation in a cropped form on the user interface while rotating the scope. (14) receiving a user input from an input device to rotate the scope a predetermined amount; 10. The method of claim 9, further comprising controlling the scope to rotate the predetermined amount. (15) The method of claim 9, wherein displaying the working channel indicator includes overlaying the working channel indicator on the image representation.
[0154] (16) A system comprising: a scope including a working channel for removably receiving a medical instrument; an imaging device configured to couple to the scope; a control circuit communicatively coupled to the scope and the imaging device, receiving image data from the imaging device; displaying a pictorial representation of the image data on a user interface; and displaying a working channel indicator on the user interface, the working channel indicator indicating a rotation angle of the working channel; determining an amount that the scope has rotated; rotating the working channel indicator within the user interface based at least in part on the amount the scope has been rotated; and a control circuit configured to apply a rotational adjustment to the image representation in the user interface based at least in part on the amount the scope has been rotated. (17) The system of embodiment 16, wherein the working channel is offset from the longitudinal axis of the scope. (18) The control circuit Controlling the scope to rotate in a rotational direction; 17. The system of claim 16, further configured to apply the rotational adjustment to the image representation by rotating the image representation in the rotational direction within the user interface. (19) The control circuit further configured to determine when the scope has completed rotation; 17. The system of claim 16, wherein the rotational adjustment is applied upon determining that the scope has completed rotation. (20) The system of claim 16, wherein the control circuitry is configured to apply the rotational adjustment by continuously rotating the image representation as the scope rotates.
[0155] (21) The system of embodiment 16, wherein the control circuit is further configured to determine the amount that the scope has rotated based on at least one of feedback data from a robotic arm configured to couple to the scope, sensor data generated by a position sensor associated with the scope, or control signals generated to control the robotic arm. (22) The system of claim 16, wherein the control circuitry is configured to apply the rotational adjustments by displaying the image representation in cropped form on the user interface while the scope is rotated. (23) The control circuit receiving an input control signal to rotate the working channel; 17. The system of claim 16, configured to control the scope to rotate based at least in part on the input control signal. (24) One or more non-transitory computer-readable media storing computer-executable instructions that, when executed by a control circuit, cause the control circuit to: receiving image data from an imaging device associated with a scope, the scope including a working channel; displaying a pictorial representation of the image data on a user interface; displaying a working channel indicator within the user interface, the working channel indicator indicating a rotation angle of the working channel; determining an amount that the scope has rotated; rotating the image representation within the user interface based at least in part on the amount the scope has been rotated; and and rotating the working channel indicator within the user interface from a first orientation to a second orientation based at least in part on the amount the scope has been rotated. (25) The operation is determining that the scope has completed a rotation; One or more non-transitory computer-readable media as described in embodiment 24, wherein the image representation and the working channel indicator are rotated upon determining that the scope has completed rotation.
[0156] (26) One or more non-transitory computer-readable media according to embodiment 24, wherein rotating the image representation comprises continuously rotating the image representation. (27) One or more non-transitory computer-readable media according to embodiment 24, wherein determining the amount that the scope has rotated includes tracking the amount of rotation of the scope based on at least one of feedback data from a robotic arm configured to couple to the scope, sensor data generated by a position sensor associated with the scope, or control signals generated to control the robotic arm. (28) One or more non-transitory computer-readable media according to embodiment 24, wherein rotating the image representation includes displaying the image representation in a cropped form on the user interface while rotating the scope. (29) The operation is receiving a user input from an input device to rotate the scope a predetermined amount; and controlling the scope to rotate the predetermined amount. (30) One or more non-transitory computer-readable media as described in embodiment 24, wherein the working channel indicator is displayed in an overlay manner on the image representation.
Claims
1. 1. A control system comprising: a communications interface configured to receive image data from an imaging device associated with the medical instrument; a control circuit communicatively coupled to the communication interface, generating an image representation based on the image data; Displaying the graphical representation in a user interface; determining a change in rotation of the medical instrument; and control circuitry configured to rotate the graphical representation relative to the user interface based at least in part on a change in the rotation of the medical instrument.
2. The medical instrument includes a working channel, and the control circuitry comprises: displaying a working channel indicator on the user interface, the working channel indicator indicating a rotation angle of the working channel relative to the pictorial representation; The control system of claim 1 , further configured to: rotate the working channel indicator relative to the user interface based at least in part on the change in rotation of the medical instrument.
3. The control system of claim 2, wherein the working channel indicator is displayed as an overlay on the image representation.
4. The control circuit includes: configured to determine when the medical instrument has ceased rotating; The control system of claim 1 , wherein the graphical representation is rotated in response to a determination that the medical instrument has ceased rotation.
5. The control system of claim 1, wherein the image representation is rotated simultaneously as the medical instrument changes its rotation.
6. 10. The control system of claim 1, wherein the control circuitry is configured to determine the change in rotation of the medical instrument based on feedback from a robotic arm coupled to the medical instrument, sensor data generated by a position sensor associated with the medical instrument, or control signals associated with controlling the robotic arm.
7. The control circuit includes: receiving an input control signal from an input device to rotate the medical instrument; The control system of claim 1 , configured to adjust the rotation of the medical instrument based at least in part on the input control signal.
8. The control system of claim 7, wherein the rotation of the medical instrument is adjusted a predetermined amount in response to the input control signal.
9. 1. A method comprising: Receiving image data from an imaging device associated with the medical instrument; generating an image representation based on the image data; displaying the graphical representation in a user interface; determining a change in rotation of the medical instrument; and rotating the graphical representation relative to the user interface based at least in part on the change in the rotation of the medical instrument.
10. 1. A system comprising: Medical equipment and an imaging device associated with the medical instrument; a control circuit communicatively coupled to the medical instrument and the imaging device, receiving image data from the imaging device; generating an image representation based on the image data; displaying the graphical representation in a user interface; determining a change in rotation of the medical instrument; and a control circuit configured to rotate the graphical representation relative to the user interface based at least in part on a change in the rotation of the medical instrument.
