Visualization adjustment for instrument rotation

Abstract

The present invention provides a technique to assist users in performing procedures by providing image data, indicators, and / or other data associated with medical devices in an appropriate orientation. [Solution] The user interface can present an image representation showing image data from the scope and / or a work channel indicator showing the rotation angle of the scope's work channel. When the scope rotates, the rotation adjustment can be applied to the image representation, allowing the image representation to rotate within the user interface. Furthermore, the work channel indicator can also rotate within the user interface.

Description

Technical Field

[0001] (Cross - Reference to Related Applications) This application claims priority to U.S. Provisional Patent Application No. 63 / 117,658, filed on November 24, 2020, entitled "FOR INSTRUMENT MANAGEMENT", the disclosure of which is incorporated herein by reference in its entirety.

Background Art

[0002] Various medical procedures involve the use of one or more medical devices to examine and / or treat a patient. In some cases, multiple systems / devices are implemented to control the medical devices to perform a treatment on a patient. If the use of such systems, devices, and / or medical devices is inappropriate, it may have an adverse impact on the patient's health and / or the effectiveness of the treatment.

Summary of the Invention

Means for Solving the Problems

[0003] In some embodiments, the present disclosure relates to a control system comprising a communication interface configured to receive image data from an imaging device associated with a scope including a working channel for deploying a medical instrument, and a control circuit communicatively coupled to the communication interface. The control circuit is configured to generate an image representation of the image data, display the image representation on a user interface, determine an amount by which the scope has rotated, and rotate the image representation within the user interface based at least in part on the amount by which the scope has rotated.

[0004] In some embodiments, the control circuit is configured to display a working channel indicator on the user interface and to rotate the working channel indicator within the user interface from a first orientation relative to the user interface to a second orientation relative to the user interface, at least in part, based on the amount the scope has rotated. The working channel indicator can indicate the rotation angle of the working channel relative to the image representation. Furthermore, in some embodiments, the control circuit is configured to display the working channel indicator by overlaying it on the image representation.

[0005] In some embodiments, the control circuit is configured to determine when the scope has completed its rotation and, upon determining that the scope has completed its rotation, to rotate the image representation within the user interface. Furthermore, in some embodiments, the control circuit is configured to rotate the image representation within the user interface as the scope rotates.

[0006] In some embodiments, the control circuit is configured to rotate the image representation within the user interface by rotating the scope and displaying the image representation in a cropped form on the user interface.

[0007] In some embodiments, the control circuit is configured to determine the amount the scope has rotated based on at least one of the following: feedback data from a robotic arm configured to be coupled to the scope, sensor data generated by a position sensor associated with the scope, or a control signal generated to control the robotic arm.

[0008] In some embodiments, the control circuit is configured to receive an input control signal from an input device to rotate a working channel, and to control the scope to rotate based at least in part on the input control signal. The control circuit may be configured to control the scope to rotate by controlling the scope to rotate by a predetermined amount.

[0009] In some embodiments, the Disclosure relates to a method comprising: receiving image data from an imaging device associated with a scope; displaying an image representation of the image data on a user interface using a control circuit; displaying a working channel indicator on the user interface using a control circuit; determining the amount by which the scope has rotated; rotating the image representation within the user interface based at least in part on the amount by which the scope has rotated; and rotating the working channel indicator within the user interface from a first orientation to a second orientation. The scope includes a working channel. Furthermore, the working channel indicator indicates the rotation angle of the working channel relative to the image representation.

[0010] In some embodiments, the method further includes determining that the scope has completed its rotation. Rotating the image representation and rotating the working channel indicator may be performed in response to determining that the scope has completed its rotation.

[0011] In some embodiments, rotating the image representation includes continuously rotating the image representation. Furthermore, in some embodiments, determining the amount the scope has rotated is based on at least one of the following: feedback data from a robotic arm configured to be coupled to the scope, position data generated by a position sensor associated with the scope, or a control signal generated to control the robotic arm. Furthermore, in some embodiments, rotating the image representation includes displaying the image representation in a cropped form on the user interface while rotating the scope.

[0012] In some embodiments, the method further includes receiving user input from an input device to rotate the scope by a predetermined amount, and controlling the scope to rotate by the predetermined amount. Furthermore, in some embodiments, displaying a working channel indicator includes overlaying a working channel indicator onto an image representation.

[0013] In some embodiments, the disclosure relates to a system comprising a scope including a working channel for detachably receiving a medical instrument, an imaging device configured to be coupled to the scope, and a control circuit communicatively coupled to the scope and the imaging device. The control circuit is configured to receive image data from the imaging device, display an image representation of the image data on a user interface, display a working channel indicator on the user interface, determine the amount the scope has rotated, at least in part on the amount the scope has rotated, rotate the working channel indicator within the user interface, and apply rotation adjustments to the image representation within the user interface, at least in part on the amount the scope has rotated. The working channel indicator indicates the rotation angle of the working channel.

[0014] In some embodiments, the working channel is offset from the longitudinal axis of the scope. Furthermore, in some embodiments, the control circuit is configured to control the scope to rotate in a rotational direction and apply rotational adjustment to the image representation by rotating the image representation in a rotational direction within the user interface.

[0015] In some embodiments, the control circuit is further configured to determine when the scope has completed its rotation. Rotation adjustments can be applied upon determination that the scope has completed its rotation. Furthermore, in some embodiments, the control circuit is configured to apply rotation adjustments by continuously rotating the image representation as the scope rotates.

[0016] In some embodiments, the control circuit is further configured to determine the amount by which the scope has rotated based on at least one of the following: feedback data from a robotic arm configured to be coupled to the scope, sensor data generated by a position sensor associated with the scope, or a control signal generated to control the robotic arm.

[0017] In some embodiments, the control circuit is configured to apply rotation adjustment by rotating the scope and displaying the image representation in a cropped form on the user interface. Furthermore, in some embodiments, the control circuit is configured to receive an input control signal for rotating the working channel and to control the scope to rotate based at least in part on the input control signal.

[0018] In some embodiments, the Disclosure relates to one or more non-transient computer-readable media that store computer-executable instructions, which, when executed by a control circuit, cause the control circuit to perform operations including: receiving image data from an imaging device associated with a scope; displaying an image representation of the data on a user interface; displaying a working channel indicator on the user interface; determining an amount by which the scope has rotated, at least in part on the amount by which the scope has rotated; rotating the image representation within the user interface; and rotating the working channel indicator within the user interface from a first orientation to a second orientation. The scope includes a working channel. The working channel indicator indicates the rotation angle of the working channel.

[0019] In some embodiments, the operation further includes determining that the scope has completed its rotation. Upon determining that the scope has completed its rotation, the image representation and the working channel indicator can be rotated. Furthermore, in some embodiments, rotating the image representation includes rotating the image representation continuously.

[0020] In some embodiments, determining the amount of rotation of the scope includes tracking the amount of rotation of the scope based on at least one of the following: feedback data from a robotic arm configured to be coupled to the scope, sensor data generated by a position sensor associated with the scope, or control signals generated to control the robotic arm. Furthermore, in some embodiments, rotating the image representation includes displaying the image representation in a cropped form on the user interface while rotating the scope.

[0021] In some embodiments, the operation further includes receiving user input from an input device to rotate the scope by a predetermined amount, and controlling the scope to rotate by the predetermined amount. Furthermore, in some embodiments, the working channel indicator is displayed as an overlay on the image representation.

[0022] For the purpose of summarizing this disclosure, specific embodiments, advantages, and features have been described. It should be understood that not all such advantages can necessarily be realized by any particular embodiment. Therefore, the disclosed embodiments may realize or optimize one or more advantages or groups of advantages taught herein without necessarily realizing other advantages that may be taught or suggested herein. [Brief explanation of the drawing]

[0023] Various embodiments are shown in the accompanying drawings for illustrative purposes, but should not be construed as limiting the scope of this disclosure. In addition, various features of different disclosed embodiments can be combined to form further embodiments that are part of this disclosure. Throughout the drawings, reference numbers may be reused to indicate correspondences between reference elements. [Figure 1] This figure illustrates an example of a medical system for performing various medical procedures using one or more embodiments. [Figure 2] This figure illustrates detailed examples of the control system and robot system shown in Figure 1, according to one or more embodiments. [Figure 3] This figure illustrates exemplary scopes positioned in a part of a patient's urinary tract according to one or more embodiments. [Figure 4-1] This figure illustrates an example of one or more embodiments in which a scope is rotated and rotation adjustments are applied to the scope image in the user interface when the scope has completed its rotation. [Figure 4-2]A diagram illustrating an example of rotating a scope and applying a rotation adjustment to a scope image in a user interface when the scope has completed its rotation, according to one or more embodiments. [Figure 4-3] A diagram illustrating an example of rotating a scope and applying a rotation adjustment to a scope image in a user interface when the scope has completed its rotation, according to one or more embodiments. [Figure 4-4] A diagram illustrating an example of rotating a scope and applying a rotation adjustment to a scope image in a user interface when the scope has completed its rotation, according to one or more embodiments. [Figure 4-5] A diagram illustrating an example of rotating a scope and applying a rotation adjustment to a scope image in a user interface when the scope has completed its rotation, according to one or more embodiments. [Figure 4-6] A diagram illustrating an example of rotating a scope and applying a rotation adjustment to a scope image in a user interface when the scope has completed its rotation, according to one or more embodiments. [Figure 4-7] A diagram illustrating an example of rotating a scope and applying a rotation adjustment to a scope image in a user interface when the scope has completed its rotation, according to one or more embodiments. [Figure 4-8] A diagram illustrating an example of rotating a scope and applying a rotation adjustment to a scope image in a user interface when the scope has completed its rotation, according to one or more embodiments. [Figure 4-9] A diagram illustrating an example of rotating a scope and applying a rotation adjustment to a scope image in a user interface when the scope has completed its rotation, according to one or more embodiments. [Figure 5-1] A diagram illustrating an example of rotating a scope and applying a rotation adjustment to a scope image in a user interface while the scope is rotating, according to one or more embodiments. [Figure 5-2]This figure illustrates an example of one or more embodiments of rotating a scope and applying rotation adjustments to the scope image in the user interface while the scope is rotating. [Figure 5-3] This figure illustrates an example of one or more embodiments of rotating a scope and applying rotation adjustments to the scope image in the user interface while the scope is rotating. [Figure 5-4] This figure illustrates an example of one or more embodiments of rotating a scope and applying rotation adjustments to the scope image in the user interface while the scope is rotating. [Figure 5-5] This figure illustrates an example of one or more embodiments of rotating a scope and applying rotation adjustments to the scope image in the user interface while the scope is rotating. [Figure 6] This figure illustrates exemplary user interfaces for assisting a user in controlling a scope and / or a device deployed through the scope's work channels, according to one or more embodiments. [Figure 7] An exemplary flowchart illustrates a process for applying rotation adjustment to a scoped image within a user interface, according to one or more embodiments. [Modes for carrying out the invention]

[0024] The headings provided herein are for convenience only and do not necessarily affect the scope or meaning of the disclosure. Specific embodiments and examples are disclosed below, but the subject matter extends beyond the specifically disclosed embodiments to other alternative embodiments and / or uses, as well as modifications and their equivalents. Therefore, the claims that may arise from this specification are not limited by any of the specific embodiments described below. For example, in any method or process disclosed herein, the actions or operations of the method or process may be performed in any preferred order and are not necessarily limited to any specific disclosed order. Various operations may be described sequentially as a number of distinct operations in a manner that may be helpful in understanding a particular embodiment. However, the order of description should not be construed as meaning that these operations are order-dependent. Furthermore, the structures, systems, and / or devices described herein may be embodied as integrated components or separate components. For the purpose of comparing various embodiments, specific aspects and advantages of these embodiments are described. Not all such aspects or advantages are necessarily realized by any particular embodiment. Therefore, for example, various embodiments can be implemented in a manner that realizes or optimizes one or more advantages or groups of advantages as taught herein, without necessarily realizing other embodiments or advantages that may also be taught or suggested herein.

[0025] Specific standard anatomical terms for location may be used herein to refer to animal (i.e., human) anatomical structures in relation to preferred embodiments. Certain spatially relative terms, such as “lateral,” “medial,” “super,” “subordinate,” “downward,” “upper,” “vertical,” “horizontal,” “apex,” “base,” and similar terms, are used herein to describe the spatial relationship of one device / element or anatomical structure to another, but it should be understood that these terms are used herein to simplify the description of the positional relationships between elements / structures, as illustrated in the drawings. It should be understood that spatially relative terms are intended to encompass different orientations of elements / structures in use or operation, in addition to the orientations shown in the drawings. For example, when an element / structure is described as being “above” another element / structure, it may also represent a position below or laterally to such other element / structure with respect to the patient or alternating orientation of the elements / structures, and vice versa.

[0026] Overview Many medical procedures involve the use of a scope to examine and / or treat a patient. A scope generally includes, or can be associated with, an imaging device configured to capture image data of a target site, such as a camera at the distal end of the scope. Image data can be displayed to the user to assist in controlling the scope within the patient. The scope 40 may include one or more working channels through which further tools, such as lithotomizers, basket devices, forceps, and laser devices, can be deployed to the target site. The working channels are often offset from the longitudinal axis of the imaging device and / or the scope. Furthermore, in some cases, the scope can be configured to move based on different articular planes, resulting in the scope moving with more / fewer degrees of freedom in the direction when oriented in a particular manner.

[0027] To accommodate various scope movements, treatment angles, physician preferences, and / or other circumstances, the scope may be controlled to rotate within the patient, and the associated imaging device and working channel may be rotated. In one example, the user may rotate the scope so that the tools / instruments deployed through the working channel (offset to one side of the scope) are in a more optimal position for treating the target site. In another example, the user may rotate the scope to reorient the scope's articulation plane and facilitate further movement of the scope in a particular direction. Since the user often controls the scope based on the image data captured by the scope, rotating the scope can cause the user to lose their sense of direction within the patient. For example, the image data displayed to the user rotates as the scope rotates, causing the user to lose track of a virtual horizontal / reference frame at the treatment site.

[0028] This disclosure relates to systems, devices, and methods for providing image data and / or other data associated with a medical device in an orientation suitable for assisting a user in performing a procedure. For example, the system may be configured to communicate with a scope that includes a work channel for deploying a medical device. The system may receive image data from an imaging device associated with the scope and display an image representation of the image data via a user interface. As the scope rotates, the system may adjust the orientation of the image representation, such as by rotating the image representation within the user interface, which may or may not be perceived as being done within the user interface. The system may also display a work channel indicator via the user interface that indicates the rotation angle of the work channel (e.g., the position of the work channel relative to the image representation). As the scope rotates, the system may adjust the orientation of the work channel indicator within the user interface (e.g., by rotating the work channel indicator) to indicate the precise orientation of the work channel relative to the image representation. In some cases, the image representation and / or the work channel indicator are continuously adjusted within the user interface while the scope rotates, giving the perception that a virtual horizon / reference frame (such as one displayed within the image representation) is maintained. For example, a user may perceive that only the working channel indicator moves within the user interface (e.g., the working channel indicator rotates while the virtual horizon in the image representation remains the same). In other cases, the image representation and / or the working channel indicator rotate within the user interface when the scope completes its rotation, or otherwise reaches a specific rotation angle, and / or when another event occurs. Thus, this technique can help the user maintain a reference frame within the patient by presenting image data, indicators for the working channel, and / or other data associated with the scope in an appropriate orientation.

[0029] While many techniques are discussed in the context of presenting working channel indicators for medical devices, this technique can, or alternatively, present indicators for other elements. For example, this technique can present indicators for articular planes associated with medical devices, lights associated with medical devices, needles / catheters / other devices that work in conjunction with medical devices, etc.

[0030] In some embodiments, the techniques described herein implement robot-assisted medical procedures, where the robotic tool may enable a physician to perform endoscopic and / or percutaneous access, and / or endoscopic and / or percutaneous procedures, for a target anatomical site. For example, the robotic tool may engage with and / or control one or more medical instruments, such as a scope, to access a target site within a patient and / or perform a procedure at the target site. In some cases, the robotic tool is guided / controlled by a physician. In other cases, the robotic tool operates automatically or semi-automatically. While many techniques are discussed in the context of robot-assisted medical procedures, these techniques may also be applicable to other types of medical procedures, such as procedures that do not implement a robotic tool, or that implement a robotic tool only for a relatively small number of movements (e.g., below a threshold number). For example, the techniques may be applicable to procedures that implement manually operated medical instruments, such as a manual scope, which are fully controlled by a physician.

[0031] Certain aspects of this disclosure are described herein in the context of kidney, urinary tract, and / or kidney procedures, such as kidney stone removal / treatment procedures. However, such contexts are provided for convenience, and it should be understood that the concepts disclosed herein are applicable to any suitable medical procedure. For example, the following descriptions are also applicable to other surgical / medical operations or medical procedures relating to the removal of objects from a patient, including any objects that can be removed percutaneously and / or via endoscopic access from the treatment site or the patient's body cavity (e.g., esophagus, ureter, intestine, eye, etc.), such as gallstone removal, lung (lung / transthoracic) tumor biopsy, or cataract removal. However, as stated above, a description of the anatomical structure of the kidney / urinary tract and related medical issues and procedures is provided below to help illustrate the concepts disclosed herein.

[0032] Examples of healthcare systems Figure 1 illustrates an example of a medical system 100 for performing various medical procedures according to embodiments of this disclosure. The medical system 100 includes a robotic system 110 configured to engage with and / or control one or more medical instruments / devices to perform procedures on a patient 120. In the example of Figure 1, the robotic system 110 is coupled to a scope 130 and an electromagnetic (EM) field generator 140. The medical system 100 also includes a control system 150 that interfaces with the robotic system 110 and is configured to provide information about the procedure and / or perform various other actions. For example, the control system 150 may include a display 152 configured to present specific information to assist a physician 160 when performing a procedure. The medical system 100 may include a table 170 (e.g., a bed) configured to hold the patient 120. Various actions are described herein as being performed by a physician 160. These actions may be performed directly by the physician 160, a user under the direction of the physician 160, another user (e.g., a technician), a combination thereof, and / or any other user. The devices / components of the medical system 100 can be arranged in various ways depending on the type and / or stage of the procedure.

[0033] The control system 150 can work in cooperation with the robot system 110 to perform medical procedures on the patient 120. For example, the control system 150 can communicate with the robot system 110 via wireless or wired connection to control medical devices connected to the robot system 110 and receive images captured by the medical devices. For example, the control system 150 can receive image data from the scope 130 (e.g., an imaging device associated with the scope 130) and display the image data (and / or a representation thereof) to the physician 160 to assist the physician 160 in navigating the scope 130 within the patient 120. The physician 160 can provide input via an input / output (I / O) device such as a controller 154, and the control system 150 can send control signals to the robot system 110 to control the movement of the scope 130 connected to the robot system 110. The scope 130 (and / or other medical devices) can be configured to move in various ways, such as articulating in a plane, rotating, etc., as will be discussed in more detail below.

[0034] In some embodiments, the control system 150 may provide fluid to the robot system 110 via one or more fluid channels, provide power to the robot system 110 via one or more electrical connections, or provide optics to the robot system 110 via one or more optical fibers or other components. In an example, the control system 150 may communicate with a medical device to receive sensor data (via the robot system 110 and / or directly from the medical device). The sensor data may indicate or be used to determine the position and / or orientation of the medical device. Furthermore, in an example, the control system 150 may communicate with a table 170 to position the table 170 in a specific orientation or otherwise control the table 170. Furthermore, in an example, the control system 150 may communicate with an EM field generator 140 to control the generation of an EM field around the patient 120.

[0035] The robotic system 110 may include one or more robotic arms 112 configured to engage with or control a medical instrument / device. Each robotic arm 112 may include multiple arm segments coupled to a joint, thereby providing multiple degrees of movement. The distal end of a robotic arm 112 (e.g., an end effector) may be configured to couple to an instrument / device. In the example of Figure 1, robotic arm 112(A) is coupled to an EM field generator 140, which may be configured to generate an EM field detected by a sensor on the medical instrument, such as an EM sensor on the scope 130. A second robotic arm 112(B) is coupled to a scope-driver instrument coupling 132, which can facilitate robotic control / advancement of the scope 130. Furthermore, a third robotic arm 112(C) is coupled to a handle 134 of the scope 130, which may be configured to facilitate the advancement and / or movement of medical instruments that can be deployed through the scope 130, such as instruments deployed through the working channels of the scope 130. In this example, the second robotic arm 112(B) and / or the third robotic arm 112(C) can control the movement of the scope 130 (e.g., joint movement, rotation, etc.). Although three robotic arms are connected to a specific medical instrument in Figure 1, the robotic system 110 can include any number of robotic arms configured to connect to any type of medical instrument / device.

[0036] The robot system 110 can be communicatively coupled to any component of the medical system 100. For example, in one example, the robot system 110 can be communicatively coupled to a control system 150 to receive control signals from the control system 150 and perform actions such as controlling a robotic arm 112 in a specific manner or operating medical instruments. Furthermore, the robot system 110 is configured to receive images (also referred to as image data) showing the internal anatomical structure of a patient 120 from a scope 130 and / or transmit the images to the control system 150, where the images can then be displayed on a display 152. In addition, the robot system 110 is coupled to components of the medical system 100, such as the control system 150, in a manner that allows it to receive fluids, optics, power, etc., from the components.

[0037] Medical devices can include various types of instruments such as scopes (sometimes referred to as “endoscopes”), catheters, needles, guidewires, lithotomizers, basket retrieval devices, forceps, vacuums, needles, surgical scalpels, imaging probes, imaging devices, grippers, scissors, capture devices, needle holders, micro-dissection instruments, staple applicators, tackers, suction / irrigation tools, and clip applicators. Medical devices can include direct penetration instruments, percutaneous penetration instruments, and / or other types of instruments. In some embodiments, medical devices are maneuverable devices, while in other embodiments, medical devices are non-maneuverable devices. In some embodiments, surgical tools refer to devices such as needles, surgical scalpels, and guidewires that are configured to puncture or be inserted through human tissue structures. However, surgical tools can also refer to other types of medical devices.

[0038] The terms “scope” or “endoscope” can refer to any type of elongated medical instrument having image generation, visualization, and / or acquisition functions (or configured to provide such functions using an imaging device deployed through a working channel) and configured to be introduced into any type of organ, body cavity, lumen, chamber, and / or space of the body. For example, a scope or endoscope such as Scope 130 could refer to a ureteroscope (for accessing the urinary tract), a laparoscope, a nephroscope (for accessing the kidneys), a bronchoscope (for accessing the airways such as the bronchi), a colonoscope (for accessing the colon), an arthroscope (for accessing joints), a cystoscope (for accessing the bladder), a borescope, etc. A scope / endoscope may, in some cases, include a rigid or flexible tube and may be sized to pass through an outer sheath, catheter, introducer, or other tubular device, or may be used without such device. In some embodiments, the scope includes one or more working channels into which further tools / medical instruments such as lithotomizers, basket devices, forceps, laser devices, and imaging devices can be introduced into the treatment site. In some cases, the working channels are offset from the longitudinal axis of the imaging device and / or the scope. Furthermore, in some cases, the scope may be configured to move based on different articular planes, thereby allowing the scope to move with more / fewer degrees of freedom in a particular direction when oriented in a particular manner. For example, the scope may be associated with a primary articular plane and a secondary articular plane, and the tip of the scope may be configured to move with a greater degree of freedom (e.g., more deflection, a greater bending radius, etc.) in the primary articular plane compared to the secondary articular plane. The primary articular plane may be, for example, perpendicular to the secondary articular plane.

[0039] The terms “direct penetration” or “direct access” can refer to any penetration of an instrument through a natural or artificial orifice within a patient’s body. For example, since Scope 130 penetrates the patient’s urinary tract through the urethra, Scope 130 may be referred to as a direct access instrument.

[0040] The term “percutaneous penetration” or “percutaneous access” can refer to the penetration of an instrument through the patient’s skin and any other body layer necessary to reach a target anatomical site associated with the procedure (e.g., the renal calyx retina of the kidney), such as through puncture and / or a small incision. Thus, percutaneous access instruments can refer to medical devices, instruments, or assemblies configured to puncture or be inserted through the skin and / or other tissues / anatomical structures, such as needles, surgical scalpels, guidewires, sheaths, shafts, scopes, and catheters. However, it should be understood that percutaneous access instruments can refer to other types of medical devices in the context of this disclosure. In some embodiments, percutaneous access instruments refer to instruments / devices inserted or implemented by a device that facilitates puncture and / or a small incision through the patient’s skin. For example, a catheter may be referred to as a percutaneous access instrument when it is inserted through a sheath / shaft that punctures the patient’s skin.

[0041] In some embodiments, the medical device includes a sensor (also referred to as a position sensor) configured to generate sensor data. In an example, the sensor data may indicate the position and / or orientation of the medical device and / or can be used to determine the position and / or orientation of the medical device. For example, the sensor data may indicate the position and / or orientation of a scope, which may include rotation of the distal end of the scope. The position and orientation of the medical device may be referred to as the posture of the medical device. The sensor may be positioned at the distal end of the medical device and / or any other location. In some embodiments, the sensor may provide sensor data to a control system 150, a robotic system 110, and / or another system / device so that one or more positioning techniques can be performed to determine / track the position and / or orientation of the medical device.

[0042] In some embodiments, the sensor may include an electromagnetic (EM) sensor with a coil of conductive material. Here, an EM field generator, such as EM field generator 140, can provide an EM field that is detected by the EM sensor on the medical device. The magnetic field can induce a small current in the coil of the EM sensor, and by analyzing this small current, the distance and / or angle / orientation between the EM sensor and the EM field generator can be determined. Furthermore, the sensor may include other types of sensors such as cameras, distance sensors (e.g., depth sensors), radar devices, shape-sensing fibers, accelerometers, gyroscopes, satellite-based positioning sensors (e.g., Global Positioning System (GPS)), and radio frequency transceivers.

[0043] In some embodiments, the medical system 100 may also include an imaging device (not illustrated in Figure 1) which may be integrated with a C-arm and / or configured to take images during a procedure, such as in the case of a fluoroscopy-type procedure. The imaging device may be configured to capture / generate one or more images of the patient 120 during the procedure, such as one or more X-ray or CT images. In an example, images from the imaging device may be provided in real time to allow the physician 160 to visualize anatomical structures and / or medical instruments within the patient 120 to assist in performing the procedure. The imaging device may be used to perform fluoroscopy or another type of imaging technique (e.g., with a contrast agent within the patient 120).

[0044] Various components of the medical system 100 can be connected to communicate with one another via a network that may include wireless and / or wired networks. Exemplary networks include one or more personal area networks (PANs), local area networks (LANs), wide area networks (WANs), Internet area networks (IANs), cellular networks, and the Internet. Furthermore, in some embodiments, the components of the medical system 100 are connected via one or more support cables, tubes, etc., for purposes such as data communication, fluid / gas exchange, and power exchange.

[0045] In some embodiments, the medical system 100 can provide information to assist the physician 160 in controlling the scope 130 and / or tools deployed through the scope 130. For example, the control system 150 can display an image representation 154 within an interface 156, which includes or is based on image data captured by an imaging device associated with the scope 130. The imaging device can be positioned on the distal end of the scope 130, deployed from the distal end of the scope 130 through the working channel of the scope 130, or otherwise associated with the scope 130. In the example in Figure 1, the image representation 154 shows an internal view of the kidney 190 of patient 120 (i.e., from the viewpoint of the scope 130 within the renal calyces 191 of the kidney 190). Furthermore, the control system 150 can provide a working channel indicator 158 that shows the rotation angle of the working channel of the scope 130 relative to the image representation 154. For example, the location of the working channel indicator 158(A) within interface 156(A) indicates that the working channel is positioned to the right of Figure 1 relative to the image representation 154(A). Although not shown in Figure 1, the control system 150 can also provide other types of indicators, such as an indicator for the orientation of the articular plane of the scope 130, an indicator for the orientation of the light associated with the scope 130, and an indicator for the position / orientation of needles / catheters / other instruments working in conjunction with the scope 130.

[0046] In some embodiments, the control system 150 can adjust the image representation 154 and / or the working channel indicator 158 to reflect changes in the orientation of the scope 130. For example, the control system 150 can rotate the working channel indicator 158 when the scope 130 rotates to accurately indicate the orientation of the working channel relative to the image representation 154. Furthermore, the control system 150 can adjust the orientation of the image representation 154 to provide the same virtual horizontal line / reference frame before and after rotation. For example, the control system 150 can rotate the image representation 154 when the scope 130 rotates so that the view of the internal anatomical structures of the patient 120 has the same orientation before and after the scope 130 rotates.

[0047] For illustrative purposes, assume that, as shown in Figure 1, the scope 130 is positioned within the kidney 190 in close proximity to the kidney stone 192 and includes a laser device that is deployed through a working channel to break up the kidney stone 192. Also assume that the scope 130 is initially positioned in a first orientation where the laser from the laser device may diverge from the kidney stone 192 (i.e., extend along a path 193 that contacts the anatomical structures of the patient 120). Here, the control system 150 can display an interface 156(A) having a working channel indicator 158(A) positioned on the right side. To adjust the orientation of the laser device, the physician 160 can rotate the scope 130 to a second orientation where the laser from the laser device may contact the kidney stone 192 (i.e., extend along a path 194). In this figure, the scope 130 is rotated 180 degrees. However, the scope 130 can be rotated by any angle. Next, the control system 150 can present an updated user interface 156(B) including an updated working channel indicator 158(B). The control system 150 can also rotate the image data 156 by 180 degrees, as shown in the updated user interface 156(B), to present the same horizontal line within the image data 156. That is, the control system 150 can adjust the orientation of the image data 156 in the user interface 156 so that the reference frame within the patient 120 is maintained (for example, so that the kidney stone 192 appears on the left side both before and after rotating the scope 130). The reference frame within the patient 120 can be maintained by adjusting the image representation 154 and / or the working channel indicator 158 in this way. In an alternative solution, such rotation of the scope 130 could rotate the image representation 154 by 180 degrees from its initial orientation, potentially disorienting the physician 160.

[0048] In some embodiments, the control system 150 may apply orientation adjustments to the image representation 154 (and / or working channel indicator 158) after the scope 130 has completed its rotation. For example, upon determining that the scope 130 has rotated 180 degrees, the control system 150 may rotate the image representation 154 to the view shown in the user interface 156(B). Here, the physician 160 can perceive the content within the image representation 154 as having rotated 180 degrees following the rotation of the scope 130, and then rotating 180 degrees back. Examples of such orientation adjustments are described below with reference to Figures 4-1 to 4-9. Alternatively, in some embodiments, the control system 150 may continuously update the user interface 156 to maintain a reference frame within the patient 120. For example, the control system 150 may track the rotation of the scope 130 and continuously update the image representation 154 to maintain a virtual horizon within the patient 120 throughout the entire rotation of the scope 130. Here, the physician 160 can perceive that the image representation 154 remains the same while the working channel indicator 158 rotates around the image representation 154. Examples of such continuous orientation adjustments are described below with reference to Figures 5-1 to 5-5.

[0049] In some cases, a medical system 100 is implemented to perform medical procedures related to the anatomical structure of the kidney, such as treating kidney stones. Kidney stone disease, also known as urolithiasis, is a medical condition that involves the formation of solid fragments of a substance in the urinary tract, which are called “kidney stones,” “urinary tract stones,” “kidney stones,” “urolithiasis,” or “urolithiasis.” Urinary tract stones may form and / or be found in the kidneys, ureters, and bladder (called “bladder stones”). Such urinary tract stones may form as a result of the concentration of minerals in the urine, and such stones can cause significant abdominal pain when they grow large enough to obstruct the flow of urine through the ureters or urethra. Urinary tract stones may form from calcium, magnesium, ammonia, uric acid, cysteine, and / or other compounds, or combinations thereof.

[0050] In general, there are several methods for treating patients with kidney stones. These include observation, medical intervention (such as expulsion therapy), non-invasive procedures (such as extracorporeal shock wave lithotripsy, ESWL), and surgical treatment (such as ureteroscopy and percutaneous nephrolithotomy, PCNL). In the surgical approach (e.g., ureteroscopy and PCNL), the physician accesses the lesion (i.e., the object to be removed, e.g., the stone) and breaks the stone into smaller pieces or fragments, mechanically removing the relatively small stone fragments / microparticles from the kidney.

[0051] To remove urinary tract stones from the bladder and ureters, a physician may insert a ureteroscope into the urinary tract through the urethra. Typically, the ureteroscope includes an endoscope at its distal end configured to allow visualization of the urinary tract. The ureteroscope may also include a lithotomy device for capturing or fragmenting urinary tract stones. During the procedure, one physician / technician may control the position of the ureteroscope while another physician / technician controls the lithotomy device. To remove relatively large stones (i.e., “kidney stones”) from the kidney, a physician may use the percutaneous nephrolithotomy (“PCNL”) technique. This involves inserting a nephroscope through the skin (i.e., percutaneously) and intervening tissue to provide access to the treatment site for fragmenting and / or removing the stones.

[0052] In some of the examples described herein, robot-assisted percutaneous procedures can be performed in connection with various medical procedures, such as kidney stone removal, and robotic tools (e.g., one or more components of medical system 100) can enable physicians / urologists to perform endoscopic (e.g., ureteroscopy) targeted access and percutaneous access / procedures. However, this disclosure is not limited to kidney stone removal and / or robot-assisted procedures. In some embodiments, robotic medical solutions can provide relatively high precision, superior control, and / or superior eye-hand coordination with respect to specific instruments compared to strictly manual procedures. For example, robot-assisted percutaneous access to the kidney in some procedures can advantageously enable urologists to perform both direct intrusion endoscopic kidney access and percutaneous kidney access. While some embodiments of this disclosure are presented in the context of catheters, nephroscopes, ureteroscopes, and / or the anatomical structures of the human kidney, it should be understood that the principles disclosed herein can be implemented in any type of endoscopic / percutaneous procedure or another type of procedure.

[0053] In one exemplary and non-limiting procedure, the medical system 100 can be used to remove kidney stones 192 from a patient 120. During setup for the procedure, the physician 160 can position the robotic arm 112 of the robotic system 110 into a desired configuration and / or attach appropriate medical instruments. For example, as shown in Figure 1, the physician 160 can position the first robotic arm 112(A) near the procedure site and attach an EM field generator 140, which can help track the location of the scope 130 and / or other instruments / devices during the procedure. Furthermore, the physician 160 can position the second robotic arm 112(B) between the legs of the patient 120 and attach a scope-driver instrument coupling 132, which can facilitate robotic control / advancement of the scope 130. The physician 160 can insert a medical instrument 136 into the urethra 195 of the patient 120, through the bladder 196, and / or over the ureter 197. Physician 160 can connect the medical instrument 136 to the scope-driver instrument coupling 132. The medical instrument 136 may include a tubular device configured to receive the scope 130, thereby assisting in the insertion of the scope 130 (e.g., an access sheath) into the anatomical structure of the patient 120. Although the medical instrument 136 is shown in Figure 1, in some embodiments the medical instrument 136 is not used (e.g., the scope 130 is inserted directly into the urethra 195). The physician 160 can then insert the scope 130 into the medical instrument 136 manually, robotically, or a combination thereof. The physician 160 may attach the handle 134 of the scope 130 to a third robotic arm 112(C), which may be configured to facilitate the advancement and / or movement of a basket device, laser device, and / or other medical instrument deployed through the scope 130.

[0054] The physician 160 can interact with the control system 150 to cause the robotic system 110 to advance and / or navigate the scope 130 into the kidney 190. For example, the physician 160 can navigate the scope 130 to locate the kidney stone 192. The control system 150 can provide information about the scope 130 via the display 152 to assist the physician 160 in navigating the scope 130, such as an interface 156 for viewing image representations 154 (e.g., real-time images captured by the scope 130), a working channel indicator 158 (e.g., showing the current orientation of the working channel), etc. In some embodiments, the control system 150 can use localization techniques to determine the position and / or orientation of the scope 130, which can be viewed by the physician 160 via the display 152. Furthermore, other types of information, such as X-ray images of the patient's internal anatomical structures, can also be presented via the display 152 to assist the physician 160 when controlling the scope 130.

[0055] When the scope 130 reaches the site of the kidney stone 192 (for example, within the renal calyces 191 of the kidney 190), the scope 130 can be used to designate / tag a target site for the catheter to percutaneously access the kidney 190. To minimize damage to the kidney 190 and / or surrounding anatomical structures, the physician 160 may designate the papilla as a target site for percutaneous entry into the kidney 190 using the catheter. However, other target sites can be designated or determined. In some embodiments where the papilla 1512 is designated, the physician 160 can navigate the scope 130 to make contact with the papilla, and the control system 150 can determine the location of the scope 130 (for example, the location of the distal end of the scope 130) using localization techniques, and the control system 150 can associate the location of the scope 130 with the target site. Furthermore, in some embodiments, the physician 160 can navigate the scope 130 so that it is within a specific distance of the nipple (e.g., placed in front of the nipple) and provide input indicating that the target location is within the field of view of the scope 130. The control system 150 may perform image analysis and / or other localization techniques to determine the location of the target location. Furthermore, in some embodiments, the scope 130 can deliver a reference point for marking the nipple as the target location.

[0056] Once a target location is specified, a catheter (not shown) can be inserted into the patient 120 through a percutaneous access pathway to reach the target site. For example, the catheter can be connected to a first robotic arm 112(A) (when the EM field generator 140 is removed), and the physician 160 can interact with a control system 150 to cause the robotic system 110 to advance and / or navigate the catheter. In some embodiments, a needle or another medical device is inserted into the patient 120 to form a percutaneous access pathway. The control system 150 can provide information about the catheter via a display 152 to assist the physician 160 in navigating the catheter. For example, the interface can provide image data from the viewpoint of a scope 130. The image data may show the catheter (for example, when it is within the field of view of the imaging device of the scope 130).

[0057] Once the scope 130 and / or catheter are positioned at the target site, the physician 160 can use the scope 130 to break up the kidney stone 192 and / or use the catheter to remove the fragments of the kidney stone 192 from the patient 120. For example, the scope 130 can deploy a tool (e.g., a laser, cutting instrument, etc.) through its working channel to fragment the kidney stone 192 into fragments, and the catheter can aspirate the fragments from the kidney 190 through a percutaneous access route. In this example, the catheter and / or scope 130 can provide irrigation and / or aspiration to facilitate the removal of the kidney stone 192. For example, the catheter (and / or associated medical device) can be coupled to an irrigation and / or aspiration system.

[0058] At any point before, during, or after a procedure, the physician 160 can rotate the scope 130 to provide a more optimal position / orientation of the scope 130. In one example, when positioning the scope 130 to break up / remove a kidney stone 192, the physician 160 can rotate the scope 130 to reposition the working channel of the scope 130, thereby providing a more optimal orientation for deploying the instrument through the working channel. In another example, the scope 130 can be associated with different articular planes (e.g., a primary plane and a secondary plane). Here, the physician 160 can control the scope 130 to rotate to reposition the articular plane, resulting in the scope 130 being able to articulate in a more optimal manner (e.g., flexing with more deflection). In some cases, the physician 160 can provide input to rotate the scope 130 by a predetermined amount, such as reversing the working channel by 180 degrees. In other examples, the physician 160 can provide input to rotate the scope 130 by any angle. In either case, as the scope 130 rotates, the control system 150 can update the working channel indicator 158 in the user interface 156 to maintain the precise orientation of the working channel indicator 158 relative to the image representation 154. Furthermore, the control system 150 can apply orientation adjustments to the image representation 154 to maintain the reference frame of the image representation 154. This allows the physician 160 to maintain orientation within the kidney 190 of the patient 120 and to visually confirm the precise orientation of the working channel relative to the image representation 154 / user interface.

[0059] In some embodiments, the control system 150 and / or the robot system 110 prevent the scope 130 from rotating (and thus the working channel from rotating) when the scope 130 is articulated beyond a certain degree (e.g., 20 degrees, 30 degrees, etc.) from the longitudinal axis. For example, when a user input is received to reverse the working channel of the scope 130, the control system 150 and / or the robot system 110 can prevent the working channel from reversing if the scope 130 is articulated beyond 30 degrees. However, in some cases, the scope 130 can rotate when articulated by any amount.

[0060] While many examples are described in relation to displaying data associated with a scope (e.g., displaying image data associated with a scope, displaying working channel indicators associated with a scope, etc.), this technique can be implemented in relation to other medical devices. For example, this technique can be implemented to present information associated with another type of medical device, including / deploying an imaging device.

[0061] Furthermore, while many examples present indicators for the working channel, other types of indicators can be presented. For example, this technique can display articular plane indicators showing the orientation of the articular plane of a medical instrument relative to the image representation within the user interface, light indicators for lights positioned at the distal end of a medical instrument (which can show the orientation of the light relative to the image representation), and needle / catheter / instrument indicators for needles / catheters / other instruments working in conjunction with the medical instrument (which can show the orientation of the needle / catheter / instrument relative to the image representation). Such indicators can be oriented / updated within the user interface as the scope / medical instrument rotates to accurately indicate the orientation of the item represented by the indicator. In some cases, needles / catheters / instruments working with the scope may be outside the scope's field of view and therefore not displayed in the image representation from the scope's viewpoint (e.g., the needle / catheter / instrument is off-screen). In this case, the needle / catheter / instrument indicator can still be maintained in the appropriate orientation within the image representation before and after the scope is rotated. For example, a needle / catheter / instrument indicator can be positioned on the right side of the user interface (indicating that the needle / catheter / instrument is off-screen to the right). Such positioning of the needle / catheter / instrument indicator can be the same both before and after rotating the scope, assuming the needle / catheter / instrument does not move. In this example, position / orientation information regarding the needle / catheter / instrument can be obtained based on position sensors within the needle / catheter / instrument, such as an EM sensor or shape detection.

[0062] In many examples, the rotation of the scope 130 is controlled by the robotic arm 112, but the robotic arm 112 remains in a relatively stationary position (for example, the end effector of the robotic arm 112 can remain in the same position). For example, the robotic arm 112 can control / drive rotational features / mechanisms on the scope 130 to rotate the scope 130. However, in some cases, the scope 130 can be rotated by rotating the end effector of the robotic arm 112. For example, the end effector can rotate around the longitudinal axis of the scope 130, so that the handle 134 of the scope 130 rotates around the longitudinal axis.

[0063] The medical system 100 offers various benefits, such as providing guidance to assist physicians in performing procedures (e.g., instrument tracking, instrument navigation, instrument calibration), enabling physicians to perform procedures from ergonomic positions without requiring skilled arm movements and / or positioning, allowing one physician to perform procedures using one or more medical devices, avoiding radiation exposure (e.g., associated with fluoroscopy techniques), performing procedures in a single surgical setting, and providing continuous aspiration for more efficient removal of objects (e.g., kidney stone removal). For example, the medical system 100 provides guidance information to assist physicians in accessing target anatomical features using various medical devices while minimizing bleeding and / or damage to anatomical structures (e.g., diagnostic organs, blood vessels, etc.). Furthermore, the medical system 100 can provide non-radiation-based navigation and / or localization techniques to reduce physician and patient radiation exposure and / or reduce the amount of equipment in the operating room. In addition, the medical system 100 can provide distributed functions between at least the control system 150 and the robotic system 110, thereby enabling independent mobility. Such distribution of functions and / or mobility can enable the control system 150 and / or robotic system 110 to be positioned in locations that are optimal for specific medical procedures, thereby maximizing the work area around the patient and / or providing an optimized location for the physician to perform the procedure.

[0064] While various techniques and systems are discussed as being implemented as robot-assisted procedures (e.g., procedures using medical system 100 in at least part), these techniques and systems can also be implemented in other procedures, such as fully robotic medical procedures or human-only procedures (e.g., procedures that do not include a robotic system). For example, medical system 100 can be used to perform procedures (e.g., fully robotic procedures that rely on relatively little input to direct the procedure) without the physician holding / manipulating medical instruments and without the physician controlling the movement of the robotic system / arm. That is, each medical instrument used during the procedure can be held / controlled by a component of medical system 100, such as the robotic arm 112 of robotic system 110.

[0065] Examples of control systems and robot systems Figure 2 illustrates a detailed example of the control system 150 and robot system 110 of Figure 1 in one or more embodiments. While certain components of the control system 150 and / or robot system 110 are shown in Figure 2, it should be understood that further components not shown may be included in embodiments of this disclosure. Furthermore, any of the illustrated components can be omitted, replaced, and / or integrated into other devices / systems such as the table 170, medical instruments, etc.

[0066] Referring to Figure 2, the control system 150 may include one or more components, devices, modules, and / or units (hereinafter referred to as “components”) of one or more I / O components 202, one or more communication interfaces 204, one or more power supply units 206, and / or one or more movable components 208 (e.g., casters or other types of wheels), either separately / individually and / or in combination / collectively. In some embodiments, the control system 150 may have a housing / enclosure configured to house or include at least one or more of the components of the control system 150 and / or a dimensioned housing / enclosure. In this example, the control system 150 is illustrated as a cart-based system that is movable with one or more movable components 208. In some cases, after reaching a suitable position, one or more movable components 208 can be fixed in place using wheel locks to hold the control system 150 in place. However, the control system 150 may be implemented as a fixed system and integrated with another system / device, etc.

[0067] Various components of the control system 150 can be electrically and / or communicatively coupled using specific connecting circuits / devices / functions, which may or may not be part of the control circuit. For example, the connecting function may include one or more printed circuit boards configured to facilitate the mounting and / or interconnection of at least some of the various components / circuits of the control system 150. In some embodiments, two or more components of the control system 150 can be electrically and / or communicatively coupled to one another.

[0068] One or more I / O components / devices 202 may include various components for receiving inputs and / or providing outputs, such as to interface with a user to assist in performing medical procedures. One or more I / O components 202 may be configured to receive touch, speech, gestures, or any other type of input. In the example, one or more I / O components 202 can be used to provide inputs for controlling a device / system, such as controlling a robotic system 110, navigating a scope or other medical instrument attached to (and / or deployed via) the robotic system 110, controlling a table 170, or controlling a fluoroscopy device. For example, a physician 160 (not shown) may provide inputs via the I / O component 202, to which the control system 150 can send control signals to the robotic system 110 to operate the medical instrument. In the example, the physician 160 may use the same I / O device to control multiple medical instruments (e.g., switching control between instruments).

[0069] As illustrated, one or more I / O components 202 may include one or more displays 152 configured to display data (sometimes referred to as "one or more display devices 152"). One or more displays 152 may include one or more liquid crystal displays (LCDs), light-emitting diode (LED) displays, organic LED displays, plasma displays, electronic paper displays, and / or any other type of technology. In some embodiments, one or more displays 152 may include one or more touchscreens configured to receive input and / or display data. Furthermore, one or more I / O components 202 may include one or more I / O devices / controllers 210, which may include touchpads, controllers (e.g., handheld controllers, video game type controllers, tactile interfaces, haptic interfaces, finger-based controls / finger clutches that enable finger-like movements, etc.), mice, keyboards, wearable devices (e.g., optical head-mounted displays), virtual reality devices or augmented reality devices (e.g., head-mounted displays), foot panels (e.g., buttons under the user's feet), etc. Furthermore, one or more I / O components 202 may include one or more speakers configured to output sound based on an audio signal, and / or one or more microphones configured to receive sound and generate an audio signal. In some embodiments, one or more I / O components 202 include or are implemented as a console.

[0070] In some embodiments, one or more I / O components 202 can output information related to the procedure. For example, the control system 150 can receive real-time images captured by the scope and display the real-time images and / or a visual / image representation of the real-time images via the display 152. The display 152 can present an interface such as one of the interfaces described herein, which may include image data from the scope and / or other medical devices. Further or alternatively, the control system 150 can receive signals (e.g., analog signals, digital signals, electrical signals, acoustic / sound wave signals, pneumatic signals, tactile signals, hydraulic signals, etc.) from medical monitors and / or sensors associated with the patient, and the display 152 can present information about the patient's health or the environment. Such information may include, for example, information displayed via a medical monitor, such as heart rate (e.g., ECG, HRV, etc.), blood pressure / blood flow velocity, muscle biosignals (e.g., EMG), body temperature, blood oxygen saturation (e.g., SpO2), CO2, electroencephalogram (e.g., EEG), ambient temperature and / or local or core body temperature.

[0071] One or more communication interfaces 204 can be configured to communicate with one or more devices / sensors / systems. For example, one or more communication interfaces 204 can transmit / receive data wirelessly and / or via a wired method over a network. The network according to embodiments of this disclosure may include a local area network (LAN), a wide area network (WAN) (e.g., the Internet), a personal area network (PAN), a body area network (BAN), and the like. In some embodiments, one or more communication interfaces 204 can implement wireless technologies such as Bluetooth, Wi-Fi, and near-field communication (NFC).

[0072] One or more power units 206 may be configured to manage and / or provide power to the control system 150 (and / or optionally to the robot system 110). In some embodiments, one or more power units 206 include one or more batteries, such as lithium batteries, lead-acid batteries, alkaline batteries, and / or other types of batteries. That is, one or more power units 206 may comprise one or more devices and / or circuits configured to provide power and / or power management functions. Furthermore, in some embodiments, one or more power units 206 include a mains power connector configured to couple to an alternating current (AC) or direct current (DC) mains power supply.

[0073] Although not shown in Figure 2, the control system 150 may include and / or control one or more other components, such as pumps, flow meters, valve control devices, and / or fluid access components, to provide controlled irrigation and / or suction capabilities to a medical device (e.g., a scope), a device that can be deployed through the medical device, etc. In some embodiments, the irrigation and suction capabilities may be delivered directly to the medical device via a separate cable. Furthermore, the control system 150 may include a voltage protector and / or surge protector designed to provide filtered and / or protected power to another device, such as a robotic system 110, thereby avoiding the need to place a power transformer and other auxiliary power components within the robotic system 110, making the robotic system 110 smaller and more mobile.

[0074] In some embodiments, the control system 150 may include support equipment for sensors deployed throughout the medical system 100. For example, the control system 150 may include optoelectronic equipment for detecting, receiving, and / or processing data received from optical sensors and / or cameras. Such optoelectronic equipment can be used to generate real-time images for display on any number of devices / systems, including the control system 150. Similarly, the control system 150 may include an electronic subsystem for receiving and / or processing signals received from deployed electromagnetic (EM) sensors. In some embodiments, the control system 150 may also be used to house and / or position an EM field generator for detection by EM sensors within or on a medical device.

[0075] Furthermore, in some embodiments, the control system 150 can be used to connect to the robot system 110, the table 170, and / or medical instruments via one or more cables or connectors (not shown). In some embodiments, support functions from the control system 150 can be provided via a single cable, simplifying and tidying up the operating room. In other embodiments, specific functions can be coupled via separate cables and connectors. For example, power can be provided via a single power cable, while support for control, optics, fluid mechanics, and / or navigation can be provided via separate cables.

[0076] The robotic system 110 generally includes an elongated support structure 210 (also referred to as the “column”), a robotic system base 212, and a console 214 at the top of the column 210. The column 210 may include one or more carriages 216 (also referred to as “arm support members 216”) for supporting the deployment of one or more robotic arms 112. The carriages 216 may include individually configurable arm mounts that rotate along a vertical axis to adjust the base of the robotic arm 112 for positioning relative to a patient. The carriages 216 also include a carriage interface 218 that allows the carriages 216 to translate vertically along the column 210. The carriage interface 218 may be connected to the column 210 through slots such as slots 220 located on both sides of the column 210 to guide the vertical translation of the carriages 216. The slots 220 may include a vertical translation interface for positioning and / or holding the carriages 216 at various vertical heights relative to the base 212. As the carriage 216 moves vertically, the robot system 110 can adjust the reach of the robot arm 112 to accommodate various table heights, patient sizes, physician preferences, etc. Similarly, the robot arm base 222 of the robot arm 112 can be angled in various configurations by individually configurable arm mounts on the carriage 216. The column 210 may contain a mechanism, such as gears and / or motors, designed to use vertically aligned lead screws to translate the carriage 216 in a mechanized manner in response to control signals generated by user inputs, such as inputs from I / O devices.

[0077] The base 212 can balance the weight of the column 210, carriage 216, and / or robot arm 112 on a surface such as the floor. Thus, the base 212 can accommodate one or more heavier components such as electronics, motors, power supplies, and components that enable and / or fix the movement of the robot system 110. For example, the base 212 may include swivel wheels 224 (also referred to as “casters 224” or “movable components 224”) that allow the robot system 110 to move around the room for treatment. After reaching a suitable position, the casters 224 can be fixed using wheel locks to hold the robot system 110 in place during treatment. As shown, the robot system 110 also includes handles 226 to assist in maneuvering and / or stabilizing the robot system 110. In this example, the robot system 110 is shown as a movable cart-based system. However, the robot system 110 can be implemented as a fixed system or integrated into a table, etc.

[0078] The robotic arm 112 may generally comprise a robotic arm base 222 and an end effector 228, separated by a series of linkage mechanisms 230 (also referred to as “arm segments 230”) connected by a series of joints 232. Each joint 232 may have an independent actuator, and each actuator may have an independently controllable motor. Each independently controllable joint 232 represents an independent degree of freedom available to the robotic arm 112. For example, each arm 112 may have seven joints and thus seven degrees of freedom. However, any number of joints can be implemented with any number of degrees of freedom. In the example, a large number of joints can provide a large number of degrees of freedom, enabling “redundant” degrees of freedom. Redundant degrees of freedom allow the robotic arm 112 to position each end effector 228 to a specific position, orientation, and / or trajectory in space using different linkage mechanism positions and / or joint angles. In some embodiments, the end effector 228 may be configured to engage with and / or control medical instruments, devices, objects, etc. The degrees of freedom of motion of the arm 112 allow the robotic system 110 to position and / or orient a medical device from a desired point in space, and / or allow a physician to move the arm 112 to a clinically convenient position away from the patient, thereby avoiding collisions with the arm and providing access to the device.

[0079] Each end effector 228 of the robotic arm 112 may be equipped with an instrument device manipulator (IDM) that can be attached using a mechanism exchange interface (MCI). In some embodiments, the IDM can be removed and replaced with a different type of IDM. For example, a first type of IDM may be able to operate an endoscope, a second type of IDM may be able to operate a catheter, a third type of IDM may be able to hold an EM field generator, and so on. The MCI may include connectors for transmitting pneumatic, electrical, electrical, and / or optical signals from the robotic arm 112 to the IDM. The IDM 228 may be configured to operate medical instruments (e.g., surgical tools / instruments) using techniques including, for example, direct drive, harmonic drive, gear drive, belt and pulley, magnetic drive, etc. In some embodiments, the IDM 228 may be attached to one of each of the robotic arms 112, and the robotic arm 112 is configured to insert or withdraw the respective coupled medical instrument into or from a treatment site.

[0080] In some embodiments, the robotic arm 112 can be configured to control the position, orientation, and / or articular movement of a medical instrument attached to it (e.g., the sheath and / or leader of the scope). For example, the robotic arm 112 can be configured to operate the scope using elongated moving members. Examples of elongated moving members include one or more pull wires (e.g., pull or push wires), cables, fibers, and / or flexible shafts. For example, the robotic arm 112 can be configured to actuate multiple pull wires coupled to the scope to deflect the tip of the scope. The pull wires can include any suitable or desirable materials such as metallic materials and / or non-metallic materials such as stainless steel, Kevlar, tungsten, and carbon fiber. In some embodiments, the scope is configured such that the elongated moving members exhibit nonlinear behavior in response to applied forces. The nonlinear behavior may be based on the rigidity and compressibility of the scope, as well as variability in slack or rigidity between different elongated moving members.

[0081] As shown in the figure, the console 214 is positioned at the upper end of the column 210 of the robotic system 110. The console 214 may include a display 234 that provides a user interface for receiving user input and / or providing output (e.g., a dual-purpose device such as a touchscreen) to provide the physician / user with preoperative and / or intraoperative data, information for configuring the robotic system 110, etc. Potential preoperative data may include preoperative planning, navigation, and / or mapping data derived from preoperative computed tomography (CT) scans, and / or notes from preoperative patient interviews. Intraoperative data may include optical information, sensor and / or coordinate information provided by tools, as well as vital patient statistics such as respiration, heart rate, and / or pulse. The console 214 can be positioned and tilted so that the physician can access the console 214 from the side opposite the arm support member 216 of the column 214. From this position, the physician may operate the console 214 from behind the robotic system 110 while viewing the console 214, the robotic arm 112, and the patient.

[0082] The robot system 110 may include one or more I / O components / devices 236 that receive inputs and / or provide outputs for purposes such as interfacing with a user. One or more I / O components 236 may be configured to receive touch, speech, gestures, or any other type of input. In an example, one or more I / O components 236 may be used to provide inputs for controlling a device / system in order to control / configure the robot system 110, etc. As illustrated, one or more I / O components 236 may include one or more displays 234 configured to display data. One or more displays 234 may include one or more liquid crystal displays (LCDs), light-emitting diode (LED) displays, organic LED displays, plasma displays, electronic paper displays, and / or any other type of technology. In some embodiments, one or more displays 234 may include one or more touchscreens configured to receive inputs and / or display data. Furthermore, one or more I / O components 236 may include one or more I / O devices / controllers 238, which may include touchpads, controllers, mice, keyboards, wearable devices (e.g., optical head-mounted displays), virtual reality devices, or augmented reality devices (e.g., head-mounted displays). Furthermore, one or more I / O components 236 may include one or more speakers configured to output sound based on audio signals, and / or one or more microphones configured to receive sound and generate audio signals. In some embodiments, one or more I / O components 236 may include or be implemented as a console 214. Furthermore, one or more I / O components 236 may include one or more physically pressable buttons, such as a button on the distal end of the robot arm 112 (which can enable / disable the robot arm 112 admittance control mode).

[0083] Various components of the robot system 110 can be electrically and / or communicatively coupled using specific connecting circuits / devices / functions, which may or may not be part of the control circuit. For example, the connecting function may include one or more printed circuit boards configured to facilitate the mounting and / or interconnection of at least some of the various components / circuits of the robot system 110. In some embodiments, two or more components of the robot system 110 can be electrically and / or communicatively coupled to one another.

[0084] One or more communication interfaces 240 can be configured to communicate with one or more devices / sensors / systems. For example, one or more communication interfaces 240 can transmit / receive data wirelessly and / or via a wired method over a network. The network according to embodiments of this disclosure may include a local area network (LAN), a wide area network (WAN) (e.g., the Internet), a personal area network (PAN), a body area network (BAN), and the like. In some embodiments, one or more communication interfaces 240 can implement wireless technologies such as Bluetooth, Wi-Fi, and near-field communication (NFC).

[0085] One or more power units 242 can be configured to manage and / or provide power to the robot system 110. In some embodiments, one or more power units 242 include one or more batteries, such as lithium-ion batteries, lead-acid batteries, alkaline batteries, and / or other types of batteries. That is, one or more power units 242 may comprise one or more devices and / or circuits configured to provide power and / or power management functions. Furthermore, in some embodiments, one or more power units 242 include a mains power connector configured to couple to an alternating current (AC) or direct current (DC) mains power supply. Furthermore, in some embodiments, one or more power units 242 include a connector configured to couple to the control system 150 to receive power from the control system 150.

[0086] The robot system 110 may also include one or more actuators / hardware 244 to facilitate the movement of the robot arm 112. Each actuator 244 may be equipped with a motor, which can be mounted at a joint or elsewhere within the robot arm 112 to facilitate the movement of the joint and / or connected arm segment / link mechanism. Furthermore, the robot system 110 may include various other components such as pneumatics and light sources.

[0087] The control system 150 and / or the robot system 110 may include a control circuit 246 and / or a data storage device / memory 248 configured to perform the functions described herein. For ease of explanation and illustration, the control circuit 246 and the data storage device 248 are shown in a block between the control system 150 and the robot system 110. It should be understood that in many embodiments, the control system 150 and the robot system 110 may include separate instances of the control circuit 246 and the data storage device 248. That is, the control system 150 may include its own control circuit and data storage device (for example, to perform processing on the control system 150), while the robot system 110 may include its own control circuit and data storage device (for example, to perform processing on the robot system 110). In many embodiments, references to control circuits herein may refer to circuits embodied in the robot system, the control system, or any other component of the medical system, such as any component of the medical system 100 shown in Figure 1.

[0088] Although the control circuit 246 is illustrated as a component separate from the other components of the control system 150 / robot system 110, it should be understood that any or all of the other components of the control system 150 / robot system 110 can be at least partially embodied in the control circuit 246. For example, the control circuit 246 may include various devices (active and / or passive), semiconductor materials and / or areas, layers, regions and / or parts thereof, conductors, leads, vias, connections, etc., and one or more and / or parts thereof of the other components of the control system 150 / robot system 110 can be at least partially formed and / or embodied in such a circuit component / device.

[0089] As illustrated, the data storage device 248 may include a user interface component 250 configured to facilitate one or more user interfaces. For example, the user interface component 250 may generate and / or provide user interface data so that the user interface can be displayed via the display 152. The user interface may include an image representation from the viewpoint of a medical instrument (e.g., a scope), indicators for elements associated with the medical instrument (e.g., a working channel indicator, an articular plane indicator, etc.), a control unit or other interface elements, and / or other information. The user interface component 250 may adjust the orientation of the image representation, for example, by applying rotation adjustments to the image representation in the user interface as the medical instrument rotates / changes orientation. The user interface component 250 may maintain the orientation of indicators relative to the image representation to indicate the precise orientation of elements (e.g., working channels) relative to the image representation. The user interface component 250 may use orientation / position information relating to the medical instrument to provide the image representation and / or working channel indicators.

[0090] The data storage device 248 may also include an instrument drive component 252 configured to drive / control the medical instrument. For example, the instrument drive component 248 may be configured to process input signals from the I / O component 202, process position / orientation data indicating the position / orientation of the medical instrument, process image orientation data indicating the orientation of image data / representation in the user interface, generate control signals for controlling the robot system 110 and / or a medical instrument / tool ​​connected to the robot system 110, and transmit control signals to the robot system 110 to control the robot system 110 and / or the medical instrument / tool, etc. In some embodiments, user control of the medical instrument may be disabled during rotation of the medical instrument to prevent unintended movement of the medical instrument and / or complexity in controlling the mechanical components of the robot system 110. For example, once scope rotation has started, the instrument drive component 252 may ignore received user input until the scope rotation is complete, and as a result, the user cannot further articulate the scope during rotation.

[0091] In some embodiments, the instrument drive component 252 can control the medical instrument to move in a manner that correlates with a view provided via a user interface. For example, suppose physician 160 is driving the scope from the scope's viewpoint (e.g., based on image data from the scope provided via a user interface), and physician 160 moves the scope upward relative to the I / O component 202 by providing input via the I / O component 202, such as selecting upward movement control on the I / O component 202. The instrument drive component 252 can determine the orientation / position of the scope and / or the orientation of the image data / image representation displayed within the user interface. Using such orientation / position information, the instrument drive component 252 can move the scope upward on the user interface. Thus, the instrument drive component 252 can configure / update the control of the medical instrument based on the orientation of the image data / image representation displayed within the user interface. In the example, the manner in which the medical instrument is controlled can be reconfigured as the medical instrument rotates.

[0092] Furthermore, the data storage device 248 may include a positioning component 254 configured to determine / track the orientation / position of an object / medical device. For example, the positioning component 254 can process input data to generate position / orientation data for a medical device coupled to the robotic system 110. In some cases, the position / orientation data may indicate the amount of rotation of the medical device. The position / orientation data of an object / medical device may indicate the position / orientation of the object / medical device relative to a reference frame. The reference frame may be a reference frame relative to the anatomical structure of a patient, a known object (e.g., an EM field generator), a coordinate system / space, etc. In some embodiments, the position / orientation data may indicate the position and / or orientation of the distal end (and / or possibly the proximal end) of the medical device. For example, the position / orientation data of a scope may indicate the position and orientation of the distal end of the scope, including the amount of rotation of the distal end of the scope. The position and orientation of an object may be referred to as the orientation of the object. In the example, the technique for determining and / or tracking the position and / or orientation of a medical device may be referred to as a positioning technique.

[0093] Exemplary input data that can be used to generate object / medical device position / orientation data can take various different forms, such as sensor data from sensors associated with the medical device (e.g., EM field sensor data, visual / image data captured by imaging devices / depth sensors / radar devices on the medical device, accelerometer data from accelerometers on the medical device, gyroscope data from gyroscopes on the medical device, fiber optic-based shape detection, satellite-based positioning data from satellite-based sensors (e.g., Global Positioning System (GPS))), feedback data (also referred to as "kinematic data") from robotic arms / components (e.g., data showing how the robotic arm / component moved / operated), robot command data for robotic arms / components (e.g., control signals sent to robotic system 110 / robotic arm 112 to control the movement of robotic arm 112 / medical device), model data on the patient's anatomical structure (e.g., models of internal / external parts of the patient's anatomical structure), patient position data (e.g., data showing how the patient is positioned on the table), preoperative data, etc.

[0094] In some embodiments, the positioning component 254 can determine the position and / or orientation of an object using electromagnetic tracking. For example, the positioning component 254 can use real-time EM tracking to determine the real-time location of a medical device in a coordinate system / space, which can be recorded in the patient's anatomical structure, and this location can be represented by a preoperative model or other model. In EM tracking, an EM sensor (or tracker) containing one or more sensor coils can be embedded in one or more locations and / or orientations within a medical device (e.g., a scope, needle, etc.). The EM sensor can measure fluctuations in the EM field formed by one or more static EM field generators positioned at known locations. The location information detected by the EM sensor can be stored as EM data. The positioning component 254 can process the EM data to determine the position and / or orientation of an object, such as a medical device. The EM field generator (or transmitter) can be placed near the patient (e.g., within a predetermined distance) to form a low-intensity magnetic field that the EM sensor can detect. A magnetic field can induce a small current (e.g., below a threshold) within the sensor coil of an EM sensor, which can be analyzed to determine the distance and / or angle between the EM sensor and the EM field generator. These distances and / or orientations can be intraoperatively "recorded" on the patient's anatomical structure (e.g., a preoperative model) to determine the geometric transformation that aligns a single location in the coordinate system with the position of the patient's anatomical structure in a preoperative model of the patient's structure. Once recorded, an EM sensor (e.g., an implanted EM tracker) at one or more locations on the medical device (e.g., the distal tip of an endoscope, a needle, etc.) can display the position and / or orientation of the medical device through the patient's anatomical structure in real time.

[0095] Furthermore, in some embodiments, the localization component 254 can perform vision-based techniques to determine the position and / or orientation of an object. For example, a medical device may include a camera, a distance sensor (sometimes referred to as a “depth sensor”), a radar device, etc., to provide sensor data in the form of vision / image data. The localization component 254 can process the vision data to facilitate vision-based location tracking of the medical device. For example, preoperative model data can be used together with vision data to enable computer vision-based tracking of a medical device (e.g., an endoscope). In this example, using the preoperative model data, the control system 150 can generate a library of predicted endoscopic images based on the expected movement path of the scope, with each image linked to a location in the model. During surgery, this library can be referenced by the control system 150 to compare real-time images and / or other vision data acquired by the scope (e.g., a camera at the distal end of the endoscope) with images in the image library to assist in localization.

[0096] Furthermore, other types of vision-based techniques can be used to determine the position and / or orientation of an object. For example, the positioning component 254 can use functional tracking to determine the movement of an image sensor (e.g., a camera or other sensor), and therefore, a medical device associated with the image sensor. In some cases, the positioning component 254 can identify circular shapes in preoperative model data corresponding to anatomical lumens and track changes in those shapes to determine which anatomical lumen was selected, as well as the relative rotational and / or translational movement of the medical device. The use of topology maps can also further enhance vision-based algorithms or techniques. In addition, the positioning component 254 can use optical flow, another computer vision-based technique, to analyze the displacement and / or translation of image pixels in a video sequence within the vision data to infer camera movement. Examples of optical flow techniques may include motion detection, object segmentation calculation, luminance, motion compensation coding, and stereo disparity measurement. By comparing multiple frames across multiple iterations, the positioning component 254 can determine the movement and location of an image sensor (and therefore an endoscope).

[0097] Furthermore, in some embodiments, the positioning component 254 can determine the position and / or orientation of an object using robot commands and / or kinematic data. The robot commands and / or kinematic data may indicate the position / orientation of a robot arm (e.g., pitch, yaw, etc.) resulting from joint movement / actuation commands. For example, the robot commands and / or kinematic data may indicate the amount / direction of action of a joint, gear, pulley, belt, or another actuating element of the robot arm. The positioning component 254 can determine the position / orientation of a medical device attached to the robot arm using data indicating the position / orientation of the robot arm. For example, based on the position / orientation of a robot arm attached to a catheter, commands transmitted to control the catheter, and / or the characteristics of the catheter (e.g., catheter length, catheter capacity, etc.), the positioning component 254 can determine / estimate the position / orientation of the catheter. Furthermore, in an example, the positioning component 254 can use robot command data to determine how far a medical device has been inserted into / retracted into a patient, etc., based on commands to control the medical device, marks on the medical device indicating distance, etc. In some intraoperative embodiments, calibration values ​​can be used in combination with known insertion depth information to estimate the position and / or orientation of a medical device. Alternatively or additionally, these calculations can be analyzed in combination with EM, visual, and / or topology modeling to estimate the position and / or orientation of a medical device.

[0098] Furthermore, in some embodiments, the positioning component 254 can determine the position and / or orientation of an object using other types of data. For example, the positioning component 254 can analyze sensor data from shape-sensing fibers embedded in the medical device (which can provide shape data regarding the location / shape of the medical device), accelerometers, gyroscopes, satellite-based positioning sensors (e.g., the Global Positioning System (GPS)), radio frequency transceivers, etc. Such data can indicate the position and / or orientation of the medical device.

[0099] In some embodiments, the localization component 254 can process preoperative data to determine the location and / or orientation of an object. Preoperative data (sometimes referred to as “mapping data”) can be generated by performing computed tomography (CT) scans, such as low-dose CT scans. Preoperative CT images from the scans can be reconstructed into three-dimensional images, which are visualized (for example, as “slice” of cross-sections of the patient’s internal anatomical structures). When analyzed as a whole, image-based models can be generated of the anatomical cavities, spaces, and structures of the patient’s anatomical structures, such as the patient’s lung network and the anatomical structures of the kidneys. Centerline shapes can be determined and approximated from the CT images to create two-dimensional or three-dimensional volumes of the patient’s anatomical structures, referred to as model data (also referred to as “preoperative model data” if generated using only preoperative CT scans). Network topology models can also be derived from CT images.

[0100] In some embodiments, the localization component 254 can use a combination of input data. For example, the localization component 254 can use a probabilistic approach in which confidence weights are assigned to the position / orientation determined from multiple forms of input data. For example, if the EM data is unreliable (as in the case of EM interference), the EM data may be associated with a relatively low confidence value and can rely on other forms of input data such as visual data, robot commands, and kinematic data.

[0101] Many embodiments are discussed in the context of a user interface component 250, an instrument drive component 252, and / or a localization component 254, which are implemented (or include) as one or more instructions executable by the control circuit 246, but any of these components can be implemented at least partially as a control circuit.

[0102] The term "control circuit" can refer to one or more processors, processing circuits, processing modules / units, chips, dies (e.g., semiconductor dies including one or more active and / or passive devices and / or connection circuits), microprocessors, microcontrollers, digital signal processors, microcomputers, central processing units, graphics processing units, field-programmable gate arrays, application-specific integrated circuits, programmable logic devices, state machines (e.g., hardware state machines), logic circuits, analog circuits, digital circuits, and / or any set of any devices that manipulate signals (analog and / or digital) based on hardcoding of circuits and / or operation instructions. A control circuit can further include one or more storage devices, which can be embodied in a single memory device, multiple memory devices, and / or embedded circuits of a device. Examples of such data storage devices include read-only memory, random-access memory, volatile memory, non-volatile memory, static memory, dynamic memory, flash memory, cache memory, data storage registers, and / or any device that stores digital information. In embodiments in which the control circuit includes a hardware state machine (and / or implements a software state machine) and includes analog circuits, digital circuits, and / or logic circuits, note that data storage devices / registers for storing any associated operation instructions can be embedded inside or outside the circuit including the state machine, analog circuits, digital circuits, and / or logic circuits.

[0103] The term “memory” can refer to any suitable or desirable type of computer-readable medium. For example, computer-readable mediums may include one or more volatile data storage devices, non-volatile data storage devices, removable data storage devices, and / or non-removable data storage devices, which are implemented using any technology, layout, and / or data structure / protocol, and which include any suitable or desirable computer-readable instructions, data structures, program modules, or other types of data.

[0104] Computer-readable media that can be implemented by embodiments of this disclosure include, but are not limited to, the following: phase-change memory, static random-access memory (SRAM), dynamic random-access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage devices or other magnetic storage devices, or any other non-temporary media that can be used to store information for access by a computing device. In the specific contexts used herein, computer-readable media in general may not include communication media such as modulated data signals and carrier waves. Therefore, computer-readable media should be understood in general to refer to non-temporary media.

[0105] Exemplary scope Figure 3 illustrates a scope 130 positioned in a part of a patient's urinary tract according to one or more embodiments of the present disclosure. In this example, the scope 130 is implemented as an endoscope / ureteroscope. As mentioned above, ureteroscopy procedures can be implemented to investigate and / or treat abnormalities in the human ureter. For example, a ureteroscopy procedure can be implemented to treat and / or remove kidney stones. Such procedures can be performed at least partially manually and / or using at least partially robotic technology, such as the robotic system 110 shown in Figure 1. For example, the use of robotic devices and / or systems for certain endoscopic procedures can provide relatively higher precision, control, and / or coordination compared to strictly manual procedures.

[0106] The scope 130 may include a rigid or flexible tube and / or be sized to pass through an outer sheath, catheter, introducer, or other tubular device. However, in some examples, the scope 130 can be used without such a device. For example, the scope 130 may be configured to pass through an access sheath 136 to access a target anatomical structure. The access sheath 136 may have a diameter large enough to allow the scope 130 to be drawn through the access sheath along with the object captured by an instrument (e.g., a basket device) when the size of the object / stone is not too large (e.g., within threshold size). In some examples, the access sheath 136 may advance through the ureter 197 to a location near the renal pelvis 302 and / or the ureteropelvic junction 304. The distal end of the access sheath 136 may be placed in a location within the ureter 197 and / or renal pelvis 302, and such placement may depend at least in part on the anatomical structure. However, in other cases, the access sheath 136 may be positioned only within the urethra or other anatomical structure, without being positioned near the renal pelvis 302. In general, the access sheath 136 does not need to be articulate to the extent that the scope 130 can articulate, and therefore, navigating / driving the access sheath 136 into the kidney may not be practical.

[0107] The scope 130 may be articulate, such as with respect to at least the distal portion of the scope, so that the scope 130 can be maneuvered within the anatomical structure of a human. In some embodiments, the scope 130 is configured to be articulate with 5 degrees of freedom (DOF), including, for example, XYZ coordinate translation, and pitch and yaw. Furthermore, in some embodiments, the scope 130 is articulate with 6 DOF, including XYZ coordinate translation, and pitch, yaw, and rotation. In other embodiments, the scope 130 is articulate with other DOF. In embodiments in which the scope 130 is equipped with a position sensor, the position sensor can provide position information such as 3-DOF position information (e.g., x, y, and z coordinates), 5-DOF position information (e.g., x, y, and z coordinates, as well as pitch and yaw angles), and 6-DOF position information (e.g., x, y, and z coordinates, as well as pitch, yaw, and rotation angles). In some embodiments, the scope 130 may include expandable sections such as an inner leader portion and an outer sheath portion, which may be operated to extend the scope 130 in an expandable or contractible manner.

[0108] The scope 130 may include one or more elongated moving members (not shown) configured to control the movement of the scope 130, such as the distal end of the scope 130. The elongated moving members may include one or more wires (e.g., pull wires or push wires), cables, fibers, and / or flexible shafts. The pull wire may include any suitable or desirable material, such as metallic and nonmetallic materials, such as stainless steel, Kevlar, tungsten, carbon fiber, and equivalents. In some embodiments, the scope 130 is configured to exhibit nonlinear behavior in response to forces applied by the elongated moving members. The nonlinear behavior may be based on the stiffness and / or compressibility of the scope 130, as well as variability in slack or stiffness between different elongated moving members. In embodiments of a robotic arm, one or more pull wires coupled to the scope 130 may be actuated to deflect the tip 306 of the scope 130. In contrast, in a user-handheld embodiment, the user can provide manual input via an actuator to actuate one or more pull wires of the scope 130, thereby deflecting the tip 306 of the scope 130.

[0109] The scope 130 may include an imaging device 308 configured to capture image data, such as image data representing the internal anatomical structures of a patient. The imaging device 308 may include a camera, such as an optical camera and / or another imaging device. The imaging device 308 may include optical fibers, fiber arrays, and / or lenses. One or more optical components of the imaging device 308 may move with the tip 306 of the scope 130, and the movement of the tip 306 results in a change in the image captured by the imaging device 308. In some embodiments, the scope 130 may house an optical assembly and wires and / or optical fibers for transmitting signals between the distal end 306 of the scope 130. Furthermore, the scope 130 may be further configured to house optical fibers to transport light from a proximal light source, such as a light-emitting diode, to the distal end 306 of the scope. The distal end 306 of the scope 130 may include a port for a light source 310 to illuminate the anatomical space, which may be useful when using the imaging device 308. Although multiple light sources 310 are shown, scope 130 can be implemented using any number of light sources.

[0110] The scope 130 may also include a working channel 312 for deploying instruments 314. Exemplary instruments 314 include a laser device 314a configured to provide a laser, a basket device 314b configured to capture / recover an object (e.g., a fragment of a kidney stone), forceps 314c configured to grasp / hold an object, a surgical scalpel 314d configured to cut an object, a lithotomyer (not shown), and an irrigation / suction device (not shown) configured to provide irrigation / suction to a target site. The working channel 312 may extend longitudinally through the scope 130 from its proximal end to its distal end 306. In this example, the working channel 312 is offset to one side of the scope 130 (e.g., offset from the longitudinal axis 316 of the scope 130), as shown in Figure 3. In other examples, the working channel 312 is located at the center of the scope 130 or elsewhere. Although the imaging device 308 is shown as being attached to the distal end 306 of the scope 130 (e.g., integrated with the scope 130), in some cases the imaging device 308 is a separate device deployed through a working channel 312. Furthermore, although a single working channel 312 is shown, any number of working channels may be implemented.

[0111] In some embodiments, the scope 130 includes a sensor (sometimes referred to as a “position sensor”) configured to generate and / or transmit sensor data to another device. This sensor data (sometimes referred to as “sensor position data”) can indicate the position and / or orientation of the scope 130 (e.g., its distal end 306) and / or can be used to determine / predict the position / orientation of the scope 130. For example, the sensor may provide sensor data to a control system, which is then used to determine the position and / or orientation of the scope 130. The sensor may be positioned at the distal end 306 of the scope 130 and / or elsewhere. In some embodiments, the sensor may be an electromagnetic (EM) sensor having a coil of conductive material, or other forms / embodiments of an antenna. However, the scope 130 may include other types of sensors, such as shape-sensing fibers, accelerometers, gyroscopes, satellite-based positioning sensors (e.g., Global Positioning System (GPS) sensors), and radio frequency transceivers.

[0112] In some embodiments, the scope 130 can be configured to move by different amounts in different planes so that it moves with more / fewer degrees of freedom in a particular direction when the scope is oriented in a particular manner. For example, the scope 130 can be associated with a primary articular plane and a secondary articular plane, which are perpendicular to each other. The tip 306 of the scope 130 can be configured to move with a greater degree of freedom in the primary articular plane compared to the secondary articular plane (e.g., exhibiting more deflection).

[0113] The scope 130 and / or instrument 314 may be controllable in any preferred or desired manner based on manual operation of a handle component, electronic user input, or automatic input. For example, Figure 318 shows an exemplary robotic control configuration for controlling the scope 130 and / or instrument 314 using one or more robotic arms, where each scope-driver instrument coupling 132 and / or handle 134 can be coupled to a robotic arm capable of operating the scope-driver instrument coupling 132 and / or handle 134 to control the scope 130 / instrument 314. On the other hand, Figure 320 shows an exemplary manual control configuration, where the scope 130 includes a handheld handle 322 configured to be held / operated by a user to facilitate movement of the scope 130 / instrument 314.

[0114] Example image / indicator adjustment Figures 4-1 to 4-9 show examples of one or more embodiments in which the scope is rotated and rotation adjustments are applied to the scope image in the user interface when the scope has completed its rotation (sometimes referred to as a “stepwise approach”). On the other hand, Figures 5-1 to 5-5 show examples of one or more embodiments in which the scope is rotated and rotation adjustments are applied to the scope image in the user interface while the scope is rotating (sometimes referred to as a “continuous approach”). In these examples, the scope may rotate by a predetermined amount to rotate the working channel of the scope by the same amount, thereby providing a better orientation for the scope to treat the target area. In particular, the scope rotates 180 degrees to reorient the working channel. However, any amount of rotation can be performed, and this may or may not be predetermined. Scope rotation can be initiated in various ways, such as by receiving user input via an I / O device / component (via the user interface or otherwise), or by automatically determining whether to rotate the scope.

[0115] In these examples, the user interface 402 / 502 provides information to assist the user in controlling the scope 404 / 504 and / or instruments deployed through the scope 404 / 504, such as for performing medical procedures. The user interface 402 / 502 may provide an image representation 406 / 506 representing a view from the viewpoint of the scope 404 / 504, and a work channel indicator 408 / 508 indicating the rotation angle of the work channel of the scope 404 / 504 relative to the image representation 406 / 506. The work channel indicator 408 / 508 may be displayed within the user interface 402 / 502 at all times and / or at other times when a work channel rotation is performed. The image representation 406 / 506 may include, based on, and / or generated from, image data captured by an imaging device associated with the scope 404 / 504. Here, image representations 406 / 506 show a kidney stone 410 / 510 and a body cavity 412 / 512 adjacent to the kidney stone 410 / 510, which are usually visible when performing a procedure to remove the kidney stone 410 / 510 (however, the kidney stone 410 / 510 may be oriented in other ways). As shown in the figure, user interface 402 / 502 may include user interface elements 414 / 514 for rotating the working channel of the scope 404 / 504 (i.e., rotating the scope 404 / 504). Meanwhile, images 416 / 516 and elements 418 / 518 in each figure show the orientation of the scope 404 / 504 within the patient's anatomical structure for each figure. Elements 418 / 518 represent the orientation of the tip and coordinate frame of the scope 404 / 504 for each image 416 / 516.

[0116] Generally, the orientation of an image representation within a user interface can be correlated (with or without offset) to the orientation of the scope's tip / imaging device. In other words, the user's reference frame via the user interface (also referred to as the "user's view") is correlated to the orientation of the scope's tip / imaging device. Therefore, when the scope 404 rotates, the image representation generally rotates within the user interface unless rotation adjustments are applied. For example, in the context of Figures 4-1 to 4-9, the coordinate frame of the scope 404 can initially be oriented as shown at 418 in Figure 4-1, and the upward direction of movement relative to the coordinate frame of the scope 404 (e.g., the direction pointed to by the arrow at 418) corresponds to the left direction in Figure 4-1. Here, the coordinate frame of the imaging device of the scope 404 is the same, but may be offset in some cases. In this example, the user interface 402 provides an image representation 406 associated with the scope 404 having the offset in Figure 4-1, such that the upward direction in the user interface 402 corresponds to the right direction in the coordinate frame of the scope 404. Such an offset can be maintained until rotational adjustment is applied.

[0117] Furthermore, scope control can generally be correlated with the user interface so that the scope can be controlled intuitively by the user. For example, suppose the user provides input via an I / O component to move the scope upward relative to the I / O component, such as by selecting control for upward movement on the I / O component. Information regarding the orientation / position of the scope and / or the orientation of the image representation within the user interface can be used to move the scope upward on the user interface, even if such orientation is not upward relative to the scope's coordinate frame. Thus, the user can perceive the scope as moving upward on the user interface.

[0118] As described above, Figures 4-1 to 4-9 show examples of applying rotation adjustments to the image representation 406 after the scope 404 has completed its rotation. In these figures, the arrow next to image 416 indicates the direction of rotation of the scope 404, and the arrow next to user interface 402 indicates the direction of rotation of the image representation 406 within user interface 402. For example, in Figures 4-6 to 4-9, the arrow next to user interface 402 indicates that the image representation 406 is rotating counterclockwise within user interface 402, which is due to rotation adjustments applied to adjust / correct the orientation of the image representation 406.

[0119] As illustrated, Figure 4-1 shows the image representation 402 and working channel indicator 404 within the user interface 402 when the scope 404 is oriented in its initial position. Here, the scope 404 is oriented as shown by 418. In this example, the user rotates the scope 404 by a predetermined amount by selecting the interface element 414. As shown in Figures 4-2 to 4-5, the scope 404 rotates 180 degrees counterclockwise. However, the scope 404 can also rotate clockwise to achieve the same amount of rotation. Figures 4 to 5 show the orientation of the user interface 402 and the scope 404 after the scope 404 has completed its rotation. When the scope 404 rotates, the content in the image representation 406 rotates clockwise within the user interface 402, as also shown in Figures 4-2 to 4-5. That is, the virtual horizon associated with the image representation 406 can rotate clockwise due to the rotation of the scope 404 (and the imaging device associated with the scope 404). A virtual horizontal line can represent a reference line for the content within the image representation 406 (for example, a reference horizontal line for the anatomical structure of a patient). In relation to Figures 4-1 to 4-9, the virtual horizontal line can initially be the horizontal line in Figure 4-1.

[0120] While the scope 404 is rotating, the image representation 406 can be displayed in a cropped form. For example, as shown in Figures 4-2 to 4-5, the image representation 406 can be displayed as a circle within the user interface 402. This avoids the display of black bars on the user interface 402 due to the loss of image data from the imaging device. The cropped form can also indicate to the user that the scope 404 is rotating. While many examples are discussed in the context of presenting the image representation 406 in a cropped form (e.g., a circle) during and after the rotation of the scope 404, the image representation 406 can also be displayed in an uncropped form and / or in other forms during rotation, such as the initial / original form of the image representation 406 (based on the field of view of the imaging device), or another shape.

[0121] As the scope 404 rotates, the working channel indicator 408 can maintain the rotation angle of the working channel relative to the image representation 406. In particular, as shown in Figures 4-2 to 4-5, the working channel indicator 408 remains stationary within the user interface 402 when the virtual horizontal line in the image representation 406 rotates to accurately indicate the position of the working channel relative to the content of the image representation 406. That is, the working channel indicator 408 can maintain the same rotation angle within the user interface 402 when the scope 404 rotates 180 degrees from its initial orientation. In this example, the working channel indicator 408 is presented as an overlay on the image representation 406 as a partially transparent interface element. This allows the image representation 406 to be at least partially visible through the working channel indicator 408. However, the working channel indicator 408 can be displayed in other ways, such as as a non-overlay on 406, as an opaque interface element (e.g., solid), as a fully transparent interface element, and / or in other forms / methods.

[0122] Once scope 404 has completed its rotation, resulting in a 180-degree rotation of the working channel, rotation adjustments can be applied to adjust / correct the orientation of image representation 406. In the example, the determination that scope 404 has completed its rotation can be made based on one or more of the techniques described herein (e.g., positioning techniques). As shown in Figures 4-6 to 4-9, rotation adjustments may include rotating image representation 406 counterclockwise within the user interface 402 until the virtual horizon associated with image representation 406 returns to the initial orientation shown in Figure 4-1 (e.g., the content within image representation 406 returns to the initial orientation). Image representation 406 can rotate by the same amount as scope 404 has rotated (e.g., 180 degrees). Image representation 406 can rotate within the user interface 402 while scope 404 remains in the same orientation (e.g., rotated / inverted position). Figure 4-9 shows the user interface 402 (and working channel indicator 408) when the image representation 406 is rotated 180 degrees to return the virtual horizon to its initial orientation. Although the rotation adjustment is described in the context of rotating the image representation 406 counterclockwise, the rotation adjustment can alternatively be performed by rotating the image representation 406 clockwise.

[0123] While rotating the image representation 406, the image representation 406 can be displayed in a cropped form in the same manner as described while rotating the scope 404. As shown in Figures 4 to 9, when rotation adjustment is applied to the image representation 406 and the working channel 408 is rotated, the image representation 406 can be returned to its uncropped form. However, the image representation 406 can be displayed in other forms during and / or upon completion of the scope rotation. Thus, rotation adjustment can provide the user with the same reference frame (e.g., the same virtual horizon) before and after rotating the scope 404, while allowing the user to accurately visualize the position of the working channel (e.g., the working channel is inverted).

[0124] As shown in Figures 4-6 to 4-9, during rotation adjustment of the image representation 406, the working channel indicator 408 can be rotated counterclockwise within the user interface 402 to maintain an accurate representation of the angle of the working channel relative to the image representation 406. The working channel indicator 408 can be rotated by the same amount as the scope 404 has rotated (e.g., 180 degrees).

[0125] In some embodiments, the rotation adjustment techniques discussed in the context of Figures 4-6 to 4-9 are implemented when it may be difficult to accurately track the orientation of the tip of the scope 404 during rotation. For example, rotating the scope 404 may include disengaging / engaging the mechanical components of the scope 404 / robot arm, evaluating the tightening of the elongated shaft of the scope 404, or sending one or more control signals to components of the robot system. Such actions may take an unknown amount of time and / or be associated with a certain amount of delay. Furthermore, in examples where the action is performed at the proximal end of the scope 404, such action may not need to be instantaneously translated into rotation at the distal end of the scope 404. Moreover, in some examples, the scope 404 may be rotated when the EM field generator is not in use, causing the control system to rely on data other than EM sensor data to track the rotation of the scope 404, which may be less accurate / unreliable.

[0126] Figures 4-1 to 4-9 illustrate a situation where the image representation 406 rotates when the scope 404 completes its rotation, but the image representation 406 can rotate when other events occur. For example, the image representation 406 can rotate incrementally, rotating by a predetermined amount each time the scope 404 completes a certain amount of rotation (for example, rotating 15 degrees for every 15 degrees of rotation completed by the scope 404).

[0127] On the other hand, Figures 5-1 to 5-5 show an example of applying rotation adjustment using a continuous approach while rotating the scope 504. In these figures, the arrow next to image 516 indicates the direction of rotation of the scope 504. Here, the image representation 506 is continuously updated within the user interface 502 as the scope 504 rotates from its initial orientation to its final orientation of 180 degrees. As illustrated, the virtual horizon for the image representation 506 remains the same throughout the entire rotation of the scope 504, while the working channel indicator 508 rotates around the image representation 506. In particular, the scope 504 rotates counterclockwise with respect to the coordinate frame of the scope 504, and the working channel indicator 508 rotates in the same direction (i.e., counterclockwise) with respect to the user interface 502. Furthermore, the image representation 506 can rotate counterclockwise in an imperceptible manner. To make such adjustments to the working channel indicator 508 and the image representation 506, the orientation of the scope 504 can be tracked in real time using one or more of the localization techniques described herein. This allows the user interface 502 to present information in an accurate manner. Similarly, as described above, while rotating the scope 504, the image representation 506 can be displayed in a cropped form, as shown in Figures 5-2 to 5-4. Once the scope 504 has completed its rotation, the image representation 506 can return to its uncropped form, as shown in Figure 5-5. By performing such rotation adjustments continuously, the user may perceive the image representation 506 as remaining the same, which may further help the user avoid losing their sense of direction within the patient's anatomical structure. That is, in Figures 5-1 to 5-5, the orientation of the image representation 506 can be continuously updated to maintain a virtual horizontal line relative to the patient's anatomical structure (e.g., maintaining the user's reference frame).

[0128] Figures 4-1 to 4-9 and 5-1 to 5-9 show various image representations in cropped or uncropped form during specific stages of scope rotation, but the image representation can be displayed in cropped or uncropped form at any time before, during, and / or after scope rotation.

[0129] Exemplary User Interface Figure 6 illustrates an exemplary user interface 602 for assisting a user in controlling a scope and / or an instrument deployed through the scope's working channel, according to one or more embodiments. The user interface 602 may provide an image representation 604 representing a view from the scope's viewpoint and a working channel indicator 606 indicating the rotation angle of the scope's working channel relative to the image representation 604. The image representation 604 may include, be based on, and / or generated from image data captured by an imaging device associated with the scope. Here, the image representation 604 shows a kidney stone 608 and a body cavity 610 adjacent to the kidney stone 608. In this example, the user interface 602 is presented during a medical procedure. However, the user interface 602 may be presented at other times.

[0130] As illustrated, the user interface 602 may include a work channel element 612 for rotating the work channel of the scope. The work channel indicator 606 may be implemented as a translucent semicircular element overlaid on the image representation 604. However, the work channel element 612 may take other forms and / or be located elsewhere within the user interface 602. In one example, the user can select the work channel element 612 to rotate the scope 180 degrees (e.g., invert it from its initial orientation to an orientation rotated 180 degrees), but rotation is also possible. In another example, the user may provide a first input via an I / O component (e.g., the user interface 602, a controller, etc.) to start the scope rotation and a second input to stop the scope rotation. In yet another example, the user may provide an input for rotating the scope by selecting and holding a control element on an I / O component, and the scope may rotate while the control element is selected. In any of these cases, one or more of the rotation adjustment techniques discussed herein may be implemented to adjust the image representation 604 when / after the scope is rotated. Furthermore, the working channel indicator 606 can be adjusted to indicate the orientation of the working channel of the scope. The user interface 602 may also include a horizontal rotation element 614 for rotating the scope without applying rotation adjustments to the image representation 604 within the user interface 602 (e.g., without rotational correction of the scope image). This can be used to align the visual horizon with the principal plane of the kidney or other functional parts. This can also be used to align the principal / main plane of articular movement of the scope so that the scope can be more easily reached / navigated.

[0131] Furthermore, the user interface 602 may include joint motion bars 616, 618 around the image representation 604 to visualize the amount of joint movement associated with the selected instrument (e.g., a scope or tool in this example). The apex / bottom joint motion bar 616 can indicate the amount of vertical joint movement. For example, the apex / bottom joint motion bar 616 can be positioned above or below the image data 604 and / or its length can be extended / contracted to indicate vertical joint flexion of the medical instrument. In the example in Figure 6, the apex / bottom joint motion bar 616 is positioned above the image representation 604 to indicate that the scope is jointly moving upward. The right / left joint motion bar 618 can indicate the amount of horizontal joint movement. For example, the right / left joint motion bar 618 can be positioned to the right or left of the image representation 604 and / or its length can be extended / contracted to indicate horizontal joint movement of the medical instrument. In the example in Figure 6, the right / left joint movement bar 618 is positioned to the right of the image representation 604, indicating that the scope is being moved to the right.

[0132] The user interface 602 may also include interface elements 620, 622 for selecting medical instruments to calibrate / control. Interface element 620 enables control of the scope, and interface element 622 enables control of instruments (also referred to as “tools”) associated with the scope. Instruments associated with the scope can be deployed through the scope’s work channel. In this example, the scope is selected, which allows the user to navigate the scope and causes the user interface 602 to display information about the scope.

[0133] Flowchart example Figure 7 illustrates an exemplary flowchart of a process 700 for rotational adjustment and application to a scope image within a user interface, according to one or more embodiments. Various actions / behaviors associated with process 700 can be performed by control circuits implemented in any of the devices / systems described herein, such as the control system 150, the robotic system 110, the table 170, medical instruments, and / or other devices, or in combination thereof. While various blocks are illustrated as part of process 700, any such block can be removed. Furthermore, additional blocks can be implemented as part of process 700. The order in which the blocks are illustrated is for illustrative purposes only, and the blocks can be implemented in any order. In some embodiments, one or more blocks of process 700 are executable instructions, implemented as executable instructions that, when performed by a control circuit, cause the control circuit to perform the described functionality / action. However, one or more blocks of process 700 can be implemented in other ways, such as by other devices / systems, the user, etc.

[0134] In block 702, process 700 may include receiving image data from an imaging device associated with the scope. The scope may include a work channel for deploying medical instruments / tools. The work channel may be offset from the longitudinal axis of the scope. In some cases, the imaging device is located on the end of the scope, while in other cases, the imaging device is deployed through the work channel of the scope. In the example, the image data may be received in real time, such as receiving multiple images from the imaging device.

[0135] In block 704, process 700 may include generating an image representation based at least partially on image data. For example, the image representation may include / represent image data from a scope. In some cases, the image data may be reformatted / converted to generate the image representation.

[0136] In block 706, process 700 may include displaying an image representation on a user interface. In one example, displaying image data may include sending data to the display so that the display can present the image representation via the user interface. In another example, displaying an image representation may include displaying / presenting the image representation via the display.

[0137] In block 708, process 700 may include displaying a work channel indicator. The work channel indicator may indicate the rotation angle of the work channel relative to the image representation. In one example, displaying the work channel indicator may include sending data to the display so that the display can present the image representation via a user interface. In another example, displaying the image representation may include displaying / presenting the work channel indicator via the display. In some cases, the work channel indicator is overlaid on the image representation and / or positioned on the periphery of the image representation.

[0138] In block 710, process 700 may include receiving an input control signal from an input device to rotate a work channel. For example, a user may use an input device to provide an input for rotating a scope, and the input device may generate / transmit an input signal indicating that the scope should be rotated. In some cases, the input is to rotate the scope / work channel by a predetermined amount.

[0139] In block 712, process 700 may include controlling the scope to rotate based at least partially on an input control signal. For example, a control signal may be sent to a robot system / robot arm to control the robot arm to rotate the scope, which may be in a specific direction of rotation. In some cases, the scope may be controlled to rotate by a predetermined amount, such as reversing the working channel from one orientation to another.

[0140] In block 714, process 700 may include determining that the scope has completed its rotation. Such determination may be based on feedback data from a robotic arm configured to be coupled to the scope, sensor data generated by position sensors associated with the scope, and / or control signals generated to control the robotic arm. In some cases, the rotation of the scope may be tracked over time.

[0141] In block 718, process 700 may include determining the amount by which the scope has rotated. Such determination may be based on feedback data from a robotic arm configured to be coupled to the scope, sensor data generated by position sensors associated with the scope, and / or control signals generated to control the robotic arm. Process 700 shows blocks 714 and 716, but in some cases, one or more of such blocks (and / or other blocks of process 700) may not be implemented.

[0142] In block 718, process 700 may include rotating a work channel indicator within the user interface. For example, based at least in part on the amount the scope has rotated, the work channel indicator may rotate from a first orientation relative to the user interface to a second orientation relative to the user interface. The first orientation may indicate a first rotation angle of the work channel, and the second orientation may indicate a second rotation angle of the work channel.

[0143] In block 720, process 700 may include applying rotation adjustments to the image representation within the user interface. For example, the image representation may be rotated within the user interface based at least in part on the amount the scope has rotated. In some cases, the image representation may be rotated within the user interface upon determination that the scope has completed its rotation. In other examples, the image representation may be rotated as the scope rotates, such as continuously based on the current orientation of the scope. Here, the image representation may be rotated without being perceived by the user through the user interface. In the example, the image representation may be displayed in a cropped form while the scope is rotating and / or the image representation is rotating. Furthermore, in the example, the image representation may be rotated within the user interface in the same direction of rotation (e.g., clockwise or counterclockwise) as the scope rotates with respect to a reference frame of the scope's tip (e.g., the scope's coordinate frame).

[0144] Further Embodiments Depending on the embodiment, any particular action, event, or function among the algorithms or processes described herein may be performed in a different order, added, merged, or completely excluded. Thus, in some embodiments, not all of the described actions or events are necessary for the execution of the process.

[0145] In particular, conditional language used herein, such as “can,” “could,” “might,” “may,” “eg,” and equivalents, is intended in its ordinary sense unless otherwise specifically described or understood in the context in which it is used, and is generally intended to convey that a particular embodiment includes a particular feature, element, and / or step, but other embodiments do not. Therefore, such conditional language is not generally intended to suggest that the feature, element, and / or step is required in any way for one or more embodiments, or that one or more embodiments necessarily include logic for determining whether these features, elements, and / or steps are included in or implemented in any particular embodiment, with or without input or prompting. Terms such as “comprising,” “including,” “having,” and equivalents are used in their ordinary sense, in a non-restrictive and comprehensive manner, and do not exclude further elements, features, actions, behaviors, etc. Furthermore, the term “or,” when used, for example to connect an enumeration of elements, is used in its inclusive sense (and not its exclusive sense), meaning one, some, or all of the enumerated elements. Unless otherwise specifically stated, connecting language such as “at least one of X, Y, and Z” is understood in the context in which it is commonly used to convey that an item, term, element, etc., may be any of X, Y, or Z. Thus, such connecting language is not intended in general to imply that a particular embodiment requires the presence of at least one of X, at least one of Y, and at least one of Z, respectively.

[0146] In the above description of embodiments, it should be understood that various features are sometimes grouped together in a single embodiment, figure, or description for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various aspects of the invention. However, the method of the disclosure should not be construed as reflecting an intention that any claim requires more features than expressly described in that claim. Furthermore, any component, feature, or step illustrated and / or described in a particular embodiment of this specification may be applied to or used in conjunction with any other embodiment. Moreover, no component, feature, step, or group of components, features, or steps is required or essential for any particular embodiment. Accordingly, the scope of the disclosure disclosed herein and claimed below is not intended to be limited by the particular embodiments described above, but should be determined by a fair reading of the following claims.

[0147] It should be understood that certain ordinal terms (e.g., "first" or "second") may be provided for ease of reference and do not necessarily imply any physical characteristics or ordering. Therefore, when used herein, ordinal terms (e.g., "first," "second," "third," etc.) used to modify elements such as structure, components, and behavior do not necessarily indicate the priority or order of an element relative to any other element, but rather, generally, they may distinguish an element from another element having a similar or identical name (apart from the use of ordinal terms). Furthermore, when used herein, the indefinite articles ("a" and "an") may indicate "one or more" rather than "one." Additionally, actions performed "on the basis of" a condition or event may also be performed on the basis of one or more other conditions or events not explicitly listed.

[0148] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as those generally understood by those skilled in the art to which the embodiments belong. Terms such as those defined in commonly used dictionaries should be interpreted as having the meaning consistent with their meaning in the context of the relevant art, and should not be interpreted in an ideal or overly formal sense unless expressly defined herein.

[0149] The spatially relative terms “outside,” “inside,” “top,” “bottom,” “downward,” “upward,” “vertical,” and “horizontal,” and similar terms, may be used herein to facilitate explanations of the relationship between one element or component and another, as illustrated in the drawings. It should be understood that spatially relative terms are intended to encompass different orientations of a device during use or operation, in addition to the orientation depicted in the drawings. For example, if a device shown in a drawing is inverted, a device positioned “below” or “below” another device may be positioned “above” another device. Thus, the illustrative term “downward” may include both lower and upper positions. Devices may also be oriented in other directions, and therefore, spatially relative terms may be interpreted differently depending on the orientation.

[0150] Unless otherwise specified, comparative and / or quantitative terms such as "less," "more," and "greater" are intended to encompass the concept of equality. For example, "less" can mean not only "less" in the strict mathematical sense, but also "less than or equal to."

[0151] [Implementation Method] (1) A control system, A communication interface configured to receive image data from an imaging device associated with a scope, including a work channel for deploying medical instruments, A control circuit that is communicatively coupled to the aforementioned communication interface, An image representation of the aforementioned image data is generated, The image representation is displayed on the user interface, The amount by which the scope has rotated is determined, A control system comprising: a control circuit configured to rotate the image representation within the user interface based at least partially on the amount by which the scope has been rotated. (2) The control circuit is Displaying a work channel indicator on the user interface, wherein the work channel indicator indicates the rotation angle of the work channel relative to the image representation. The control system according to Embodiment 1, configured to rotate the work channel indicator within the user interface from a first orientation relative to the user interface to a second orientation relative to the user interface, based at least in part on the amount by which the scope has rotated. (3) The control system according to Embodiment 2, wherein the control circuit is configured to display the work channel indicator by overlaying the work channel indicator on the image representation. (4) The control circuit is The scope determines that it has completed its rotation, The control system according to Embodiment 1, configured to rotate the image representation within the user interface upon determination that the scope has completed its rotation. (5) The control system according to Embodiment 1, wherein the control circuit is configured to rotate the image representation within the user interface as the scope rotates.

[0152] (6) The control system according to Embodiment 1, wherein the control circuit is configured to determine the amount by which the scope has rotated based on at least one of the following: feedback data from a robot arm configured to be coupled to the scope, sensor data generated by a position sensor associated with the scope, or a control signal generated to control the robot arm. (7) The control circuit is The input device receives an input control signal for rotating the work channel. The control system according to Embodiment 1, configured to control the rotation of the scope based at least partially on the input control signal. (8) The control system according to embodiment 7, wherein the control circuit is configured to control the rotation of the scope by controlling the scope to rotate by a predetermined amount. (9) A method, Receiving image data from an imaging device associated with a scope, wherein the scope includes a working channel. The control circuit displays the image representation of the image data on the user interface, The control circuit displays a work channel indicator on the user interface, wherein the work channel indicator indicates the rotation angle of the work channel relative to the image representation. The amount by which the scope has rotated is determined, Based at least partially on the amount by which the scope has rotated, Rotating the aforementioned image representation within the user interface, A method comprising rotating the work channel indicator within the user interface from a first orientation to a second orientation. (10) Further includes determining that the scope has completed its rotation, The method according to embodiment 9, wherein the rotation of the image representation and the rotation of the work channel indicator are performed after it is determined that the scope has completed its rotation.

[0153] (11) The method according to embodiment 9, wherein rotating the image representation includes continuously rotating the image representation. (12) The method according to Embodiment 11, wherein determining the amount by which the scope has rotated is based on at least one of feedback data from a robot arm configured to be coupled to the scope, position data generated by a position sensor associated with the scope, or a control signal generated to control the robot arm. (13) The method according to Embodiment 11, wherein rotating the image representation includes displaying the image representation in a cropped form on the user interface while rotating the scope. (14) Receiving user input from an input device to rotate the scope by a predetermined amount, The method according to embodiment 9, further comprising controlling the scope to rotate by the predetermined amount. (15) The method according to embodiment 9, wherein displaying the work channel indicator includes overlaying the work channel indicator on the image representation.

[0154] (16) a system, A scope including a working channel for removably receiving medical devices, An imaging device configured to be coupled to the scope, A control circuit that is communicatively coupled to the scope and the imaging device, Receiving image data from the aforementioned imaging device, To display the image representation of the aforementioned image data on the user interface, Displaying a work channel indicator on the user interface, wherein the work channel indicator displays the rotation angle of the work channel. The amount by which the scope has rotated is determined, Rotating the work channel indicator within the user interface based at least partially on the amount by which the scope has rotated, A system comprising: a control circuit configured to apply rotation adjustments to the image representation in the user interface based at least partially on the amount by which the scope has been rotated; and (17) The system according to embodiment 16, wherein the working channel is offset from the longitudinal axis of the scope. (18) The control circuit is The scope is controlled to rotate in the rotational direction, The system according to embodiment 16, configured to apply the rotation adjustment to the image representation by rotating the image representation in the rotation direction within the user interface. (19) The control circuit is The scope is further configured to determine when the rotation has been completed, The system according to embodiment 16, wherein the rotation adjustment is applied upon determination that the scope has completed its rotation. (20) The system according to embodiment 16, wherein the control circuit is configured to apply the rotation adjustment by continuously rotating the image representation as the scope rotates.

[0155] (21) The system according to embodiment 16, wherein the control circuit is further configured to determine the amount by which the scope has rotated based on at least one of feedback data from a robot arm configured to be coupled to the scope, sensor data generated by a position sensor associated with the scope, or a control signal generated to control the robot arm. (22) The system according to embodiment 16, wherein the control circuit is configured to apply the rotation adjustment by rotating the scope and displaying the image representation in a cropped form on the user interface. (23) The control circuit is The input control signal for rotating the aforementioned work channel is received. The system according to embodiment 16, configured to control the rotation of the scope based at least partially on the input control signal. (24) One or more non-temporary computer-readable media for storing computer executable instructions, wherein when a computer executable instruction is executed by a control circuit, the control circuit receives Receiving image data from an imaging device associated with a scope, wherein the scope includes a working channel. Displaying the image representation of the aforementioned image data on the user interface, Displaying a work channel indicator within the user interface, wherein the work channel indicator displays the rotation angle of the work channel. The amount by which the scope has rotated is determined, The image representation is rotated within the user interface based at least partially on the amount by which the scope has been rotated. One or more non-temporary computer-readable media that cause the scope to rotate by at least a portion of the amount by which it has rotated to perform an action including rotating the work channel indicator within the user interface from a first orientation to a second orientation. (25) The above operation is The further includes determining that the scope has completed its rotation, One or more non-temporary computer-readable media according to Embodiment 24, wherein, upon determination that the scope has completed its rotation, the image representation and the work channel indicator are rotated.

[0156] (26) Rotating the image representation includes rotating the image representation continuously, one or more non-temporary computer-readable media according to Embodiment 24. (27) Determining the amount by which the scope has rotated includes tracking the amount of rotation of the scope based on at least one of feedback data from a robotic arm configured to be coupled to the scope, sensor data generated by a position sensor associated with the scope, or a control signal generated to control the robotic arm, according to one or more non-temporary computer-readable media as described in Embodiment 24. (28) One or more non-temporary computer-readable media according to Embodiment 24, wherein rotating the image representation includes displaying the image representation in a cropped form on the user interface while rotating the scope. (29) The above operation is The input device receives user input to rotate the scope by a predetermined amount, One or more non-temporary computer-readable media according to Embodiment 24, further comprising controlling the scope to rotate by the predetermined amount. (30) One or more non-temporary computer-readable media according to Embodiment 24, wherein the work channel indicator is displayed in an overlay manner on the image representation.

Claims

1. A control system, A communication interface configured to receive image data from an imaging device associated with a medical device, A control circuit that is communicatively coupled to the aforementioned communication interface, Based on the aforementioned image data, an image representation is generated. The image representation is displayed on the user interface, The change in rotation of the aforementioned medical device is determined, The system comprises a control circuit configured to rotate the image representation relative to the user interface based at least partially on the change in rotation of the medical device, The aforementioned control circuit is The medical device is configured to determine when the rotation has been interrupted. A control system in which, upon determination that the rotation of the medical device has been interrupted, the image representation is rotated.

2. The control system according to claim 1, wherein the image representation is rotated simultaneously with the medical device changing its rotation.

3. The control system according to claim 1, wherein the control circuit is configured to determine the change in the rotation of the medical device based on feedback from a robot arm coupled to the medical device, sensor data generated by a position sensor associated with the medical device, or a control signal associated with the control of the robot arm.

4. The aforementioned control circuit is The input device receives an input control signal for rotating the medical instrument. The control system according to claim 1, configured to adjust the rotation of the medical device based at least partially on the input control signal.

5. The control system according to claim 4, wherein the medical device is rotated by a predetermined amount defined by the input control signal.

6. A method for operating a control system, The communication interface of the control system receives image data from an imaging device associated with a medical device, The control circuit of the control system generates an image representation based on the image data, The control circuit displays the image representation on the user interface, The control circuit determines the change in rotation of the medical device, The control circuit includes rotating the image representation relative to the user interface based at least partially on the change in the rotation of the medical device, The aforementioned control circuit is The medical device is configured to determine when the rotation has been interrupted. A method in which, upon determination that the rotation of the medical device has been interrupted, the image representation is rotated.

7. It is a system, Medical devices and An imaging device associated with the aforementioned medical device, A control circuit that is communicatively coupled to the medical device and the imaging device, Receiving image data from the aforementioned imaging device, To generate an image representation based on the aforementioned image data, Displaying the aforementioned image representation in the user interface, To determine the change in rotation of the aforementioned medical device, A control circuit is configured to rotate the image representation relative to the user interface based at least partially on the change in rotation of the medical device, The aforementioned control circuit is The medical device is configured to determine when the rotation has been interrupted. A system in which, upon determination that the rotation of the medical device has been interrupted, the image representation is rotated.

Images