Visualized adjustment of instrument roll-over

Abstract

The technology relates to providing image data, indicators, and / or other data associated with a medical instrument having a proper orientation to assist a user in performing a procedure. For example, a user interface can present: an image representation to represent image data from a scope; and / or a working channel indicator to indicate a rotation angle of a working channel of the scope. When the scope is flipped, a rotation adjustment can be applied to the image representation to rotate the image representation within the user interface. Further, the working channel indicator can be rotated within the user interface.

Description

[0001] Related applications This application claims priority to U.S. Provisional Application No. 63 / 117,658, filed November 24, 2020, entitled “VISUALIZATION ADJUSTMENTSFOR INSTRUMENT ROLL”, the disclosure of which is incorporated herein by reference in its entirety. Background Technology

[0002] Various medical procedures involve the use of one or more medical devices to examine and / or treat a patient. In some cases, multiple systems / devices are implemented to control the medical devices in performing procedures on a patient. Improper use of such systems, devices, and / or medical devices may adversely affect the patient's health and / or the effectiveness of the procedure. Summary of the Invention

[0003] In some embodiments, this disclosure relates to a control system comprising: a communication interface configured to receive image data from an imaging device associated with a scope, the scope including a working channel for deploying medical devices; and control circuitry communicatively coupled to the communication interface. The control circuitry is configured to: generate an image representation of the image data; display the image representation in a user interface; determine the amount by which the scope has been flipped; and rotate the image representation within the user interface, at least in part based on the amount by which the scope has been flipped.

[0004] In some embodiments, the control circuitry is configured to: display a working channel indicator in the user interface; and rotate the working channel indicator within the user interface from a first orientation relative to the user interface to a second orientation relative to the user interface, based at least in part on the amount by which the peephole has been flipped. The working channel indicator may indicate the rotation angle of the working channel relative to the image representation. Furthermore, in some embodiments, the control circuitry is configured to display the working channel indicator by superimposing it onto the image representation.

[0005] In some embodiments, the control circuit is configured to: determine that the peephole has completed its rotation; and in response to determining that the peephole has completed its rotation, rotate the image display within the user interface. Furthermore, in some embodiments, the control circuit is configured to rotate the image display within the user interface as the peephole rotates.

[0006] In some implementations, the control circuit is configured to rotate the image representation within the user interface by displaying the image representation in a cropped form in the user interface while flipping the peephole.

[0007] In some implementations, the control circuit is configured to determine the amount by which the peephole has been flipped based on at least one of: feedback data from a robot arm configured to be coupled to the peephole; sensor data generated by a position sensor associated with the peephole; or a control signal generated to control the robot arm.

[0008] In some embodiments, the control circuit is configured to: receive an input control signal from an input device to rotate the working channel; and control the rotation of the endoscope at least in part based on the input control signal. The control circuit may be configured to control the rotation of the endoscope by controlling the rotation of the endoscope by a predetermined amount.

[0009] In some embodiments, this disclosure relates to a method comprising: receiving image data from an imaging device associated with a hysteroscope; displaying an image representation of the image data in a user interface by control circuitry; determining, by the control circuitry, an indicator of a working channel in the user interface how much the hysteroscope has been flipped; and, at least in part based on the amount of hysteroscope flipped, rotating the image representation within the user interface; and rotating the indicator of the working channel from a first orientation to a second orientation within the user interface. The hysteroscope includes a working channel. Furthermore, the indicator of the working channel indicates the rotation angle of the working channel relative to the image representation.

[0010] In some implementations, the method further includes determining that the endoscopic mirror has been flipped. Rotation of the image representation and rotation of the working channel indicator may occur in response to determining that the endoscopic mirror has been flipped.

[0011] In some embodiments, rotating the image representation includes rotating the image representation continuously. Furthermore, in some embodiments, determining the amount by which the peephole has been flipped is based on at least one of: feedback data from a robotic arm configured to be coupled to the peephole; position data generated by a position sensor associated with the peephole; or a control signal generated to control the robotic arm. Additionally, in some embodiments, rotating the image representation includes displaying the image representation in a cropped format in the user interface while the peephole is being flipped.

[0012] In some embodiments, the method further includes: receiving user input from an input device to rotate the endoscope by a predetermined amount; and controlling the endoscope to flip by the predetermined amount. Additionally, in some embodiments, displaying a working channel indicator includes overlaying the working channel indicator onto an image representation.

[0013] In some embodiments, this disclosure relates to a system comprising: a endoscopic endoscope including a working channel for removably receiving a medical device; an imaging device configured to be coupled to the endoscopic endoscope; and control circuitry communicatively coupled to the endoscopic endoscope and the imaging device. The control circuitry is configured to: receive image data from the imaging device; display an image representation of the image data in a user interface; display a working channel indicator in the user interface; determine an amount by which the endoscopic endoscope has been flipped; rotate the working channel indicator in the user interface, at least in part based on the amount by which the endoscopic endoscope has been flipped; and apply a rotational adjustment to the image representation in the user interface, at least in part based on the amount by which the endoscopic endoscope has been flipped. The working channel indicator indicates the rotation angle of the working channel.

[0014] In some embodiments, the working channel is offset from the longitudinal axis of the endoscope. Furthermore, in some embodiments, the control circuitry is configured to: control the endoscope to flip in the rotational direction; and apply rotational adjustment to the image representation by rotating the image representation in the rotational direction within the user interface.

[0015] In some embodiments, the control circuit is further configured to determine that the endoscope has completed its flip. In response to determining that the endoscope has completed its flip, a rotational adjustment can be applied. Furthermore, in some embodiments, the control circuit is configured to apply the rotational adjustment by continuously rotating the image representation as the endoscope flips.

[0016] In some implementations, the control circuit is further configured to determine the amount by which the peephole has been flipped based on at least one of: feedback data from a robot arm configured to be coupled to the peephole; sensor data generated by a position sensor associated with the peephole; or a control signal generated to control the robot arm.

[0017] In some embodiments, the control circuit is configured to apply rotational adjustments by displaying a cropped image representation in the user interface while the endoscope is flipped. Furthermore, in some embodiments, the control circuit is configured to: receive an input control signal to rotate the working channel; and control the flipping of the endoscope at least in part based on the input control signal.

[0018] In some embodiments, this disclosure relates to one or more non-transitory computer-readable media storing computer-executable instructions that, when executed by control circuitry, cause the control circuitry to perform the following operations: receiving image data from an imaging device associated with a hysteroscope; displaying an image representation of the image data in a user interface; displaying a working channel indicator in the user interface; determining the amount by which the hysteroscope has been flipped; rotating the image representation within the user interface, at least in part based on the amount by which the hysteroscope has been flipped; and rotating the working channel indicator from a first orientation to a second orientation within the user interface, at least in part based on the amount by which the hysteroscope has been flipped. The hysteroscope includes a working channel. The working channel indicator indicates the rotation angle of the working channel.

[0019] In some embodiments, the operation further includes determining that the endoscope has completed its flip. The image display and working channel indicator may rotate in response to determining that the endoscope has completed its flip. Furthermore, in some embodiments, rotating the image display includes rotating the image display in a continuous manner.

[0020] In some embodiments, determining the amount by which the endoscope has been flipped includes tracking the amount of flip based on at least one of: feedback data from a robotic arm configured to be coupled to the endoscope; sensor data generated by a position sensor associated with the endoscope; or control signals generated to control the robotic arm. Furthermore, in some embodiments, rotating the image representation includes displaying a cropped image representation in a user interface while the endoscope is flipped.

[0021] In some embodiments, the operation further includes: receiving user input from an input device to rotate the endoscope by a predetermined amount; and controlling the endoscope to flip by a predetermined amount. Additionally, in some embodiments, a working channel indicator is displayed overlaid on the image representation.

[0022] To summarize this disclosure, certain aspects, advantages, and features have been described. It will be understood that not all such advantages may be realized according to any particular embodiment. Therefore, the disclosed embodiments may be implemented by achieving or optimizing one or more advantages taught herein without necessarily achieving other advantages that may be taught or proposed herein. Attached Figure Description

[0023] For illustrative purposes, various embodiments are depicted in the accompanying drawings and should not in any way be construed as limiting the scope of this disclosure. Furthermore, various features of different disclosed embodiments may be combined to form additional embodiments that are part of this disclosure. Throughout the drawings, reference numerals may be repeated to indicate the correspondence between reference elements.

[0024] Figure 1An exemplary medical system for performing various medical procedures is shown according to one or more embodiments.

[0025] Figure 2 The following are illustrated according to one or more implementation schemes. Figure 1 Exemplary details of the control system and robot system.

[0026] Figure 3 An exemplary endoscope is shown, which is placed in various parts of a patient's urinary system according to one or more embodiments.

[0027] Figures 4-1 to 4-9 An example is shown of a flip-up peephole according to one or more embodiments, and of applying rotational adjustments to the peephole image within the user interface once the peephole has been flipped.

[0028] Figures 5-1 to 5-5 An example is shown of a flip-up peephole according to one or more embodiments, and of applying rotational adjustments to the peephole image within the user interface while the peephole is flipped.

[0029] Figure 6 An exemplary user interface is shown according to one or more embodiments for assisting a user in controlling a hysteroscope and / or an instrument deployed through the working channel of the hysteroscope.

[0030] Figure 7 An exemplary flowchart is shown for applying rotational adjustments to a peephole view within a user interface, according to one or more embodiments. Detailed Implementation

[0031] The headings provided herein are for convenience only and do not necessarily affect the scope or meaning of this disclosure. Although certain embodiments and examples are disclosed below, the subject matter extends beyond the specific disclosed embodiments to other alternative embodiments and / or uses, as well as modifications and equivalents thereof. Therefore, the scope of the claims that may appear herein is not limited to any particular embodiment described below. For example, in any method or process disclosed herein, the actions or operations of the method or process may be performed in any suitable sequence and are not necessarily limited to any particular disclosed sequence. Various operations may then be described as multiple discrete operations in a manner that may aid in understanding a particular embodiment. However, the order of description should not be construed as implying that these operations depend on a sequence. Additionally, the structures, systems, and / or apparatuses described herein may be embodied as integrated components or separate components. Specific aspects and advantages of these embodiments are described for the purpose of comparing the various embodiments. Not all such aspects or advantages are necessarily achieved by any particular embodiment. Therefore, various embodiments may be performed, for example, by implementing or optimizing one or a set of advantages taught herein without necessarily achieving other aspects or advantages that may also be taught or proposed herein.

[0032] Regarding preferred embodiments, certain standard positional anatomical terms may be used herein to refer to the anatomy of the animal (i.e., human). Although specific spatial relative terms, such as “external,” “internal,” “upper,” “lower,” “below,” “above,” “vertical,” “horizontal,” “top,” “bottom,” and similar terms, are used herein to describe the spatial relationship of one device / element or anatomical structure to another device / element or anatomical structure, it should be understood that these terms are used herein for descriptive convenience to describe the positional relationship between elements / structures, as illustrated in the accompanying figures. It should be understood that spatial relative terms are intended to cover different orientations of elements / structures in use or operation other than those depicted in the accompanying figures. For example, an element / structure described as “above” another element / structure may indicate a position below or beside such other elements / structures relative to the subject patient or alternative orientation of the element / structure, and vice versa.

[0033] Overview Many medical procedures involve the use of a endoscopic endoscope to examine and / or treat patients. An endoscope typically includes or is associated with an imaging device configured to capture image data of a target site, such as a camera located at the distal end of the endoscope. The image data can be displayed to the user to assist in controlling the endoscope within the patient's body. The endoscope may also include one or more working channels through which additional tools, such as lithotripters, baskets, forceps, laser devices, etc., can be deployed to the treatment site. The working channels are typically offset from the longitudinal axis of the imaging device and / or the endoscope. Furthermore, in some cases, the endoscope may be configured to move based on different joint planes of motion, allowing the endoscope to move with more / fewer degrees of freedom in one direction when oriented in a particular manner.

[0034] To accommodate various endoscope movements, treatment angles, physician preferences, and / or other scenarios, the endoscope can be controlled to flip within the patient, thereby flipping the associated imaging devices and working channels. In one example, the user can flip the endoscope to better position tools / instruments deployed through the working channel (offset to one side of the endoscope) for treating the target site. In another example, the user can flip the endoscope to reorient its articulated planes of motion, allowing for additional movement in one direction. Because users often control the endoscope based on image data captured by it, flipping the endoscope can cause the user to lose their sense of orientation within the patient's body. For example, the image data displayed to the user will flip along with the endoscope, causing the user to lose track of the virtual horizon / reference frame at the treatment site.

[0035] This disclosure relates to systems, apparatus, and methods for providing image data and / or other data associated with a medical device having a suitable orientation to assist a user in performing procedures. For example, the system may be configured to communicate with a endoscopic device including a working channel for deploying the medical device. The system may receive image data from an imaging device associated with the endoscopic device and display an image representation of the image data via a user interface. When the endoscopic device is flipped, the system may adjust the orientation of the image representation, such as by rotating the image representation within the user interface, which may or may not be perceived occurring in the user interface. The system may also display a working channel indicator via the user interface, indicating the rotation angle of the working channel (e.g., the position of the working channel relative to the image representation). As the endoscopic device is flipped, the system may adjust the orientation of the working channel indicator within the user interface (e.g., rotate the working channel indicator) to indicate the accurate orientation of the working channel relative to the image representation. In some cases, to give the impression that a virtual horizon / reference frame (such as a horizon / reference frame displayed within the image representation) is maintained, the image representation and / or the working channel indicator are adjusted in a continuous manner in the user interface while the endoscopic device is flipped. For example, a user might perceive only the working channel indicator as moving within the user interface (e.g., the working channel indicator rotates while the virtual horizontal line within the image representation remains unchanged). In other cases, the image representation and / or the working channel indicator may rotate within the user interface when the endoscope has been flipped or otherwise reached a specific rotation angle, and / or when another event occurs. Therefore, these technologies can present image data, the working channel indicator, and / or other data associated with an endoscope of appropriate orientation to assist the user in maintaining a frame of reference within the patient's body.

[0036] While many techniques have been discussed in the context of presenting indicators for the working channels of a medical device, these techniques may additionally or alternatively present indicators for other components. For example, these techniques may present indicators for joint motion planes associated with the medical device, lights associated with the medical device, and needles / catheters / other instruments connected to the medical device.

[0037] In some specific implementations, the techniques disclosed herein realize robot-assisted medical procedures in which robotic tools enable physicians to perform endoscopy and / or percutaneous access and / or treatment at target anatomical sites. For example, a robotic tool may engage and / or control one or more medical devices (such as endoscopes) to access target sites within a patient's body and / or perform treatment at those sites. In some cases, the robotic tool is guided / controlled by the physician. In others, the robotic tool operates in an automated or semi-automatic manner. While many techniques are discussed in the context of robot-assisted medical procedures, these techniques are applicable to other types of medical procedures, such as those that do not implement robotic tools or implement robotic tools for a relatively small number of operations (e.g., less than a threshold number). For example, these techniques are applicable to procedures that implement manually operated medical devices, such as manually operated endoscopes that are entirely controlled by the physician.

[0038] Certain aspects of this disclosure are described herein in the context of renal, urological, and / or nephrological procedures (such as kidney stone removal / treatment procedures). However, it should be understood that such context is provided for convenience, and the concepts disclosed herein apply to any suitable medical procedure. For example, the following description also applies to other surgical / medical procedures or medical procedures involving the removal of objects from a patient's body (including any object that can be removed from a treatment site or patient cavity (e.g., esophagus, urinary catheter, intestine, eye, etc.) via percutaneous and / or endoscopic access), such as, for example, gallbladder stone removal, lung (lung / transthoracic) tumor biopsy, or cataract extraction. However, as mentioned, a description of renal / urinary anatomy and associated medical issues and procedures is presented below to aid in the description of the concepts disclosed herein.

[0039] Exemplary medical system Figure 1 An exemplary medical system 100 for performing various medical procedures is illustrated according to aspects of this disclosure. The medical system 100 includes a robotic system 110 configured to engage and / or control one or more medical devices / appliances to perform procedures on a patient 120. Figure 1In the example, the robotic system 110 is coupled to the endoscope 130 and the electromagnetic (EM) field generator 140. The medical system 100 also includes a control system 150 configured to interface with the robotic system 110, providing information about the procedure and / or performing various other operations. For example, the control system 150 may include a display 152 configured to present information to assist the physician 160 in performing the procedure. The medical system 100 may include a table 170 (e.g., a hospital bed) configured to support the patient 120. Various actions are described herein as being performed by the physician 160. These actions may be performed directly by the physician 160, a user under the guidance of the physician 160, another user (e.g., a technician), combinations thereof, and / or any other user. The devices / components of the medical system 100 may be arranged in various ways depending on the type and / or stage of the procedure.

[0040] Control system 150 can operate in conjunction with robotic system 110 to perform medical procedures on patient 120. For example, control system 150 can communicate with robotic system 110 via wireless or wired connections to control medical devices connected to robotic system 110, receive images captured by the medical devices, etc. For example, control system 150 can receive image data from endoscope 130 (e.g., an imaging device associated with endoscope 130) and display the image data (and / or representations generated therefrom) to physician 160 to assist physician 160 in navigating endoscope 130 within patient 120. Physician 160 can provide input via input / output (I / O) devices such as controller 154, and control system 150 can send control signals to robotic system 110 to control the movement of endoscope 130 connected to robotic system 110. Endoscope 130 (and / or another medical device) can be configured to move in various ways, such as in-plane jointing, rotation, etc., as discussed in further detail below.

[0041] In some implementations, the control system 150 may provide fluid to the robotic system 110 via one or more fluid channels, provide power to the robotic system 110 via one or more electrical connections, provide optics to the robotic system 110 via one or more optical fibers or other components, etc. In various examples, the control system 150 may communicate with a medical device to receive sensor data (via the robotic system 110 and / or directly from the medical device). The sensor data may indicate or be used to determine the position and / or orientation of the medical device. Furthermore, in various examples, the control system 150 may communicate with a tabletop 170 to position the tabletop 170 in a specific orientation or otherwise control the tabletop 170. Additionally, in various examples, the control system 150 may communicate with an EM field generator 140 to control the generation of an EM field around the patient 120.

[0042] The robotic system 110 may include one or more robotic arms 112 configured to engage and / or control a medical device / medical apparatus. Each robotic arm 112 may include multiple arm segments coupled to joints, providing multiple degrees of mobility. The distal end of the robotic arm 112 (e.g., an end effector) may be configured to couple to the instrument / apparatus. Figure 1 In this example, robotic arm 112(A) is coupled to an EM field generator 140, which can be configured to generate an EM field detected by sensors on a medical device, such as an EM sensor on endoscope 130. A second robotic arm 112(B) is coupled to an endoscope actuator device coupling 132, which facilitates robotic control / propulsion of the endoscope 130. Furthermore, a third robotic arm 112(C) is coupled to a handle 134 of the endoscope 130, which can be configured to facilitate the propulsion and / or manipulation of medical devices that can be deployed through the endoscope 130, such as devices deployed through the working channel of the endoscope 130. In this example, the second robotic arm 112(B) and / or the third robotic arm 112(C) can control the movement of the endoscope 130 (e.g., joint movements, rotation, etc.). Although in Figure 1 Three robotic arms are connected to a specific medical device, but the robotic system 110 may include any number of robotic arms configured to connect to any type of medical device / medical apparatus.

[0043] The robotic system 110 can be communicatively coupled to any component of the medical system 100. For example, the robotic system 110 can be communicatively coupled to the control system 150 to receive control signals from the control system 150 to perform operations, such as controlling the robotic arm 112 in a specific manner, manipulating medical instruments, etc. Furthermore, the robotic system 110 can be configured to receive images (also referred to as image data) depicting the internal anatomy of the patient 120 from the endoscope 130 and / or transmit such images to the control system 150, which can then be displayed on the display 152. Additionally, the robotic system 110 can allow fluids, optics, electricity, etc., to be received therefrom to be coupled to components of the medical system 100, such as the control system 150.

[0044] Medical devices can include a wide variety of instruments, such as endoscopes (sometimes called "endoscopy"), catheters, needles, guidewires, lithotripters, basket retrieval devices, forceps, vacuum devices, scalpels, imaging probes, imaging devices, jaws, scissors, graspers, needle holders, microscalpels, staplers, flathead staplers, aspiration / irrigation tools, clamp applicators, etc. Medical devices can include direct-access instruments, percutaneous instruments, and / or another type of instrument. In some embodiments, a medical device is a manipulable device, while in other embodiments, it is a non-manipulable device. In some embodiments, a surgical instrument refers to a device configured to puncture or insert through human anatomical structures, such as needles, scalpels, guidewires, etc. However, a surgical instrument can refer to other types of medical devices.

[0045] The term "spectoscope" or "endoscope" can refer to any type of elongated medical device having image generation, viewing, and / or capture capabilities (or configured to provide such capabilities by an imaging device deployed via a working channel) and configured to be introduced into any type of organ, cavity, lumen, chamber, and / or space of the body. For example, a spectoscope or endoscope (such as spectoscope 130) can refer to a ureteroscope (e.g., for access to the urinary tract), a laparoscope, a nephroscope (e.g., for access to the kidney), a bronchoscope (e.g., for access to the airway, such as the bronchus), a colonoscope (e.g., for access to the colon), an arthroscope (e.g., for access to a joint), a cystoscope (e.g., for access to the bladder), a tubing endoscope, etc. In some cases, a spectoscope / endoscope may include a rigid or flexible tube, and / or its size may be set to pass within an outer sheath, catheter, guide, or other lumen-type device, or it may be used without such a device. In some embodiments, the endoscope includes one or more working channels through which additional tools / medical instruments, such as lithotripters, basket devices, forceps, laser devices, imaging devices, etc., can be introduced into the treatment site. In some cases, the working channels are offset from the longitudinal axis of the imaging device and / or the endoscope. Furthermore, in some cases, the endoscope can be configured to move based on different joint motion planes, such that the endoscope moves with more / fewer degrees of freedom in a particular direction when oriented in a particular manner. For example, the endoscope can be associated with a primary joint motion plane and a secondary joint motion plane, wherein the tip of the endoscope is configured to move with greater degrees of freedom (e.g., with greater skewness, larger bending radius, etc.) in the primary joint motion plane compared to the secondary joint motion plane. For example, the primary joint motion plane may be perpendicular to the secondary joint motion plane.

[0046] The term "direct entry" or "direct access" can refer to any entry of an instrument through a natural or artificial opening in the patient's body. For example, a speculum 130 can be called a direct entry instrument because it enters the patient's urinary tract via the urethra.

[0047] The term "percutaneous entry" or "percutaneous access" can refer to the entry through the patient's skin and any other body layers necessary for an instrument to reach a target anatomical location (e.g., the renal calyx network of the kidney) associated with a procedure, such as entry through a puncture and / or small incision. Therefore, a percutaneous entry device can refer to a medical device, apparatus, or component configured to puncture or penetrate the skin and / or other tissue / anatomical structures, such as a needle, scalpel, guidewire, sheath, shaft, endoscope, catheter, etc. However, it should be understood that a percutaneous entry device can refer to other types of medical devices in the context of this disclosure. In some embodiments, a percutaneous entry device refers to an instrument / apparatus that utilizes means to facilitate insertion or realization through a puncture and / or small incision through the patient's skin. For example, a catheter may be referred to as a percutaneous entry device when inserted through a sheath / shaft that has punctured the patient's skin.

[0048] In some embodiments, the medical device includes a sensor (also referred to as a “position sensor”) configured to generate sensor data. In examples, the sensor data may indicate the position and / or orientation of the medical device and / or be used to determine the position and / or orientation of the medical device. For example, the sensor data may indicate the position and / or orientation of an endoscope, which may indicate the flipping of the distal end of the endoscope. The position and orientation of the medical device may be referred to as the orientation of the medical device. The sensor may be positioned at the distal end of the medical device and / or any other location. In some embodiments, the sensor may provide sensor data to a control system 150, a robotic system 110, and / or another system / device to perform one or more positioning techniques to determine / track the position and / or orientation of the medical device.

[0049] In some embodiments, the sensor may include an electromagnetic (EM) sensor having a coil of conductive material. Here, an EM field generator (such as EM field generator 140) can provide an EM field detected by the EM sensor on the medical device. The magnetic field can induce a small current in the coil of the EM sensor, and this small current can be analyzed to determine the distance and / or angle / or orientation between the EM sensor and the EM field generator. Alternatively, the sensor may include another type of sensor, such as a camera, a distance sensor (e.g., a depth sensor), a radar device, a shape-sensing fiber optic cable, an accelerometer, a gyroscope, a satellite-based positioning sensor (e.g., a Global Positioning System (GPS)), a radio frequency transceiver, etc.

[0050] In some implementations, the medical system 100 may also include an imaging device ( Figure 1 (Not shown in the image), the imaging device may be integrated into a C-arm and / or configured to provide imaging during procedures such as fluoroscopy. The imaging device may be configured to capture / generate one or more images of the patient 120 during the procedure, such as one or more X-ray or CT images. In the example, images from the imaging device may be provided in real time to observe anatomical structures and / or medical instruments within the patient 120 to assist the physician 160 in performing the procedure. The imaging device may be used to perform fluoroscopy (e.g., using contrast dyes within the patient 120) or another type of imaging technique.

[0051] The various components of the medical system 100 can be communicatively coupled to each other via a network, which may include wireless and / or wired networks. Exemplary networks include one or more personal area networks (PANs), local area networks (LANs), wide area networks (WANs), Internet local area networks (IANs), cellular networks, the Internet, etc. Furthermore, in some embodiments, the components of the medical system 100 are connected via one or more support cables, conduits, etc., for data communication, fluid / gas exchange, power exchange, etc.

[0052] In some implementations, the medical system 100 may provide information to assist a physician 160 in controlling the endoscope 130 and / or tools deployed through the endoscope 130. For example, the control system 150 may display an image representation 154 in an interface 156, wherein the image representation 154 includes or is based on image data captured by an imaging device associated with the endoscope 130. The imaging device may be mounted on the distal end of the endoscope 130, deployed outside the distal end of the endoscope 130 via the working channel of the endoscope 130, or otherwise associated with the endoscope 130. Figure 1In the example, image representation 154 depicts an internal view of the kidney 190 of patient 120 (i.e., from the perspective of the endoscope 130 within the renal calyx 191 of the kidney 190). Furthermore, control system 150 can provide a working channel indicator 158 that indicates the rotation angle of the working channel of the endoscope 130 relative to image representation 154. For example, the position of the working channel indicator 158(A) within interface 156(A) indicates: in Figure 1 In the image, the working channel is located on the right relative to image representation 154 (A). Although Figure 1 Not shown, but the control system 150 may also provide other types of indicators, such as indicators for the orientation of the articular motion plane of the endoscope 130, indicators for the orientation of the lamp associated with the endoscope 130, indicators for the position / orientation of the needle / catheter / other instrument connected to the endoscope 130, etc.

[0053] In some embodiments, the control system 150 may adjust the image representation 154 and / or the working channel indicator 158 to reflect changes in the orientation of the endoscope 130. For example, when the endoscope 130 is flipped, the control system 150 may rotate the working channel indicator 158 to accurately indicate the orientation of the working channel relative to the image representation 154. Furthermore, the control system 150 may adjust the orientation of the image representation 154 to provide the same virtual horizon / reference frame before and after flipping. For example, when the endoscope 130 is flipped, the control system 150 may rotate the image representation 154 such that the view of the internal anatomy of the patient 120 has the same orientation before and after the endoscope 130 is flipped.

[0054] For illustration, suppose the endoscope 130 is positioned within the kidney 190 near the kidney stone 192, and includes a laser device deployed via a working channel to break up the kidney stone 192, such as... Figure 1As shown. Similarly, assume that the endoscope 130 is initially positioned in a first orientation that allows the laser from the laser device to miss the kidney stone 192 (i.e., extending along path 193 that contacts the anatomical structure of the patient 120). Here, the control system 150 may display interface 156(A), in which the working channel indicator 158(A) is positioned on the right. To adjust the orientation of the laser device, the physician 160 may flip the endoscope 130 to a second orientation, in which the laser from the laser device can contact the kidney stone 192 (i.e., extending along path 194). In this illustration, the endoscope 130 is flipped 180 degrees. However, the endoscope 130 can be flipped any degree. The control system 150 may then present an updated user interface 156(B), which includes an updated working channel indicator 158(B). The control system 150 may also rotate the image data 156 by 180 degrees to present the same horizontal line within the image data 156, as shown in the updated user interface 156(B). That is, the control system 150 can adjust the orientation of the image data 156 within the user interface 156 so that the frame of reference within the patient 120 is maintained (e.g., the kidney stone 192 appears on the left side both before and after rotating the endoscope 130). The frame of reference within the patient 120 can be maintained by adjusting the image representation 154 and / or the working channel indicator 158 in this manner. In other solutions, such a flip of the endoscope 130 would rotate the image representation 154 180 degrees from its initial orientation, which could disorient the physician 160.

[0055] In some implementations, after the endoscope 130 has been flipped, the control system 150 can adjust the orientation of the image display 154 (and / or the working channel indicator 158). For example, in response to determining that the endoscope 130 has been flipped 180 degrees, the control system 150 can rotate the image display 154 to the view shown in the user interface 156(B). Here, the physician 160 can perceive the content in the image display 154 as rotated 180 degrees and then rotated back 180 degrees after the endoscope 130 has been flipped. (See below...) Figures 4-1 to 4-9 Examples of such orientation adjustments are discussed. Alternatively, in some embodiments, the control system 150 may continuously update the user interface 156 to maintain a frame of reference within the patient 120. For example, the control system 150 may track the rotation of the endoscope 130 and continuously update the image representation 154 to maintain a virtual horizontal line within the patient 120 throughout the entire rotation of the endoscope 130. Here, the physician 160 may perceive the image representation 154 as remaining constant, while the working channel indicator 158 rotates around the image representation 154. (See below) Figures 5-1 to 5-5 Examples of such continuous orientation adjustments are discussed.

[0056] In some examples, medical system 100 is implemented to perform medical procedures related to kidney anatomy, such as treating kidney stones. Kidney stones (also known as urolithiasis) are a medical condition involving the formation of solid masses of material in the urinary tract, called “kidney stones” (renal calculi, renal lithiasis, or nephrolithiasis) or “urinary stones.” Urinary stones can form and / or be found in the kidneys, ureters, and bladder (called “bladder stones”). These urinary stones can form due to the concentration of minerals in the urine, and can cause significant abdominal pain once they reach a size large enough to obstruct the flow of urine through the ureter or urethra. Urinary stones can be formed from calcium, magnesium, ammonia, uric acid, cysteine, and / or other compounds or combinations thereof.

[0057] There are generally several methods available for treating patients with kidney stones, including observation, medical treatments (such as stone removal therapy), non-invasive treatments (such as extracorporeal shock wave lithotripsy (ESWL)), and surgical treatments (such as ureteroscopy and percutaneous nephrolithotomy (“PCNL”)). In surgical methods (e.g., ureteroscopy and PCNL), the physician is able to access the lesion (i.e., the object to be removed; for example, a stone), break the stone into smaller fragments or pieces, and mechanically remove the relatively small stone fragments / particles from the kidney.

[0058] To remove urinary stones from the bladder and ureters, a physician inserts a ureteroscope into the urinary tract through the urethra. Typically, a ureteroscope includes an endoscope at its distal end, configured to visualize the urinary tract. The ureteroscope may also include a lithotripsy device for capturing or breaking up the urinary stones. During this procedure, one physician / technician controls the position of the ureteroscope while another physician / technician controls the lithotripsy device. To remove relatively large stones from the kidneys (i.e., “kidney stones”), a physician may use percutaneous nephrolithotomy (“PCNL”) techniques, which involve inserting a nephroscope through the skin (i.e., percutaneously) and intervening in the tissue to provide access to the treatment site for breaking up and / or removing the stone.

[0059] In several of the examples described herein, robot-assisted percutaneous procedures can be implemented in conjunction with various medical procedures, such as kidney stone removal procedures, wherein robotic tools (e.g., one or more components of medical system 100) enable physicians / urologists to perform endoscopic (e.g., ureteroscopy) target access as well as percutaneous access / treatment. However, this disclosure is not limited to kidney stone removal and / or robot-assisted procedures. In some specific implementations, robotic medical solutions may offer relatively higher precision, superior control, and / or superior hand-eye coordination relative to certain instruments compared to strictly manual procedures. For example, robot-assisted percutaneous access to the kidney according to some procedures can advantageously enable urologists to perform both direct-access endoscopic kidney access and percutaneous kidney access. Although some embodiments of this disclosure are presented in the context of catheters, nephroscopy, ureteroscopy, and / or human kidney anatomy, it should be understood that the principles disclosed herein can be implemented in any type of endoscopic / percutaneous procedure or another type of procedure.

[0060] In an exemplary, non-limiting procedure, medical system 100 can be used to remove kidney stones 192 from a patient 120. During the setup of this procedure, physician 160 can position the robotic arm 112 of robotic system 110 according to a desired configuration and / or attach appropriate medical devices. For example, as Figure 1 As shown, physician 160 can position a first robotic arm 112(A) near the treatment site and attach an EM field generator 140, which can assist in tracking the position of the endoscope 130 and / or other instruments / devices during the procedure. Additionally, physician 160 can position a second robotic arm 112(B) between the legs of patient 120 and attach an endoscope actuator device coupling 132, which facilitates robotic control / propulsion of the endoscope 130. Physician 160 can insert a medical device 136 into the urethra 195 of patient 120, through the bladder 196, and / or up to the ureter 197. Physician 160 can connect the medical device 136 to the endoscope actuator device coupling 132. The medical device 136 may include an endoscope-type device configured to receive the endoscope 130 (e.g., through an insert sheath), thereby assisting in inserting the endoscope 130 into the anatomy of patient 120. Although the medical device 136 is in Figure 1 As shown, but in some embodiments, medical device 136 is not used (e.g., the endoscope 130 is inserted directly into the urethra 195). The physician 160 can then insert the endoscope 130 into medical device 136 manually, automatically, or a combination of both. The physician 160 can attach the handle 134 of the endoscope 130 to a third robotic arm 112(C), which can be configured to facilitate the advancement and / or manipulation of a basket-loading device, a laser device, and / or another medical device deployed via the endoscope 130.

[0061] Physician 160 can interact with control system 150 to enable robotic system 110 to advance and / or navigate endoscope 130 into kidney 190. For example, physician 160 can navigate endoscope 130 to locate kidney stone 192. Control system 150 can provide information about endoscope 130 via display 152 to assist physician 160 in navigating endoscope 130, such as interface 156 for viewing image representation 154 (e.g., real-time images captured by endoscope 130), working channel indicator 158 (e.g., indicating the current orientation of the working channel), etc. In some embodiments, control system 150 may use positioning techniques to determine the position and / or orientation of endoscope 130, which can be viewed by physician 160 via display 152. In addition, other types of information, such as X-ray images of the internal anatomy of patient 120, may also be presented via display 152 to assist physician 160 in controlling endoscope 130.

[0062] Once located at the site of the kidney stone 192 (e.g., within the renal calyx 191 of the kidney 190), the endoscope 130 can be used to designate / mark the target location for percutaneous catheter access into the kidney 190. To minimize damage to the kidney 190 and / or surrounding anatomy, the physician 160 may designate the papilla as the target location for percutaneous access into the kidney 190. However, other target locations may be designated or identified. In some embodiments where the papilla is designated, the physician 160 may navigate the endoscope 130 to contact the papilla, and the control system 150 may use positioning techniques to determine the position of the endoscope 130 (e.g., the position of the distal end of the endoscope 130), and the control system 150 may associate the position of the endoscope 130 with the target location. Furthermore, in some embodiments, the physician 160 may navigate the endoscope 130 to a specific distance from the papilla (e.g., parking it in front of the papilla) and provide input indicating that the target location is within the field of view of the endoscope 130. The control system 150 may perform image analysis and / or other positioning techniques to determine the location of the target location. In addition, in some implementations, the endoscopic lens 130 can provide a reference point to mark the nipple as the target location.

[0063] Once the target location is specified, a catheter (not shown) can be inserted into the patient 120 via a percutaneous access path to reach the target site. For example, the catheter may be connected to a first robotic arm 112(A) (after the EM field generator 140 is removed), and the physician 160 may interact with the control system 150 to advance and / or navigate the catheter using the robotic system 110. In some embodiments, a needle or another medical device is inserted into the patient 120 to create a percutaneous access path. The control system 150 may provide information about the catheter via a display 152 to assist the physician 160 in navigating the catheter. For example, the interface may provide image data from the perspective of a endoscopic endoscope 130. The image data may depict the catheter (e.g., when it is within the field of view of the imaging device of the endoscopic endoscope 130).

[0064] When the endoscope 130 and / or catheter are in the target location, the physician 160 can use the endoscope 130 to break up the kidney stone 192 and / or use the catheter to remove fragments of the kidney stone 192 from the patient 120. For example, the endoscope 130 may have a working channel to deploy tools (e.g., laser, cutting instruments, etc.) to break the kidney stone 192 into fragments, and the catheter may have aspirate the fragments from the kidney 190 via a percutaneous access path. In various examples, the catheter and / or endoscope 130 may provide flushing and / or aspiration to facilitate the removal of the kidney stone 192. For example, the catheter (and / or associated medical device) may be coupled to a flushing and / or aspiration system.

[0065] At any point before, during, or after the procedure, the physician 160 may flip the endoscope 130 to provide a better position / orientation. In one example, when positioning the endoscope 130 to break / remove kidney stone 192, the physician 160 may flip the endoscope 130 to reposition the working channel of the endoscope 130, thereby providing a better orientation for instrument deployment through the working channel. In another example, the endoscope 130 may be associated with different joint motion planes (e.g., primary and secondary planes). Here, the physician 160 may control the flipping of the endoscope 130 to reposition the joint motion planes, allowing the endoscope 130 to perform joint movements in a better manner (e.g., bending with a greater degree of skew). In some cases, the physician 160 may provide input to rotate the endoscope 130 by a predetermined amount, such as to flip the working channel 180 degrees. In other cases, the physician 160 may provide input to rotate the endoscope 130 by any number of degrees. In any situation, when the endoscope 130 is flipped, the control system 150 can update the working channel indicator 158 in the user interface 156 to maintain the accurate orientation of the working channel indicator 158 relative to the image representation 154. Furthermore, the control system 150 can adjust the orientation of the image representation 154 to maintain its frame of reference. This allows the physician 160 to maintain consistent orientation and view the accurate orientation of the working channel relative to the image representation 154 / user interface within the kidney 190 of the patient 120.

[0066] In some implementations, the control system 150 and / or the robot system 110 prevent the endoscope 130 from flipping (and thus the working channel from flipping) when the endoscope 130 articulates more than a certain degree (e.g., 20 degrees, 30 degrees, etc.) relative to its longitudinal axis. For example, when user input is received to flip the working channel of the endoscope 130, the control system 150 and / or the robot system 110 can prevent the working channel from flipping if the endoscope 130 articulates more than 30 degrees. However, in some cases, the endoscope 130 may flip when articulated in any amount.

[0067] While many examples have been discussed in the context of displaying data associated with a endoscopic endoscope (e.g., displaying image data associated with a endoscopic endoscope, displaying working channel indicators associated with a endoscopic endoscope, etc.), these techniques can be implemented in the context of other medical devices. For example, these techniques can be implemented to present information associated with another type of medical device that includes / deploys an imaging device.

[0068] Furthermore, while many examples present indicators for the working channel, other types of indicators can be presented. For example, these techniques can display joint motion plane indicators indicating the orientation of the joint motion plane of a medical device relative to an image representation within the user interface; lamp indicators for lights positioned on the distal end of the medical device (indicating the orientation of the lamp / catheter / instrument relative to the image representation); needle / catheter / instrument indicators for needles / catheters / other instruments connected to the medical device (indicating the orientation of the needle / catheter / instrument relative to the image representation), etc. As the endoscope / medical device is flipped, such indicators can be oriented / updated within the user interface to accurately indicate the orientation of the item represented by the indicator. In some cases, the needle / catheter / instrument connected to the endoscope may be outside the endoscope's field of view, and therefore, from the endoscope's perspective, the needle / catheter / instrument is not displayed in the image representation (e.g., the needle / catheter / instrument is off-screen). Here, the needle / catheter / instrument indicator can still maintain the proper orientation within the image representation before and after flipping the endoscope. For example, a needle / catheter / instrument indicator may be positioned on the right side of the user interface (indicating the needle / catheter / instrument is off-screen on the right). Assuming the needle / catheter / instrument is not moved, this positioning of the needle / catheter / instrument indicator can be the same before and after flipping the endoscope. In each example, position / orientation information about the needle / catheter / instrument is obtained based on a position sensor (such as an EM sensor or shape sensing) within the needle / catheter / instrument.

[0069] In many examples, the flipping of the peephole 130 is controlled by a robotic arm 112, while the robotic arm 112 remains in a relatively stationary position (e.g., the end effector of the robotic arm 112 may remain in the same position). For example, the robotic arm 112 may control / drive the flipping feature / mechanism on the peephole 130 to flip the peephole 130. However, in some cases, the peephole 130 may be flipped by flipping the end effector of the robotic arm 112. For example, the end effector may be flipped relative to the longitudinal axis of the peephole 130, such that the handle 134 of the peephole 130 flips about the longitudinal axis.

[0070] Medical system 100 offers a variety of benefits, such as providing guidance to assist physicians in performing procedures (e.g., instrument tracking, instrument navigation, instrument calibration, etc.), enabling physicians to perform procedures from ergonomic positions without awkward arm movements and / or positioning, enabling a single physician to perform procedures using one or more medical instruments, avoiding radiation exposure (e.g., associated with fluorescein microscopy techniques), enabling procedures to be performed in a single surgical setting, and providing continuous aspiration for more efficient removal of objects (e.g., removal of kidney stones). For example, medical system 100 can provide guidance to assist physicians in using various medical instruments to access target anatomical features while minimizing bleeding and / or damage to anatomical structures (e.g., critical organs, blood vessels, etc.). Furthermore, medical system 100 can provide radiation-free navigation and / or positioning techniques to reduce radiation exposure for physicians and patients and / or reduce the number of devices in the operating room. Additionally, medical system 100 can provide functionality distributed between at least control system 150 and robotic system 110, which may be capable of independent movement. Such a distribution of functionality and / or mobility enables the control system 150 and / or robotic system 110 to be placed in an optimal location for a particular medical procedure, which maximizes the work area around the patient and / or provides an optimal location for the physician to perform the procedure.

[0071] While various technologies / systems are discussed for implementing robot-assisted procedures (e.g., procedures that at least partially utilize medical system 100), these technologies / systems can be implemented in other procedures, such as fully robotic medical procedures, human-only procedures (e.g., non-robotic systems), etc. For example, medical system 100 can be used to perform procedures without a physician holding / manipulating medical devices and without the physician controlling the movement of the robotic system / arm (e.g., a fully robotic procedure that relies on relatively little input to guide the procedure). That is, the medical devices used during the procedure can be held / controlled by components of medical system 100, such as the robotic arm 112 of robotic system 110.

[0072] Exemplary control systems and robot systems Figure 2 The following are illustrated according to one or more implementation schemes. Figure 1 Exemplary details of the control system 150 and the robot system 110. Although Figure 2 Certain components of the control system 150 and / or robot system 110 are shown, but it should be understood that additional components not shown may be included in embodiments of this disclosure. Furthermore, any of the shown components may be omitted, interchanged, and / or integrated into other devices / systems, such as tabletop 170, medical devices, etc.

[0073] Reference Figure 2The control system 150 may include, individually and / or in combination / collectively, one or more of the following components, devices, modules, and / or units (referred to herein as “components”): one or more I / O components 202, one or more communication interfaces 204, one or more power supply units 206, and / or one or more moving components 208 (e.g., casters or other types of wheels). In some embodiments, the control system 150 may include a housing / enclosure configured and / or sized to accommodate or contain at least a portion of one or more components of the control system 150. In this example, the control system 150 is shown as a trolley-type system that can be moved by one or more moving components 208. In some cases, after reaching the appropriate position, one or more moving components 208 may be fixed in place using wheel locks to hold the control system 150 in place. However, the control system 150 may be implemented as a fixed system, integrated into another system / device, etc.

[0074] Various components of the control system 150 may be electrically and / or communicatively coupled using certain connection circuits / devices / features, which may or may not be part of the control circuitry. For example, connection features may include one or more printed circuit boards configured to facilitate the mounting and / or interconnection of at least some of the various components / circuits of the control system 150. In some embodiments, two or more components of the control system 150 may be electrically and / or communicatively coupled to each other.

[0075] One or more I / O components / devices 202 may include various components to receive input and / or provide output, such as to interact with a user to facilitate the execution of medical procedures. One or more I / O components 202 may be configured to receive touch, voice, gesture, or any other type of input. In various examples, one or more I / O components 202 may be used to provide input regarding the control of a device / system, such as to control the robotic system 110, a navigation endoscope or other medical instruments attached to and / or deployed via the endoscope, control panel 170, control of a fluoroscopy device, etc. For example, a physician 160 (not shown) may provide input via I / O component 202, and in response, the control system 150 may send control signals to the robotic system 110 to manipulate the medical instruments. In various examples, the physician 160 may use the same I / O device to control multiple medical instruments (e.g., switching control between instruments).

[0076] As shown in the figure, one or more I / O components 202 may include one or more displays 152 (sometimes referred to as "one or more display devices 152") configured to display data. The one or more displays 152 may include one or more liquid crystal displays (LCDs), light-emitting diode (LED) displays, organic LED displays, plasma displays, electronic paper displays, and / or any other type of technology. In some embodiments, the one or more displays 152 may include one or more touchscreens configured to receive input and / or display data. Furthermore, one or more I / O components 202 may include one or more I / O devices / controls 210, which may include: touchpads, controllers (e.g., handheld controllers, video game controllers, haptic interfaces, touch-sensitive interfaces, finger-type controls / finger clutches enabling finger-like movement, etc.), mice, keyboards, wearable devices (e.g., optical head-mounted displays), virtual or augmented reality devices (e.g., head-mounted displays), pedals (e.g., buttons at the user's feet), etc. Additionally, one or more I / O components 202 may include: one or more speakers configured to output sound based on an audio signal; and / or one or more microphones configured to receive sound and generate an audio signal. In some embodiments, one or more I / O components 202 include a console or are implemented as a console.

[0077] In some implementations, one or more I / O components 202 may output procedure-related information. For example, the control system 150 may receive real-time images captured by a endoscopic endoscope and display the real-time images and / or a visual / image representation of the real-time images via a display 152. The display 152 may present an interface, such as any of the interfaces discussed herein, which may include image data from the endoscopic endoscope and / or another medical device. Additionally or alternatively, the control system 150 may receive signals (e.g., analog signals, digital signals, electrical signals, acoustic / sound signals, pneumatic signals, tactile signals, hydraulic signals, etc.) from medical monitors and / or sensors associated with the patient, and the display 152 may present information about the patient's health or environment. Such information may include information displayed via medical monitors, including, for example, heart rate (e.g., ECG, HRV, etc.), blood pressure / blood rate, muscle biosignals (e.g., EMG), body temperature, blood oxygen saturation (e.g., SpO2), CO2, brain waves (e.g., EEG), environmental and / or local or core body temperature, etc.

[0078] One or more communication interfaces 204 may be configured to communicate with one or more devices / sensors / systems. For example, one or more communication interfaces 204 may transmit / receive data wirelessly and / or via a network. Networks according to embodiments of this disclosure may include local area networks (LANs), wide area networks (WANs) (e.g., the Internet), personal area networks (PANs), body area networks (BANs), etc. In some embodiments, one or more communication interfaces 204 may implement wireless technologies such as Bluetooth, Wi-Fi, near field communication (NFC), etc.

[0079] One or more power supply units 206 may be configured to manage and / or provide power for the control system 150 (and / or the robot system 110, in some cases). In some embodiments, one or more power supply units 206 include one or more batteries, such as lithium-based batteries, lead-acid batteries, alkaline batteries, and / or other types of batteries. That is, one or more power supply units 206 may include one or more devices and / or circuits configured to provide power and / or provide power management functionality. Furthermore, in some embodiments, one or more power supply units 206 include a main power connector configured to couple to an alternating current (AC) or direct current (DC) main power supply.

[0080] although Figure 2 Not shown, but control system 150 may include and / or control other components, such as one or more pumps, flow meters, valve controls, and / or fluid inlet components, to provide controlled flushing and / or suction capabilities to medical devices, devices deployable through medical devices, etc. In some embodiments, flushing and suction capabilities may be provided directly to the medical device via a separate cable. Furthermore, control system 150 may include voltage and / or surge protectors designed to provide filtered and / or protected power to another device, such as robotic system 110, thereby avoiding the need for power transformers and / or other auxiliary power components in robotic system 110, resulting in a smaller, more mobile robotic system 110.

[0081] In some embodiments, the control system 150 may include support devices for sensors deployed throughout the medical system 100. For example, the control system 150 may include optoelectronic devices for detecting, receiving, and / or processing data received from optical sensors and / or cameras. Such optoelectronic devices can be used to generate real-time images for display in any number of devices / systems, including within the control system 150. Similarly, the control system 150 may include electronic subsystems for receiving and / or processing signals received from deployed electromagnetic (EM) sensors. In some embodiments, the control system 150 may also be used to house and / or position an EM field generator for detection by EM sensors in or on the medical device.

[0082] Furthermore, in some embodiments, the control system 150 can be coupled to the robotic system 110, the worktable 170, and / or the medical device via one or more cables or connectors (not shown). In some embodiments, support functionality from the control system 150 can be provided via a single cable, thereby simplifying and eliminating clutter in the operating room. In other embodiments, specific functions can be coupled in separate cables and connectors. For example, while power can be provided via a single power cable, support for control, optics, fluid dynamics, and / or navigation can be provided via separate cables.

[0083] The robotic system 110 generally includes an elongated support structure 210 (also referred to as a "column"), a robotic system base 212, and a console 214 located at the top of the column 210. The column 210 may include one or more brackets 216 (also referred to as "arm supports 216") for supporting the deployment of one or more robotic arms 112. The brackets 216 may include individually configurable arm mounts that rotate along a vertical axis to adjust the base of the robotic arm 112 for positioning relative to a patient. The brackets 216 also include a bracket interface 218 that allows the brackets 216 to translate vertically along the column 210. The bracket interface 218 may be connected to the column 210 via slots (such as slot 220) positioned on opposite sides of the column 210 to guide the vertical translation of the brackets 216. The slot 220 may include a vertical translation interface to position and hold the brackets 216 at various vertical heights relative to the base 212. The vertical translation of the bracket 216 allows the robotic system 110 to adjust the reach of the robotic arm 112 to accommodate various table heights, patient sizes, physician preferences, etc. Similarly, individually configurable arm mounts on the bracket 216 allow the robotic arm base 222 of the robotic arm 112 to be configured at various angles. The column 210 may internally include mechanisms (such as gears and / or motors) designed to mechanically translate the bracket 216 in response to control signals generated in response to user input (such as input from I / O devices).

[0084] The base 212 can balance the weight of the column 210, bracket 216, and / or robot arm 112 on a surface such as a floor. Therefore, the base 212 can accommodate heavier components, such as one or more electronic devices, motors, power supplies, etc., as well as components that enable the robot system 110 to move and / or remain stationary. For example, the base 212 may include rolling wheels 224 (also referred to as “casters 224” or “moving components” 224) that allow the robot system 110 to move within a room for procedures. Once in place, the casters 224 can be secured using wheel locks to hold the robot system 110 in place during the procedure. As shown, the robot system 110 also includes a handle 226 to aid in manipulating and / or stabilizing the robot system 110. In this example, the robot system 110 is shown as a movable cart-type system. However, the robot system 110 can also be implemented as a fixed system, integrated into a tabletop, etc.

[0085] A robotic arm 112 typically includes a robotic arm base 222 and an end effector 228, separated by a series of links 230 (also referred to as "arm segments 230") connected by a series of joints 232. Each joint 232 may include an independent actuator, and each actuator may include an independently controllable motor. Each independently controllable joint 232 represents an independent degree of freedom available to the robotic arm 112. For example, each arm 112 may have seven joints, thus providing seven degrees of freedom. However, any number of joints can be implemented with any degree of freedom. In this example, multiple joints can generate multiple degrees of freedom, thus allowing "redundant" degrees of freedom. Redundant degrees of freedom allow the robotic arm 112 to position its corresponding end effector 228 in a specific location, orientation, and / or trajectory in space using different link positions and / or joint angles. In some embodiments, the end effector 228 may be configured to engage and / or control medical devices, apparatuses, objects, etc. The degrees of freedom of movement of arm 112 allow robotic system 110 to locate and / or guide medical devices from desired points in space, and / or allow physicians to move arm 112 to a clinically advantageous location away from the patient to create access while avoiding arm collisions.

[0086] Each robotic arm in robotic arm 112 may include an end effector 228 that can be attached using a mechanism converter interface (MCI). In some embodiments, the IDM may be removed and replaced with a different type of IDM. For example, a first type of IDM may manipulate an endoscope, a second type of IDM may manipulate a catheter, a third type of IDM may hold an EM field generator, and so on. The MCI may include connectors for transmitting pneumatic pressure, electrical, electronic, and / or optical signals from robotic arm 112 to the IDM. The IDM 228 may be configured to manipulate medical devices (e.g., surgical instruments / instruments) using techniques including, for example, direct drive, harmonic drive, gear drive, belt and pulley, magnetic drive, etc. In some embodiments, the IDM 228 may be attached to a corresponding robotic arm in robotic arm 112, wherein robotic arm 112 is configured to insert or withdraw a corresponding coupled medical device into or from a treatment site.

[0087] In some embodiments, the robotic arm 112 may be configured to control the position, orientation, and / or tip joint movement of a medical device attached thereto (e.g., a sheath and / or guide of a endoscopic endoscope). For example, the robotic arm 112 may be configured / configurable to manipulate the endoscopic endoscope using elongated moving members. The elongated moving members may include one or more drawstrings (e.g., pull wires or push wires), cables, fibers, and / or flexible shafts. For illustration, the robotic arm 112 may be configured to actuate multiple drawstrings of the endoscopic endoscope to deflect the tip of the endoscope. The drawstrings may include any suitable or desired material, such as metallic and / or non-metallic materials, such as stainless steel, Kevlar, tungsten, carbon fiber, etc. In some embodiments, the endoscopic endoscope is configured to exhibit nonlinear behavior in response to forces applied by the elongated moving members. The nonlinear behavior may be based on the stiffness and compressibility of the endoscopic endoscope, and the variability in relaxation or stiffness between the different elongated moving members.

[0088] As shown in the figure, console 214 is positioned at the upper end of column 210 of robotic system 110. Console 214 may include display 234 to provide a user interface (e.g., a dual-purpose device such as a touchscreen) for receiving user input and / or providing output, such as providing preoperative data, intraoperative data, information for configuring robotic system 110, etc., to the physician / user. Possible preoperative data may include preoperative planning, navigation, and mapping data derived from preoperative computed tomography (CT) scans, and / or records obtained from preoperative patient interviews. Intraoperative data may include optical information from tools, sensors, and / or coordinate information from sensors, as well as important patient statistics such as respiration, heart rate, and / or pulse. Console 214 may be positioned and tilted to allow the physician to access console 214 from the side of column 214 opposite to arm support 216. From this position, the physician can view console 214, robotic arm 112, and patient while operating console 214 from behind robotic system 110.

[0089] Robot system 110 may include one or more I / O components / devices 236 to receive input and / or provide output, such as for interacting with a user. One or more I / O components 236 may be configured to receive touch, voice, gesture, or any other type of input. In the example, one or more I / O components 236 may be used to provide input regarding control of the device / system, such as for controlling / configuring robot system 110. As shown, one or more I / O components 236 may include one or more displays 234 configured to display data. One or more displays 234 may include one or more liquid crystal displays (LCDs), light-emitting diode (LED) displays, organic LED displays, plasma displays, electronic paper displays, and / or any other type of technology. In some embodiments, one or more displays 234 include one or more touchscreens configured to receive input and / or display data. Furthermore, one or more I / O components 236 may include one or more I / O devices / controls 238, which may include touchpads, controllers, mice, keyboards, wearable devices (e.g., optical head-mounted displays), virtual or augmented reality devices (e.g., head-mounted displays), etc. Additionally, one or more I / O components 236 may include: one or more speakers configured to output sound based on an audio signal; and / or one or more microphones configured to receive sound and generate an audio signal. In some embodiments, one or more I / O components 236 include a console or are implemented as a console 214. Furthermore, one or more I / O components 236 may include one or more buttons that can be physically pressed, such as buttons on the distal end of the robotic arm 112 (which can enable / disable the admittance control mode of the robotic arm 112).

[0090] Various components of the robot system 110 may be electrically and / or communicatively coupled using certain connection circuits / devices / features, which may or may not be part of the control circuitry. For example, connection features may include one or more printed circuit boards configured to facilitate the mounting and / or interconnection of at least some of the various components / circuits of the robot system 110. In some embodiments, two or more components of the robot system 110 may be electrically and / or communicatively coupled to each other.

[0091] One or more communication interfaces 240 may be configured to communicate with one or more devices / sensors / systems. For example, one or more communication interfaces 240 may transmit / receive data wirelessly and / or via a network. Networks according to embodiments of this disclosure may include local area networks (LANs), wide area networks (WANs) (e.g., the Internet), personal area networks (PANs), body area networks (BANs), etc. In some embodiments, one or more communication interfaces 240 may implement wireless technologies such as Bluetooth, Wi-Fi, near field communication (NFC), etc.

[0092] One or more power supply units 242 may be configured to manage and / or provide power for the robot system 110. In some embodiments, the one or more power supply units 242 include one or more batteries, such as lithium-based batteries, lead-acid batteries, alkaline batteries, and / or other types of batteries. That is, the one or more power supply units 242 may include one or more devices and / or circuits configured to provide power and / or provide power management functionality. Furthermore, in some embodiments, the one or more power supply units 242 include a main power connector configured to be coupled to an alternating current (AC) or direct current (DC) main power supply. Additionally, in some embodiments, the one or more power supply units 242 include a connector configured to be coupled to a control system 150 to receive power from the control system 150.

[0093] The robot system 110 may also include one or more actuators / hardware 244 to facilitate the movement of the robot arm 112. Each actuator 244 may include a motor, which may be implemented in a joint within the robot arm 112 or elsewhere, to facilitate the movement of the joints and / or connected arm segments / links. In addition, the robot system 110 may include a variety of other components, such as pneumatic devices, light sources, etc.

[0094] Control system 150 and / or robot system 110 may include control circuitry 246 and / or data storage device / memory 248 configured to perform the functions described herein. For ease of discussion and illustration, control circuitry 246 and data storage device 248 are shown as a block between control system 150 and robot system 110. It should be understood that in many embodiments, control system 150 and robot system 110 may include separate instances of control circuitry 246 and data storage device 248. That is, control system 150 may include its own control circuitry and data storage device (e.g., to implement processing on control system 150), while robot system 110 may include its own control circuitry and data storage device (e.g., to implement processing on robot system 110). In many embodiments, references to control circuitry may refer to robot systems, control systems, or medical systems (such as in...). Figure 1The circuitry in any other component of the medical system 100 shown in the diagram.

[0095] Although control circuit 246 is shown as a component separate from other components of control system 150 / robot system 110, any or all other components of control system 150 / robot system 110 may be at least partially embodied in control circuit 246. For example, control circuit 246 may include various devices (active and / or passive), semiconductor materials and / or regions, layers, areas and / or portions thereof, conductors, leads, vias, connectors, etc., wherein one or more other components of control system 150 / robot system 110 and / or portions thereof may be at least partially formed and / or implemented in or by such circuit components / devices.

[0096] As shown, data storage device 248 may include user interface component 250 configured to facilitate one or more interfaces in a user interface. For example, user interface component 250 may generate user interface data and / or provide user interface data to enable the user interface to be displayed via display 152. The user interface may include an image representation from the perspective of a medical device (e.g., a hysteroscope), indicators for elements associated with the medical device (e.g., a working channel indicator, a joint motion plane indicator, etc.), controls or other interface elements, and / or other information. As the medical device is flipped / changed in orientation, user interface component 250 may adjust the orientation of the image representation, such as by applying rotational adjustments to the image representation within the user interface. User interface component 250 may maintain the orientation of indicators relative to the image representation to indicate the accurate orientation of elements (e.g., a working channel) relative to the image representation. User interface component 250 may use orientation / position information about the medical device to provide the image representation and / or the working channel indicator.

[0097] The data storage device 248 may also include a device drive component 252 configured to drive / control the medical device. For example, the device drive component 248 may be configured to: process input signals from the I / O component 202; process position / orientation data indicating the position / orientation of the medical device; process image data indicating image orientation data representing orientation within the user interface; generate control signals for controlling the robot system 110 and / or medical devices / tools connected to the robot system 110; send control signals to the robot system 110 to control the robot system 110 and / or the medical devices / tools; and so on. In some embodiments, user control of the medical device may be disabled during the flipping of the medical device to prevent unintentional movement of the medical device and / or complications related to the mechanical components of the robot system 110. For example, once the endoscope flipping is initiated, the device drive component 252 may ignore received user input until the endoscope flipping is complete, so that the user cannot make further articulation of the endoscope during the flipping.

[0098] In some implementations, the device drive component 252 can control the medical device to move in a manner relevant to a view provided via a user interface. For example, suppose a physician 160 is driving a laparoscopic endoscope from its perspective (e.g., based on image data from the endoscope and provided via the user interface), and the physician 160 provides input via I / O component 202 to move the endoscope in an upward direction relative to I / O component 202, such as by selecting an upward control on I / O component 202. The device drive component 252 can determine the orientation / position of the endoscope and / or the orientation of the image data / representation displayed within the user interface. On the user interface, the device drive component 252 can use such orientation / position information to move the endoscope in an upward direction. Thus, based on the orientation of the image data / representation displayed within the user interface, the device drive component 252 can configure / update controls for the medical device. In various examples, the control method of the medical device can be reconfigured as the medical device is flipped.

[0099] Furthermore, the data storage device 248 may include a positioning component 254 configured to determine / track the orientation / position of an object / medical device. For example, the positioning component 254 may process input data to generate position / orientation data for a medical device coupled to the robotic system 110. In some cases, the position / orientation data may indicate the amount of flipping of the medical device. The position / orientation data of the object / medical device may indicate the position / orientation of the object / medical device relative to a reference frame. The reference frame may be relative to a patient's anatomy, a known object (e.g., an EM field generator), a coordinate system / space, etc. In some implementations, the position / orientation data may indicate the position and / or orientation of the distal end (and / or, in some cases, the proximal end) of the medical device. For example, the position / orientation data of a trocar may indicate the position and orientation of the distal end of the trocar, including the amount of flipping of the distal end of the trocar. The position and orientation of the object may be referred to as the object's pose. In various examples, the techniques used to determine and / or track the position and / or orientation of the medical device may be referred to as positioning techniques.

[0100] Exemplary input data that can be used to generate position / orientation data for an object / medical device may include: sensor data from sensors associated with the medical device (e.g., EM field sensor data, visual / image data captured by imaging devices / depth sensors / radar devices on the medical device, accelerometer data from accelerometers on the medical device, gyroscope data from gyroscopes on the medical device, fiber-optic shape sensing, satellite-based positioning data from satellite-based sensors (e.g., Global Positioning System (GPS)), etc.); feedback data from the robot arm / component (also referred to as "kinematic data") (e.g., data instructing how the robot arm / component moves / acts); robot command data for the robot arm / component (e.g., control signals sent to robot system 110 / robot arm 112 to control the movement of robot arm 112 / medical device); model data about the patient's anatomy (e.g., models of the internal / external parts of the patient's anatomy); patient position data (e.g., data instructing how the patient is positioned on the table); preoperative data; and so on.

[0101] In some implementations, the positioning component 254 may use electromagnetic tracking to determine the position and / or orientation of an object. For example, the positioning component 254 may use real-time EM tracking to determine the real-time position of a medical device in a coordinate system / space that can be registered to the patient's anatomy, which may be represented by a preoperative model or other model. In EM tracking, an EM sensor (or tracker) including one or more sensor coils may be embedded in one or more locations and / or orientations of the medical device (e.g., endoscope, needle, etc.). The EM sensor may measure changes in the EM field generated by one or more static EM field generators positioned at a known location. The positional information detected by the EM sensor may be stored as EM data. The positioning component 254 may process the EM data to determine the position and / or orientation of an object, such as a medical device. An EM field generator (or transmitter) may be placed near the patient (e.g., within a predetermined distance) to generate a low-intensity magnetic field detectable by the EM sensor. The magnetic field may induce a small current (e.g., below a threshold) in the sensor coil of the EM sensor, which may be analyzed to determine the distance and / or angle between the EM sensor and the EM field generator. These distances and / or orientations can be “registered” to the patient’s anatomy during surgery (e.g., a preoperative model) to determine the geometric transformations that align individual locations in the coordinate system with their locations in the preoperative model of the patient’s anatomy. Once registered, EM sensors (e.g., embedded EM trackers) located at one or more locations of the medical device (e.g., the distal tip of an endoscope, a needle, etc.) can provide real-time indication of the position and / or orientation of the medical device as it traverses the patient’s anatomy.

[0102] Furthermore, in some implementations, the positioning component 254 may perform vision-based techniques to determine the position and / or orientation of an object. For example, the medical device may be equipped with a camera, distance sensor (sometimes called a "depth sensor"), radar device, etc., to provide sensor data in the form of visual / image data. The positioning component 254 may process the visual data to facilitate vision-based position tracking of the medical device. For example, preoperative model data may be used in conjunction with the visual data to enable computer vision-based tracking of the medical device (e.g., an endoscope). In an example, using the preoperative model data, the control system 150 may generate a prospective endoscope image library based on the expected path of travel of the endoscope, where each image is linked to a position within the model. During surgery, the control system 150 may refer to this library to compare real-time images and / or other visual data captured at the endoscope (e.g., a camera at the distal end of the endoscope) with those images and / or other visual data in the image library to aid in positioning.

[0103] Furthermore, other types of vision-based techniques can be performed to determine the position and / or orientation of objects. For example, positioning component 254 can use feature tracking to determine the motion of an image sensor (e.g., a camera or other sensor), and thus the motion of a medical device associated with the image sensor. In some cases, positioning component 254 can identify circular geometries corresponding to anatomical cavities in preoperative model data and track changes in those geometries to determine which anatomical cavity has been selected; and track the relative rotational and / or translational motion of the medical device. The use of topology maps can also enhance vision-based algorithms or techniques. Additionally, positioning component 254 can use optical flow (another computer vision-based technique) to analyze the displacement and / or translation of image pixels in a video sequence within visual data to infer camera movement. Examples of optical flow techniques may include motion detection, object segmentation computation, brightness, motion compensation coding, stereo parallax measurement, etc. By comparing multiple frames over multiple iterations, positioning component 254 can determine the movement and position of the image sensor (and therefore the endoscope).

[0104] Furthermore, in some embodiments, the positioning component 254 can use robot commands and / or kinematic data to determine the position and / or orientation of an object. Robot commands and / or kinematic data can indicate the position / or orientation of a robotic arm generated by joint movement / actuation commands (e.g., pitch, yaw, etc.). For example, robot commands and / or kinematic data can indicate the actuation amount / direction of a joint, gear, pulley, belt, or another actuating element of the robotic arm. The positioning component 254 can use data indicating the position / or orientation of the robotic arm to determine the position / or orientation of a medical device attached to the robotic arm. For example, based on the position / or orientation of the robotic arm attached to a catheter, commands sent to control the catheter, and / or characteristics of the catheter (e.g., catheter length, catheter capacity, etc.), the positioning component 254 can determine / estimate the position / or orientation of the catheter. Additionally, in various examples, the positioning component 254 can use robot command data based on commands to control the medical device, markings on the medical device indicating distance, etc., to determine how far the medical device has been inserted / retracted within the patient's body. In some intraoperative embodiments, the position and / or orientation of the medical device can be estimated using calibrated measurements in conjunction with known insertion depth information. Alternatively or additionally, these calculations may be combined with EM, vision and / or topology modeling for analysis to estimate the location and / or orientation of the medical device.

[0105] Furthermore, in some implementations, the positioning component 254 may use other types of data to determine the position and / or orientation of an object. For example, the positioning component 254 may analyze sensor data from shape-sensing optical fibers embedded in the medical device (e.g., which can provide shape data about the position / shape of the medical device), accelerometers, gyroscopes, satellite-based positioning sensors (e.g., Global Positioning System (GPS)), radio frequency transceivers, etc. Such data can indicate the position and / or orientation of the medical device.

[0106] In some implementations, the positioning component 254 can process preoperative data to determine the location and / or orientation of an object. Preoperative data (sometimes referred to as “mapped data”) can be generated by performing computed tomography (CT) scans, such as low-dose CT scans. Preoperative CT images from the scans can be reconstructed into three-dimensional images, which are then visualized (e.g., “slices” of sectional views of the patient’s internal anatomy). When aggregated analysis is performed, image-based models of the anatomical cavities, spaces, and / or structures of the patient’s anatomy (such as the patient’s lung network, kidney anatomy, etc.) can be generated. Centerline geometry can be determined and / or estimated from the CT images to form a two-dimensional or three-dimensional volume of the patient’s anatomy, referred to as model data (also called “preoperative model data” when generated using preoperative CT scans). Network topology models can also be derived from CT images.

[0107] In some implementations, the positioning component 254 may use the input data in combination. For example, the positioning component 254 may use a probabilistic method, wherein confidence weights are assigned to the position / orientation determined by multiple forms of input data. To illustrate, if the EM data is unreliable (in cases where EM interference may exist), the EM data may be associated with relatively low confidence values ​​and may rely on other forms of input data, such as visual data, robot commands, and kinematic data.

[0108] Although many implementations are discussed in the context of the user interface component 250, the instrument drive component 252 and / or the positioning component 254 being implemented as (or including) one or more instructions executable by the control circuitry 246, any of these components may be implemented at least partially as control circuitry.

[0109] The term "control circuitry" can refer to one or more processors, processing circuitry, processing modules / units, chips, dies (e.g., semiconductor dies that include one or more active devices and / or passive devices and / or connection circuitry), microprocessors, microcontrollers, digital signal processors, microcomputers, central processing units, graphics processing units, field-programmable gate arrays, application-specific integrated circuits (ASICs), programmable logic devices, state machines (e.g., hardware state machines), logic circuits, analog circuits, digital circuits, and / or any means of manipulating signals (analog and / or digital) based on hard-coded circuit and / or operating instructions. Control circuitry can also include one or more storage devices, which can be embodied in a single memory device, multiple memory devices, and / or embedded circuitry. Such data storage devices can include read-only memory, random access memory, volatile memory, non-volatile memory, static memory, dynamic memory, flash memory, cache memory, data storage registers, and / or any means of storing digital information. It should be noted that in implementations where the control circuit includes a hardware state machine (and / or implements a software state machine), analog circuits, digital circuits, and / or logic circuits, a data storage device / register storing any associated operation instructions may be embedded within or outside the circuit that includes the state machine, analog circuits, digital circuits, and / or logic circuits.

[0110] The term "memory" can refer to any suitable or desired type of computer-readable medium. For example, a computer-readable medium may include one or more volatile data storage devices, non-volatile data storage devices, removable data storage devices, and / or non-removable data storage devices, implemented using any technology, layout, and / or data structure / protocol, including any suitable or desired computer-readable instructions, data structures, program modules, or other types of data.

[0111] Computer-readable media that can be implemented according to embodiments of this disclosure include, but are not limited to, phase-change memory, static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, optical disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage devices, cassette tape, magnetic tape, disk storage devices or other magnetic storage devices, or any other non-transitory medium that can be used to store information for access by a computing device. As used in certain contexts herein, computer-readable media generally may not include communication media such as modulated data signals and carrier waves. Therefore, computer-readable media should generally be understood as non-transitory media.

[0112] Exemplary peephole Figure 3 An example of a speculum 130 disposed in various parts of a patient's urinary system according to one or more embodiments of the present disclosure is shown. In this example, the speculum 130 is implemented as an endoscope / ureteroscope. As mentioned above, ureteroscopic procedures can be performed to examine and / or treat abnormalities of the human ureter. For example, ureteroscopic procedures can be performed to treat and / or remove kidney stones. Such procedures can be performed at least partially manually and / or at least partially using robotic techniques (such as... Figure 1 The robotic system 110 shown is used to perform this procedure. For example, using robotic devices and / or systems for a particular endoscopic procedure can provide relatively better accuracy, control, and / or coordination compared to a completely manual procedure.

[0113] The endoscope 130 may include a rigid or flexible tube, and / or its size may be configured to pass through an external sheath, catheter, guide, or other endoscopic device. However, in some cases, the endoscope 130 may be used without such a device. For example, the endoscope 130 may be configured to pass through an access sheath 136 to access a target anatomical structure. When the object / stone is not too large (e.g., within a threshold size), the diameter of the access sheath 136 may be sufficient to allow the endoscope 130 to pass through it along with the object captured by an instrument (e.g., a basket device). In some examples, the access sheath 136 may be advanced through the ureter 197 to a location near the renal pelvis 302 and / or the ureteropelvic junction 304. The distal end of the access sheath 136 may be positioned at a location within the ureter 197 and / or the renal pelvis 302, wherein such positioning may be at least partially dependent on the anatomical structure. However, in other examples, the access sheath 136 may be located only within the urethra or other anatomical structures, rather than near the renal pelvis 302. Typically, the articulation of the access sheath 136 may not be sufficient to allow articulation of the endoscope 130, and therefore navigating / driving the access sheath 136 into the kidney may not be practical.

[0114] The endoscope 130 may be articulated, for example, with at least its distal portion capable of articulation relative to the endoscope, allowing the endoscope 130 to be steered within the human anatomy. In some embodiments, the endoscope 130 is configured to articulate in, for example, five degrees of freedom (DOF), including XYZ coordinate movement as well as pitch and yaw. Furthermore, in some embodiments, the endoscope 130 may articulate in six DOFs, including XYZ coordinate movement as well as pitch, yaw, and roll. In other embodiments, the endoscope 130 may articulate in other DOFs. In embodiments where the endoscope 130 is equipped with a position sensor, the position sensor may provide position information such as 5-DOF position information (e.g., x, y, and z coordinates, and pitch and yaw angles), 6-DOF position information (e.g., x, y, and z coordinates, and pitch, yaw, and roll angles), etc. In some embodiments, the endoscope 130 may include telescopic components, such as an internal guide portion and an external sheath portion, which can be manipulated to telescopically extend the endoscope 130.

[0115] The endoscope 130 may include one or more elongated moving members (not shown) configured to control movement of the endoscope 130 (such as the distal end of the endoscope 130). The elongated moving member may include one or more wires (e.g., pull wires or push wires), cables, optical fibers, and / or flexible shafts. The pull wire may include any suitable or desired material, such as metallic and non-metallic materials, such as stainless steel, Kevlar, tungsten, carbon fiber, etc. In some embodiments, the endoscope 130 is configured to exhibit nonlinear behavior in response to forces applied by the elongated moving members. This nonlinear behavior may be based on the stiffness and / or compressibility of the endoscope 130, and the variability in relaxation or stiffness between the different elongated moving members. For example, a robotic arm may be configured to actuate one or more pull wires coupled to the endoscope 130 to deflect the tip 306 of the endoscope 130. Conversely, in a user-handheld implementation, the user can provide manual input via an actuator to actuate one or more pull wires of the endoscope 130 to deflect the tip 306 of the endoscope 130.

[0116] The endoscope 130 may include an imaging device 308 configured to capture image data, such as image data representing internal anatomical structures of a patient. The imaging device 308 may include a camera, such as an optical camera, and / or another imaging device. The imaging device 308 may include optical fibers, fiber arrays, and / or lenses. One or more optical components of the imaging device 308 may move with the tip 306 of the endoscope 130 such that movement of the tip 306 causes a change in the image captured by the imaging device 308. In some embodiments, the endoscope 130 may house wires and / or optical fibers to transmit signals to / from the optical components and the distal end 306 of the endoscope 130. Furthermore, the endoscope 130 may be configured to house optical fibers to carry light from a proximal light source, such as a light-emitting diode, to the distal end 306 of the endoscope. The distal end 306 of the endoscope 130 may include a port for illuminating the anatomical space by the light source 310, which may be useful when using the imaging device 308. Although multiple light sources 310 are shown, the peephole 130 can be implemented with any number of light sources.

[0117] The endoscope 130 may also include a working channel 312 for deploying instruments 314. Exemplary instruments 314 include a laser device 314a configured to provide a laser, a basket device 314b configured to capture / retrieve an object (e.g., fragments of a kidney stone), forceps 314c configured to grasp / seize an object, a scalpel 314d configured to cut an object, a lithotripter (not shown), a flushing / aspiration device (not shown) configured to provide flushing / aspiration to a target site, etc. The working channel 312 may extend longitudinally from the proximal end of the endoscope 130 through the endoscope 130 to the distal end 306. In various examples, the working channel 312 is offset to one side of the endoscope 130 (e.g., offset from the longitudinal axis 316 of the endoscope 130), such as... Figure 3 As shown. In other examples, the working channel 312 is positioned at the center of the endoscope 130 or at another location. Although the imaging device 308 is shown attached to the distal end 306 of the endoscope 130 (e.g., integrated with the endoscope 130), in some cases, the imaging device 308 is a separate device deployed via the working channel 312. Furthermore, although a single working channel 312 is shown, any number of working channels can be implemented.

[0118] In some embodiments, the peephole 130 includes a sensor (sometimes referred to as a “position sensor”) configured to generate sensor data and / or transmit the sensor data to another device. The sensor data (sometimes referred to as “sensor position data”) may indicate the position and / or orientation of the peephole 130 (e.g., its distal end 306), and / or may be used to determine / infer the position / or orientation of the peephole 130. For example, the sensor may provide sensor data to a control system, which is then used to determine the position and / or orientation of the peephole 130. The sensor may be located on the distal end 306 of the peephole 130 and / or at another location. In some embodiments, the sensor may include another form / implementation of an electromagnetic (EM) sensor or antenna with a coil of conductive material. However, the peephole 130 may include other types of sensors, such as shape-sensing fiber optics, accelerometers, gyroscopes, satellite-based positioning sensors (e.g., Global Positioning System (GPS) sensors), radio frequency transceivers, etc.

[0119] In some implementations, the endoscope 130 may be configured to move by different amounts in different planes, such that the endoscope, when oriented in a particular manner, moves with more / fewer degrees of freedom in a particular direction. For example, the endoscope 130 may be associated with a primary joint motion plane and a secondary joint motion plane, wherein the primary plane and the secondary plane are perpendicular to each other. Compared to the secondary joint motion plane, the tip 306 of the endoscope 130 may be configured to move with greater degrees of freedom in the primary joint motion plane (e.g., exhibiting a greater degree of skew).

[0120] The endoscope 130 and / or instrument 314 can be controlled in any suitable or desired manner, based on manual manipulation of the handle component, electronic user input, or automatic control. For example, Figure 318 illustrates an exemplary robotic control configuration for controlling the endoscope 130 and / or instrument 314 using one or more robotic arms. Here, the endoscope driver instrument coupler 132 and / or handle 134 can each be coupled to a robotic arm that can manipulate the endoscope driver instrument coupler 132 and / or handle 134 to control the endoscope 130 / instrument 314. Meanwhile, Figure 320 illustrates an exemplary manual control configuration in which the endoscope 130 includes a handheld handle 322 configured for a user to hold / manipulate in order to move the endoscope 130 / instrument 314.

[0121] Example image / indicator adjustment Figures 4-1 to 4-9 An example is shown of a peephole that flips according to one or more embodiments and applies rotational adjustments to the peephole image within the user interface once the peephole has finished flipping (sometimes referred to as a "phased approach"). Meanwhile, Figures 5-1 to 5-5Examples of flipping a laparoscopy according to one or more embodiments and applying rotational adjustments to the laparoscopy image within a user interface while the laparoscopy is flipped (sometimes referred to as a "continuous method") are shown. In these examples, the laparoscopy is flipped by a predetermined amount to rotate the working channel of the laparoscopy by the same amount, which can provide better orientation of the laparoscopy for treating the target site. Specifically, the laparoscopy is flipped 180 degrees to reorient the working channel. However, any amount of flipping can be implemented, which can be a predetermined amount or other amounts. Laparoscopy flipping can be initiated in various ways, such as by receiving user input via I / O devices / components (via a user interface or otherwise), automatically determining when to flip the laparoscopy, etc.

[0122] In these examples, user interface 402 / 502 provides information to assist the user in controlling endoscope 404 / 504 and / or instruments deployed through endoscope 404 / 504, such as to perform medical procedures. User interface 402 / 502 may provide an image representation 406 / 506 to represent a view from the perspective of endoscope 404 / 504 and a work channel indicator 408 / 508 indicating the rotation angle of the work channel of endoscope 404 / 504 relative to the image representation 406 / 506. When work channel rotation is implemented, the work channel indicator 408 / 508 may be constantly and / or displayed in user interface 402 / 502 at another time. Image representation 406 / 506 may include image data captured by an imaging device associated with endoscope 404 / 504, based on, and / or generated from, the image data. Here, images 406 / 506 depict kidney stones 410 / 510 and the adjacent cavity 412 / 512, which is typically visible during the procedure for removing kidney stones 410 / 510 (although kidney stones 410 / 510 may be otherwise oriented). As shown, user interface 402 / 502 may include user interface element 414 / 514 to flip the working channel of endoscope 404 / 504 (i.e., endoscope 404 / 504 is flipped). Meanwhile, images 416 / 516 and element 418 / 518 in each figure depict the orientation of endoscope 404 / 504 within the anatomy of the patient for the corresponding figure. Element 418 / 518 indicates the orientation of the tip and coordinate system of endoscope 404 / 504 for the corresponding image 416 / 516.

[0123] Generally, the orientation of the image representation within the user interface can be related to (with or without offset) the orientation of the endoscope tip / imaging device. In other words, the user reference frame (also known as the "user view") via the user interface is related to the orientation of the endoscope tip / imaging device. Thus, when the endoscope 404 is flipped, the image representation typically rotates within the user interface unless a rotation adjustment is applied. To illustrate, in Figures 4-1 to 4-9In the context, you can press Figure 4-1 The coordinate system of the peephole 404 is initially oriented as shown at point 418, where the upward direction relative to the coordinate system of the peephole 404 (e.g., the direction the arrow in 418 points) corresponds to... Figure 4-1 The left-hand direction. Here, the coordinate system is the same for the imaging device of the peephole 404; although in some cases it can be offset. In this example, the user interface 402 provides a coordinate system with... Figure 4-1 The image representation 406 associated with the offset peephole 404 is such that the upward direction in the user interface 402 corresponds to the rightward direction in the coordinate system of the peephole 404. This offset can be maintained until rotation adjustment is applied.

[0124] Furthermore, the control of the peephole is often associated with the user interface, allowing the peephole to be controlled intuitively by the user. For example, suppose the user provides input via an I / O component to move the peephole in the upward direction relative to the I / O component, such as by selecting an upward control on the I / O component. Information about the orientation / position of the peephole and / or the orientation represented by an image within the user interface can be used to move the peephole in the upward direction on the user interface, even if such a direction is not the upward direction relative to the peephole's coordinate system. In this way, the user can view the peephole moving in the upward direction on the user interface.

[0125] As mentioned above, Figures 4-1 to 4-9 An example is shown where rotational adjustment is applied to image representation 406 after the peephole 404 has been flipped. In these figures, the arrow next to image 416 indicates the flipping direction of the peephole 404, while the arrow next to user interface 402 indicates the rotational direction of image representation 406 within user interface 402. For example, in Figures 4-6 to 4-9 In the image, the arrow next to the user interface 402 indicates that the image representation 406 is rotated counterclockwise within the user interface 402. This is due to rotation adjustments made to adjust / correct the orientation of the image representation 406.

[0126] As shown in the figure Figure 4-1 An image representation 402 and a working channel indicator 404 within the user interface 402 are shown when the peephole 404 is oriented in its initial position. Here, the peephole 404 is oriented as shown at 418. In this example, the user flips the peephole 404 a predetermined amount by selecting interface element 414. Figures 4-2 to 4-5 As shown, the viewing mirror 404 is rotated 180 degrees counterclockwise. However, the viewing mirror 404 can be rotated clockwise to achieve the same amount of rotation. Figure 4-5The image shows the user interface 402 and the orientation of the peephole 404 once it has been flipped. As the peephole 404 flips, the contents within the image representation 406 rotate clockwise within the user interface 402, as well as... Figures 4-2 to 4-5 As shown. That is, due to the flipping of the endoscope 404 (and the associated imaging device of the endoscope 404), the virtual horizontal line associated with the image representation 406 can rotate clockwise. The virtual horizontal line may represent a reference line for contents in the image representation 406 (e.g., a reference horizontal line for patient anatomy). Figures 4-1 to 4-9 In the context of the virtual horizon, the virtual horizon can initially be Figure 4-1 The horizontal line in the middle.

[0127] As the viewing mirror 404 flips, the image representation 406 can be displayed in a cropped format. For example, as... Figures 4-2 to 4-5 As shown, within user interface 402, image representation 406 can be displayed in a circular form. This avoids black bars appearing in user interface 402 due to the loss of image data from the imaging device. The cropped form also indicates to the user that the peephole 404 is flipping. Although many examples have been discussed in the context of presenting image representation 406 in a cropped (e.g., circular) manner during the flipping of peephole 404, image representation 406 can be displayed in an uncropped manner and / or in other forms during the flipping, such as the initial / original form of image representation 406 (based on the field of view of the imaging device), another shape, etc.

[0128] As the viewing mirror 404 flips, the working channel indicator 408 can maintain the rotation angle of the working channel relative to the image representation 406. Specifically, as Figures 4-2 to 4-5 As shown, as the virtual horizontal line within image representation 406 rotates, the working channel indicator 408 remains stationary within user interface 402 to accurately indicate the position of the working channel relative to the contents of image representation 406. That is, as the peephole 404 flips 180 degrees from its initial orientation, the working channel indicator 408 can maintain the same rotation angle within user interface 402. In this example, the working channel indicator 408 is presented overlaid on image representation 406 as a partially transparent interface element. This allows image representation 406 to be viewed at least partially through the working channel indicator 408. However, the working channel indicator 408 can be displayed in other ways, such as in a non-overlay manner relative to 406, as a non-transparent interface element (e.g., a stereoscopic graphic), a fully transparent interface element, and / or in other forms / methods.

[0129] Once the endoscope 404 has been flipped and the working channel has thus rotated 180 degrees, rotational adjustments can be applied to adjust / correct the orientation of the image representation 406. In various examples, it can be determined that the endoscope 404 has been flipped based on one or more of the techniques discussed herein (e.g., positioning techniques). Figures 4-6 to 4-9 As shown, rotation adjustment may include: within the user interface 402, rotating the image representation 406 counterclockwise until the virtual horizontal line associated with the image representation 406 is restored. Figure 4-1 Until the initial orientation is shown (e.g., the contents within image representation 406 are restored to the initial orientation). Image representation 406 can be rotated by the same amount as the peephole 404 is flipped (e.g., 180 degrees). While the peephole 404 remains in the same orientation (e.g., flipped / turned position), image representation 406 can be rotated within user interface 402. Figure 4-9 The user interface 402 (and working channel indicator 408) is shown when the image representation 406 has been rotated 180 degrees to restore the virtual horizon to its initial orientation. Although rotation adjustment is discussed in the context of rotating the image representation 406 counterclockwise, rotation adjustment can alternatively be achieved by rotating the image representation 406 clockwise.

[0130] While rotating image representation 406, image representation 406 can be displayed in a cropped form, in a manner similar to that discussed when flipping the peephole 404. Once image representation 406 has been rotated and the working channel 408 has been rotated, as... Figure 4-9 As shown, image representation 406 can be restored to its uncropped form. However, during and / or after the peephole is flipped, image representation 406 can be displayed in other forms. In this way, rotational adjustments can provide the user with the same frame of reference (e.g., the same virtual horizon) before and after flipping the peephole 404, while allowing the user to accurately see the position of the working channel (e.g., the working channel has been flipped).

[0131] Also Figures 4-6 to 4-9 As shown, during rotational adjustment of the image display 406, the working channel indicator 408 can be rotated counterclockwise within the user interface 402 to maintain an accurate representation of the angle of the working channel relative to the image display 406. The working channel indicator 408 can be rotated by the same amount as the peephole 404 is flipped (e.g., 180 degrees).

[0132] In some implementations, this is achieved in the following situations Figures 4-6 to 4-9The rotational adjustment techniques discussed in the context may be difficult to accurately track the orientation of the tip of the peephole 404 during flipping. For example, flipping the peephole 404 may include: disengaging / engaging mechanical components of the peephole 404 / robot arm; assessing the tightness of the elongated shaft of the peephole 404; sending one or more control signals to components of the robot system; and so on. Such operations may take an unknown amount of time and / or be associated with certain delays. Furthermore, in the examples where operations are performed at the proximal end of the peephole 404, such operations may not instantaneously translate into flipping at the distal end of the peephole 404. Additionally, in some examples, the peephole 404 may flip when no EM field generator is used, causing the control system to rely on data other than EM sensor data to track the flipping of the peephole 404, which may be less accurate / reliable.

[0133] Although discussed in the context of the rotation of image representation 406 when the peephole 404 has completed its flip. Figures 4-1 to 4-9 However, image representation 406 can rotate after other events occur. For example, image representation 406 can rotate in an incremental manner, wherein image representation 406 rotates a predetermined amount (e.g., 15 degrees for every 15-degree flip completed by the peephole 404) whenever the peephole 404 completes such a rotation.

[0134] at the same time, Figures 5-1 to 5-5 An example of applying rotational adjustments in a continuous manner while the peephole 504 is flipped is illustrated. In these figures, the arrow next to image 516 indicates the flipping direction of the peephole 504. Here, as the peephole 504 flips from its initial orientation to its final orientation at 180 degrees, the image representation 506 is continuously updated within the user interface 502. As shown, while the working channel indicator 508 rotates around the image representation 506, the virtual horizontal line of the image representation 506 remains the same throughout the flipping of the peephole 504. Specifically, the peephole 504 flips counterclockwise relative to its coordinate system, and the working channel indicator 508 rotates in the same direction (i.e., counterclockwise) relative to the user interface 502. Furthermore, the image representation 506 can rotate counterclockwise in an imperceptible manner. To make such adjustments to the working channel indicator 508 and the image representation 506, one or more of the positioning techniques discussed herein can be used to track the orientation of the peephole 504 in real time. This allows the user interface 502 to present information in an accurate manner. As discussed similarly above, while the peephole 504 flips, the image representation 506 can be displayed in a cropped format, such as... Figures 5-2 to 5-4 As shown. Figure 5-5As shown, when the endoscope 504 has been flipped, the image representation 506 can be restored to its uncropped form. By achieving this rotational adjustment in a continuous manner, the user can perceive the image representation 506 as remaining the same, which also helps prevent the user from losing their sense of orientation within the patient's anatomy. That is, in Figures 5-1 to 5-5 In this context, the orientation of image representation 506 can be updated in a continuous manner to maintain a virtual horizontal line for the patient's anatomy (e.g., to maintain the user's frame of reference).

[0135] Despite Figures 4-1 to 4-9 as well as Figures 5-1 to 5-5 The image representations depict various cropped or uncropped forms during specific phases of the peephole flip, but may be displayed in cropped or uncropped form at any time before, during, and / or after the peephole flip.

[0136] Exemplary User Interface Figure 6 An exemplary user interface 602, according to one or more embodiments, is shown for assisting a user in controlling a endoscopic endoscope and / or an instrument deployed through the working channel of the endoscope. The user interface 602 may provide: an image representation 604 representing a view from the perspective of the endoscope; and a working channel indicator 606 indicating the rotation angle of the working channel of the endoscope relative to the image representation 604. The image representation 604 may include image data captured by an imaging device associated with the endoscope, based on the image data, and / or generated from the image data. Here, the image representation 604 depicts a kidney stone 608 and a cavity 610 adjacent to the kidney stone 608. In this example, the user interface 602 is presented during a medical procedure. However, the user interface 602 may be presented at other times.

[0137] As shown, the user interface 602 may include a working channel element 612 to rotate the working channel of the peephole. The working channel indicator 606 may be implemented as a semi-transparent, semi-circular element superimposed on the image representation 604; however, the working channel element 612 may take other forms and / or be positioned at other locations within the user interface 602. In one example, the user may select the working channel element 612 to flip the peephole 180 degrees (e.g., from an initial orientation to a 180-degree rotation orientation); however, rotation is also possible. In another example, the user may provide a first input via an I / O component (e.g., the user interface 602, a controller, etc.) to initiate the peephole flip; and provide a second input to stop the peephole flip. In yet another example, by selecting and holding a control element on an I / O component, the user can provide input to flip the peephole, causing the peephole to flip simultaneously with the selection of the control element. In any case, one or more of the rotation adjustment techniques discussed herein may be implemented to adjust the image representation 604 during / after the peephole flip. Additionally, the working channel indicator 606 can be adjusted to indicate the orientation of the endoscope's working channel. The user interface 602 may also include a horizontal flipping element 614 to flip the endoscope without applying rotational adjustments to the image representation 604 in the user interface 602 (e.g., without flipping to correct the endoscope image). This can be used to align the visual horizon with the principal plane of the kidney or other features. It can also be used to align the endoscope's primary / major articulation planes, making it easier for the endoscope to reach / navigate to a particular location.

[0138] Additionally, the user interface 602 may include joint motion bars 616, 618 surrounding the image representation 604 to view the amount of joint motion associated with the selected instrument (e.g., a hysteroscope or tool in this example). The top / bottom joint motion bar 616 may indicate the amount of vertical joint motion. For example, the top / bottom joint motion bar 616 may be positioned above or below the image 604 and / or expand / contract in length to indicate the vertical joint motion of the instrument. Figure 6 In the example, the top / bottom joint motion bar 616 is positioned below the image representation 604 to indicate upward joint movement of the endoscope. The right / left joint motion bar 618 can indicate the amount of horizontal joint movement. For example, the right / left joint motion bar 618 can be positioned to the right or left of the image representation 604 and / or expand / contract in length to indicate horizontal joint movement of the instrument. Figure 6 In the example, the right / left joint motion bar 618 is positioned to the left of the image representation 604 to indicate that the endoscope is making joint motion to the right.

[0139] User interface 602 may also include interface elements 620, 622 for selecting an instrument to be controlled. Interface element 620 enables control of the endoscope, and interface element 622 enables control of the instrument (also referred to as a "tool") associated with the endoscope. The instrument associated with the endoscope is deployed through the endoscope's working channel. In this example, the endoscope is selected, which allows the user to navigate the endoscope and for user interface 602 to display information about the endoscope.

[0140] Exemplary flowchart Figure 7 An exemplary flowchart of process 700 for applying rotational adjustments to a peephole view within a user interface, according to one or more embodiments, is shown. Various operations / actions associated with process 700 may be performed by control circuitry implemented in any or a combination of the devices / systems discussed herein, such as control system 150, robot system 110, tabletop 170, medical device, and / or another device. Although various boxes are shown as part of process 700, any such box may be removed. Furthermore, additional boxes may be implemented as part of process 700. The order in which these boxes are shown is provided for illustrative purposes only, and these boxes may be implemented in any order. In some embodiments, one or more boxes in the various frames of process 700 are implemented as executable instructions that, when executed by control circuitry, cause control circuitry to perform the discussed functionality / operation. However, one or more boxes in the various frames of process 700 may be implemented in other ways, such as by other devices / systems, users, etc.

[0141] At box 702, process 700 may include receiving image data from an imaging device associated with the endoscope. The endoscope may include a working channel for deploying medical devices / tools. The working channel may be offset from the longitudinal axis of the endoscope. In some cases, the imaging device is mounted on the end of the endoscope, while in others, the imaging device is deployed through the working channel of the endoscope. In various examples, image data may be received in real time, such as receiving multiple images from the imaging device.

[0142] At box 704, process 700 may include: generating an image representation based at least in part on image data. For example, the image representation may include / represent image data from a peephole. In some cases, the image data is reformatted / converted to generate the image representation.

[0143] At box 706, process 700 may include: displaying an image representation in a user interface. In one example, displaying an image representation may include: sending data to a display such that the display can render the image representation via the user interface. In another example, displaying an image representation may include: displaying / rendering the image representation via a display.

[0144] At box 708, process 700 may include: displaying a working channel indicator. The working channel indicator may indicate the rotation angle of a working channel relative to the image representation. In one example, displaying the working channel indicator may include: sending data to a display such that the display can render the image representation via a user interface. In another example, displaying the image representation may include: displaying / presenting the working channel indicator via the display. In some cases, the working channel indicator is superimposed on the image representation and / or positioned at the perimeter of the image representation.

[0145] At block 710, process 700 may include receiving an input control signal from an input device to rotate the working channel. For example, a user may use the input device to provide input to rotate the endoscope, and the input device may generate / send an input signal instructing the rotation of the endoscope. In some cases, the input is to rotate the endoscope / working channel by a predetermined amount.

[0146] At block 712, process 700 may include controlling the rotation of the endoscope, at least in part, based on an input control signal. For example, a control signal may be sent to a robot system / robot arm to control the robot arm to rotate (potentially in a specific rotational direction) the endoscope. In some cases, the endoscope may be rotated by a predetermined amount, such as flipping the working channel from one orientation to another.

[0147] At box 714, process 700 may include: determining that the peephole has completed its flip. Such determination may be based on feedback data from a robotic arm configured to be coupled to the peephole, sensor data generated by a position sensor associated with the peephole, and / or control signals generated to control the robotic arm. In some cases, the flipping of the peephole is tracked over time.

[0148] At box 718, process 700 may include determining the amount by which the peephole has been flipped. Such determination may be based on feedback data from a robotic arm configured to be coupled to the peephole, sensor data generated by a position sensor associated with the peephole, and / or control signals generated to control the robotic arm. Although boxes 714 and 716 are shown in process 700, in some cases, one or more of these boxes (and / or other boxes in process 700) are not implemented.

[0149] At box 718, process 700 may include rotating a work channel indicator within the user interface. For example, based at least in part on the amount by which the peephole has been flipped, the work channel indicator may be rotated from a first orientation relative to the user interface to a second orientation relative to the user interface. The first orientation may indicate a first rotation angle of the work channel, and the second orientation may indicate a second rotation angle of the work channel.

[0150] At box 720, process 700 may include applying rotational adjustments to an image representation in the user interface. For example, the image representation may be rotated within the user interface, at least in part, based on the amount by which the peephole has been flipped. In some cases, the image representation may be rotated within the user interface in response to determining that the peephole has been flipped. In other cases, the image representation may rotate as the peephole flips, such as rotating continuously based on the current orientation of the peephole. Here, the image representation may rotate without being perceived by the user through the user interface. In various examples, the image representation may be displayed in a cropped form while the peephole flips and / or while the image representation rotates. Furthermore, in various examples, within the user interface, the image representation may rotate in the same direction of rotation (e.g., clockwise or counterclockwise) as when the peephole flips relative to a reference system at the peephole tip (e.g., the peephole's coordinate system).

[0151] Additional Implementation Plan Depending on the implementation, specific actions, events, or functions of any of the processes or algorithms described herein may be performed in different orders, added, combined, or completely ignored. Therefore, in a particular implementation, not all described actions or events are necessary for the practice of the process.

[0152] Unless otherwise specifically stated or otherwise understood in the context in which they are used, the conditional language used herein, such as “may,” “can,” “possibly,” “can,” “e.g.,” etc., refers in its ordinary sense and is generally intended to convey that a particular embodiment includes a particular feature, element, and / or step that is not included in other embodiments. Therefore, such conditional language is not generally intended to imply that a feature, element, and / or step is necessary in any way for one or more embodiments, or to imply that one or more embodiments must, with or without input or prompting, include logic for determining whether such features, elements, and / or steps are included in any particular embodiment or whether they will be performed in any particular embodiment. The terms “comprising,” “including,” “having,” etc., are synonymous and used in their ordinary sense, and are used inclusively in an open-ended manner, without excluding additional elements, features, actions, operations, etc. Moreover, the term “or” is used in its inclusive sense (rather than in its exclusive sense) such that when used, for example, to connect a series of elements, the term “or” refers to one, some, or all of the elements in that series. Unless otherwise specified, conjunctions such as “at least one of X, Y, and Z” are generally understood to convey that items, terms, elements, etc., may be X, Y, or Z. Therefore, such conjunctions are generally not intended to imply that certain implementations require the existence of at least one of X, at least one of Y, and at least one of Z.

[0153] It should be understood that in the above description of the embodiments, various features are sometimes grouped together in a single embodiment, figure, or description therein in order to simplify this disclosure and aid in understanding one or more of the various aspects. However, this approach of the disclosure should not be construed as reflecting an intention that any claim requires more features than those expressly recited in that claim. Furthermore, any component, feature, or step illustrated and / or described in the specific embodiments herein may be applied to or used with any other embodiment. Moreover, for each embodiment, no component, feature, step, or group of components, features, or steps is necessary or indispensable. Therefore, it is expected that the scope of this disclosure, as disclosed herein and claimed below, is not limited to the specific embodiments described above, but should be determined by a fair reading of the appended claims.

[0154] It should be understood that specific ordinal terms (e.g., "first" or "second") may be provided for ease of reference and do not necessarily imply physical characteristics or ordering. Therefore, as used herein, ordinal terms used to modify elements such as structures, components, operations (e.g., "first," "second," "third," etc.) do not necessarily indicate the priority or order of that element relative to any other element, but rather serve to generally distinguish that element from another element with a similar or identical name (but used in ordinal terms). Additionally, as used herein, indefinite articles ("a (a)" and "an (an)") may indicate "one or more" rather than "one." Furthermore, an operation performed "based on" a condition or event may also be performed based on one or more other conditions or events not explicitly stated.

[0155] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which the exemplary embodiments pertain. It will be further understood that terms, such as those defined in common dictionaries, shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field and shall not be interpreted in an idealized or overly formal sense, unless expressly defined herein.

[0156] For ease of description, the spatial relative terms “external,” “internal,” “upper,” “lower,” “below,” “above,” “vertical,” “horizontal,” and similar terms are used herein to describe the relationship between one element or component illustrated in the accompanying drawings and another. It should be understood that the spatial relative terms are intended to cover different orientations of the device in use or operation other than those depicted in the drawings. For example, if the device shown in the drawings is flipped, a device located “below” or “under” another device may be positioned “above” another device. Therefore, the descriptive term “below” can include both a lower position and an upper position. The device may also be oriented in another direction, and therefore the spatial relative terms may be interpreted differently depending on the orientation.

[0157] Unless otherwise explicitly stated, comparative and / or quantitative terms, such as “less,” “more,” “greater,” etc., are intended to encompass the concept of equality. For example, “less” may mean not only “less” in the strictest mathematical sense, but may also mean “less than or equal to.”

Claims

1. A control system, comprising: A communication interface configured to receive image data from an imaging device associated with a endoscopic endoscope, the endoscope including a working channel for deploying medical devices; and Control circuitry, which is communicatively coupled to the communication interface and configured to: An image representation of the image data is generated; The image representation is displayed in the user interface; Determine the amount by which the endoscopic sight has been flipped; and The image representation is rotated within the user interface, at least in part based on the amount by which the peephole has been flipped. The control circuit is configured as follows: The user interface displays a working channel indicator that indicates the rotation angle of the working channel relative to the image representation; as well as Based at least in part on the amount by which the peephole has been flipped, the working channel indicator is rotated within the user interface from a first orientation relative to the user interface to a second orientation relative to the user interface.

2. The control system according to claim 1, wherein, The control circuit is configured to display the working channel indicator by overlaying it onto the image representation.

3. The control system according to claim 1, wherein, The control circuit is configured as follows: Confirm that the viewing mirror has completed the flipping; and In response to determining that the peephole has completed the flip, the image representation is rotated within the user interface.

4. The control system according to claim 1, wherein, The control circuit is configured to rotate the image display within the user interface as the peephole flips.

5. The control system according to claim 1, wherein, The control circuit is configured to determine the amount by which the peephole has been flipped based on at least one of the following: feedback data from a robotic arm configured to be coupled to the peephole; Sensor data generated by a position sensor associated with the peephole; or control signals generated to control the robotic arm.

6. The control system according to claim 1, wherein, The control circuit is configured as follows: Receive input control signals from the input device to rotate the working channel; and The peephole is controlled to flip at least in part based on the input control signal.

7. The control system according to claim 6, wherein, The control circuit is configured to control the rotation of the endoscopic mirror by controlling the mirror to rotate by a predetermined amount.

8. A method executed by a control system, comprising: Image data is received from an imaging device associated with a peephole, the peephole including a working channel; An image representation of the image data displayed in the user interface by a control circuit; The control circuit displays a working channel indicator in the user interface, the working channel indicator indicating the rotation angle of the working channel relative to the image representation; Determine the amount by which the endoscopic sight has been flipped; and Based at least in part on the amount by which the endoscopic mirror has been flipped: Rotate the image representation within the user interface; as well as Within the user interface, rotate the working channel indicator from the first orientation to the second orientation.

9. The method according to claim 8, further comprising: It is confirmed that the viewing mirror has completed the flipping process; The rotation of the image representation and the rotation of the working channel indicator occur in response to determining that the peephole has completed the flip.

10. The method according to claim 8, wherein, Rotating the image representation includes rotating the image representation in a continuous manner.

11. The method according to claim 10, wherein, The determination of the amount by which the peephole has been flipped is based on at least one of the following: feedback data from a robotic arm configured to be coupled to the peephole; position data generated by a position sensor associated with the peephole; or a control signal generated to control the robotic arm.

12. The method according to claim 10, wherein, Rotating the image representation includes displaying the image representation in a cropped form in the user interface while flipping the peephole.

13. The method of claim 8, further comprising: Receive user input from the input device to rotate the endoscopic mirror by a predetermined amount; as well as Control the viewing mirror to flip the predetermined amount.

14. The method according to claim 8, wherein, Displaying the working channel indicator includes overlaying the working channel indicator onto the image representation.

15. A control system, comprising: A hysteroscope, the hysteroscope including a working channel for removably receiving a medical device; An imaging device configured to be coupled to the endoscope; and A control circuit, communicatively coupled to the endoscope and the imaging device, is configured to: Receive image data from the imaging device; The image representation of the image data is displayed in the user interface; Display a working channel indicator in the user interface, the working channel indicator indicating the rotation angle of the working channel; Determine the amount by which the viewing mirror has been flipped; The working channel indicator in the user interface is rotated, at least in part, based on the amount by which the peephole has been flipped. as well as The rotation adjustment is applied to the image representation in the user interface based at least in part on the amount by which the peephole has been flipped.

16. The system according to claim 15, wherein, The working channel is offset from the longitudinal axis of the endoscope.

17. The system according to claim 15, wherein, The control circuit is configured as follows: Controlling the viewing mirror to flip in the rotational direction; and The rotation adjustment is applied to the image representation by rotating it in the rotation direction within the user interface.

18. The system according to claim 15, wherein, The control circuit is further configured as follows: It is confirmed that the viewing mirror has completed the flipping process; The rotation adjustment is applied in response to determining that the peephole has completed the flipping.

19. The system according to claim 15, wherein, The control circuit is configured to apply the rotation adjustment by continuously rotating the image representation as the peephole flips.

20. The system according to claim 15, wherein, The control circuit is further configured to determine the amount by which the peephole has been flipped based on at least one of the following: feedback data from a robotic arm configured to be coupled to the peephole; Sensor data generated by a position sensor associated with the peephole; or control signals generated to control the robotic arm.

21. The system according to claim 15, wherein, The control circuit is configured to apply the rotation adjustment by displaying the image representation in a cropped form in the user interface while the peephole is flipped.

22. The system according to claim 15, wherein, The control circuit is configured as follows: Receive input control signals to rotate the working channel; and The peephole is controlled to flip at least in part based on the input control signal.

23. One or more non-transitory computer-readable media storing computer-executable instructions, which, when executed by control circuitry, cause the control circuitry to perform operations including: Image data is received from an imaging device associated with a peephole, the peephole including a working channel; An image representation of the image data displayed in the user interface; The user interface displays a working channel indicator that indicates the rotation angle of the working channel. Determine the amount by which the viewing mirror has been flipped; The image representation is rotated within the user interface based at least in part on the amount by which the peephole has been flipped; as well as Based at least in part on the amount by which the peephole has been flipped, the working channel indicator is rotated from a first orientation to a second orientation within the user interface.

24. One or more non-transitory computer-readable media according to claim 23, wherein, The operation also includes: It is confirmed that the viewing mirror has completed the flipping process; In response to determining that the peephole has completed the flip, the image representation and the working channel indicator are rotated.

25. One or more non-transitory computer-readable media according to claim 23, wherein, Rotating the image representation includes rotating the image representation in a continuous manner.

26. One or more non-transitory computer-readable media according to claim 23, wherein, Determining the amount by which the peephole has been flipped includes tracking the amount of flipping of the peephole based on at least one of the following: feedback data from a robotic arm configured to be coupled to the peephole; Sensor data generated by a position sensor associated with the peephole; Or it may be generated as a control signal to control the robotic arm.

27. One or more non-transitory computer-readable media according to claim 23, wherein, Rotating the image representation includes displaying the image representation in a cropped form in the user interface while flipping the peephole.

28. One or more non-transitory computer-readable media according to claim 23, wherein, The operation also includes: Receive user input from the input device to rotate the endoscopic mirror by a predetermined amount; and Control the viewing mirror to flip the predetermined amount.

29. One or more non-transitory computer-readable media according to claim 23, wherein, The working channel indicator is displayed overlaid on the image representation.

Images