Method for controlling a robotic surgical system

Abstract

This invention discloses a display system for a robotic surgical system, which may include a display module configured to display a graphical user interface (GUI) on a display screen to provide information about one or more medical instruments. The GUI may include: an image display area for displaying endoscopic images extending from the sheath; a first instrument simulator configured to provide a first simulated representation of one or more medical instruments extending from the sheath from a first perspective view; and a second instrument simulator configured to provide a second simulated representation of the one or more medical instruments extending from the sheath from a second perspective view different from the first perspective view.

Description

Display system for robotic surgery system The present invention relates to robotic surgical systems, eg, for use in minimally invasive surgery, including but not limited to endovascular surgery and single-site surgery. Minimally invasive procedures, such as endocavity and single-site robotic surgery, offer significant advantages over conventional robotic surgery. For example, endocavity robotic surgery eliminates the need for incisions to access difficult-to-reach locations within a patient's natural cavity, significantly reducing and / or eliminating recovery time and improving surgical safety. Single-site systems reduce incisions to a single, minimal site, eliminating the need for multiple incisions to provide access for certain procedures. Certain intracavitary and single-site robotic surgical systems have been proposed, and examples of such systems and related components can be found in U.S. Patent Nos. US20210322046, US20210322045, US20190117247, US20210275266, US20210267702, US20200107898, US20200397457, US202000397456, US20200315645 and US20210322046, all of which are hereby incorporated by reference into this document for reference. Conventional surgical robots and systems are generally considered adequate for their intended purposes. However, there remains a need in the art for improved robotic surgical systems, devices, methods, controls, and assemblies, particularly those configured for endoluminal and single-site surgery. For example, the present invention provides improvements in these areas. This application claims priority to U.S. Provisional Application No. 63 / 284,125, filed on November 30, 2021, the entire contents of which are incorporated herein by reference. According to at least one aspect of the present invention, a display system for a robotic surgery system may include a display module configured to display a graphical user interface (GUI) on a display to provide information about one or more medical instruments. The GUI may include: an image display area for displaying an image from an endoscope extending from an overtube; a first instrument simulator configured to provide a first simulated representation of the one or more medical instruments extending from the overtube from a first perspective; and a second instrument simulator configured to provide a second simulated representation of the one or more medical instruments extending from the overtube from a second perspective different from the first perspective. The GUI may include an instrument control indicator comprising at least one of: an instrument name indicator; a connection status indicator indicating connection of the instrument to a hub; an attachment status indicator indicating attachment of the instrument to a robotic instrument controller; an instrument life indicator; and / or one or more instrument-specific operation indicators. The GUI may include a system clutch indicator indicating whether a system clutch is activated for operation to decouple one or more medical instruments, endoscopes, and / or overtubes of the system. The GUI may include a finger clutch indicator indicating whether a finger clutch is depressed for operation to decouple the one or more medical instruments associated with the finger clutch. The GUI may include a notification indicator for displaying a notification to the user, wherein the notification indicator is centered and located at the bottom of the display. The GUI may include an outer cannula translation indicator configured to indicate an amount of outer cannula translation and / or whether the translation is within one or more outer cannula translation limits. The GUI may include an outer cannula roll indicator for the outer cannula configured to indicate an amount of roll of the outer cannula and / or whether the roll of the outer cannula is within one or more outer cannula roll limits. The GUI may include an outer cannula bend indicator for the outer cannula configured to indicate an amount of bend of the outer cannula in a horizontal and / or vertical plane and / or whether the bend of the outer cannula is within one or more outer cannula bend limits. The GUI may include an endoscope bend indicator for the endoscope configured to indicate an amount of bend of the endoscope in a horizontal and / or vertical plane and / or whether the bend of the endoscope is within one or more endoscope bend limits. The GUI may include an endoscope shape indicator for the endoscope configured to indicate an amount of bend of a distal segment and an amount of bend of a proximal segment of the endoscope. The GUI may include an instrument roll indicator for each medical instrument configured to indicate an amount of roll of the instrument and / or whether the instrument roll is within one or more limits. The GUI may include an instrument translation indicator for each medical instrument configured to indicate an amount of translation of the instrument and / or whether the instrument translation is within one or more instrument translation limits. According to at least one aspect of the present invention, a robotic surgical system may include a patient console including a display and one or more input devices. The robotic surgical system may include any suitable embodiment of a display system as disclosed herein, for example, as described above. According to at least one aspect of the present invention, a method may include calculating a position and / or orientation of a robotically controlled elongated surgical device and generating one or more representational images of the elongated surgical device that display the position and / or orientation of the elongated surgical device relative to a neutral position. Calculating the position and / or orientation may include using instrument controller information to infer the position and / or orientation of the elongated surgical device. In some embodiments, generating the one or more representational images may include generating a plurality of two-dimensional indicators configured to indicate one or more of pitch, yaw, and roll. In some embodiments, the method may include generating a color indicator that changes color based on the relative position of one or more representative images to one or more extremes away from neutral. The color indicator may be associated with the representative image of the elongated device. The color indicator may be positioned near or around the representative image. Those skilled in the art will become more aware of these and other features of various embodiments of the present invention from the following description of the embodiments described in conjunction with the accompanying drawings. Reference will now be made to the accompanying drawings, in which like reference numerals identify like structural features or aspects of the present invention. For purposes of explanation and illustration, and not limitation, an illustrative diagram of an embodiment of a graphical user interface (GUI) according to the present invention is shown in FIG1 and generally designated by reference character 100. Other embodiments and / or aspects of the present invention are shown in FIG2 through FIG14. According to at least one aspect of the present invention, referring specifically to Figures 1-14 , a display system 100 for a robotic surgical system (e.g., an endoscopic robotic system) may include a display module 101 configured to display a graphical user interface (GUI) 200 on a display 103 to provide information related to one or more medical instruments 105 (e.g., robotically controlled endoscopic tools). Figure 1 is a perspective view of an embodiment of a system according to the present invention. Display module 101 may be operatively connected to any suitable feedback system, control module, input, and / or output of the robotic surgical system to receive any data associated with GUI 200 and / or indicators disclosed herein (e.g., as described below). Any module(s) disclosed herein may be or include any suitable hardware and / or software module(s) configured to perform any suitable functions (e.g., as disclosed herein). 2A and 2B illustrate an embodiment of a graphical user interface according to the present invention. GUI 200 may include an image display area 201 (e.g., depicted as a surgical scene in FIG. 2A or a video stream in FIG. 2B ) for displaying an image of an endoscope 1200 extending from an overtube 1100 , which may include a flexible, elongated insertion tube having one or more instrument channels therein for advancing / retracting medical devices therethrough (e.g., as shown in FIG. 11B and 11C ). 2A to 3B , the GUI 200 may also include a first instrument simulator 31 configured to provide a first simulated representation of one or more medical instruments 105 extending from the outer cannula 1100 from a first perspective (e.g., the left-view instrument simulator shown in FIG. 3A ); and a second instrument simulator 32 configured to provide a second simulated representation of one or more medical devices 105 extending from the outer cannula 1100 from a second perspective different from the first perspective (e.g., an orthogonal view such as the right-view instrument simulator shown in FIG. 3B ). Any suitable relevant perspectives are contemplated herein. Referring also to Figures 4-5D , GUI 200 may include an instrument control indicator 1 that includes, for example, at least one of: an instrument name indicator; a connection status indicator indicating the connection of the instrument to the hub; an attachment status indicator indicating the attachment of the instrument to the robotic instrument controller; an instrument life indicator; and / or one or more instrument-specific operation indicators (e.g., primary and auxiliary energy operations, as shown). Any suitable instrument control indicator is contemplated herein. In some embodiments, the instrument life may be fixed (e.g., an objectively determined fixed number of uses of the device). In some embodiments, the instrument life may vary based on one or more physical factors (e.g., actual / objectively determined wear) and / or manufacturer requirements. 6 , the GUI 200 may include a finger clutch indicator 2 for indicating whether the finger clutch is pressed to disconnect the operation of the one or more medical instruments 105 associated with the finger clutch. Furthermore, with reference to FIG7 , the GUI 200 may include a system clutch indicator 4 for indicating whether the system clutch is activated to disconnect the operation of one or more medical instruments 105, the outer cannula 1100, and / or the endoscope 1200 of the system. 8 , GUI 200 may include a notification indicator 5 for displaying notifications to the user, wherein the notification indicator is centered at the bottom of the display. Notification indicator 5 may include a partially transparent notification indicator and / or a notification indicator that changes color based on the content of the message, as shown, for example. 9A and 9B , the GUI 200 may include an overtube translation indicator 6 configured to indicate the amount of translation (e.g., axial limits) of the overtube 1100 (shown attached to the mobile patient cart / console 902) and / or whether the translation is within one or more overtube translation limits (e.g., limits as shown). Any suitable color-coding variation in the indicator based on proximity to a limit is contemplated herein. 10A-10C , the GUI 200 may include an outer tube roll indicator 7 for the outer tube 1100 and configured to indicate the amount of roll of the outer tube 1100 and / or whether the roll of the outer tube 1100 is within one or more outer tube roll limits (e.g., color-coded based on a percentage of minimum and maximum roll). In some embodiments, the image display area 201 may include a supplemental roll indicator 7A at the edge of the image, for example, as shown. 11A-11C , the GUI 200 may include an outer tube bend indicator 8 for the outer tube 1100 configured to indicate the amount of bend in the horizontal and / or vertical planes (e.g., as shown) and / or whether the bend of the outer tube 1100 is within one or more outer tube bend limits (e.g., color-coded based on a percentage of minimum and maximum bends). Referring also to FIGURES 12A-12C , the GUI 200 may include an endoscope bend indicator 9 for the endoscope 1200 configured to indicate the amount of bend in the horizontal and / or vertical planes (e.g., as shown) and / or whether the bend of the endoscope 1200 is within one or more endoscope bend limits (e.g., color-coded based on a percentage of minimum and maximum bends). Referring also to FIG. 13 , the GUI 200 may include an instrument roll indicator 10 for each medical instrument 105 configured to indicate the amount of instrument roll and / or whether the instrument roll is within one or more limits (e.g., color-coded based on a percentage of minimum and maximum roll). The instrument roll indicator 10 may be a line surrounding the image (simulated or real) of the attached medical instrument 105. For example, the line may gradually fill as more roll is presented and may change color (e.g., blue, yellow, orange, red) based on the percentage of fill. The GUI 200 may also include an instrument translation indicator 11 for each medical instrument configured to indicate the amount of instrument translation and / or whether the instrument translation is within one or more instrument translation limits (e.g., color-coded based on a percentage of minimum and maximum translation). Referring additionally to FIG. 14 , the GUI 200 may include an endoscope shape indicator 13 for the endoscope 1200, configured to indicate the amount of curvature of the distal segment 1200b and the amount of curvature of the proximal segment 1200a of the endoscope 1200. In some embodiments, the endoscope indicator 13 may be integrated with, for example, the first instrument simulator 31 and / or the second instrument simulator 32 (e.g., as shown). In some embodiments, the endoscope indicator 13 may be separate from, for example, the first instrument simulator 31 and / or the second instrument simulator 32. In some embodiments, the endoscope shape indicator 13 may be configured not only to provide a simulated representation of the endoscope in a bent state (e.g., in the bent shape shown), but also to provide simulated lines indicating the amount of curvature of the distal segment and the amount of curvature of the proximal segment of the endoscope 1200. According to at least one aspect of the present invention, a robotic surgical system (e.g., an endoscopic robotic surgical system) can include a user console 99 comprising a display and one or more input devices 97a, 97b (e.g., a left hand control device and / or a right hand control device). The robotic surgical system can include any suitable embodiment of a display system as disclosed herein, such as the system 100 described above. According to at least one aspect of the present invention, a method may include calculating a position and / or orientation of a robotically controlled elongated surgical device; and generating one or more representational images of the elongated surgical device that display the position and / or orientation of the elongated surgical device relative to a neutral position. Calculating the position and / or orientation may include using instrument controller information to infer the position and / or orientation of the elongated surgical device. In some embodiments, generating the one or more representational images may include generating a plurality of two-dimensional indicators configured to indicate one or more of pitch, yaw, and roll. In some embodiments, the method may include generating a color indicator that changes color based on the relative position of one or more representative images to one or more extremes away from neutral. The color indicator may be associated with the representative image of the elongated device. The color indicator may be located near or around the representative image. Embodiments of GUI 200 may include relative positions of indicators displayed to have advantageous ergonomic positions. Embodiments of GUI 200 may include shapes of each indicator displayed to have functional advantages. Any suitable combination of indicators and / or other configurations and / or shapes is contemplated herein. Certain embodiments of GUIs and / or indicators are disclosed above. Any other suitable GUIs and / or indicators thereof are contemplated herein. Those skilled in the art will appreciate that the present invention may be embodied as a system, method, or computer program product. Thus, the present invention may take the form of a complete hardware embodiment, a complete software embodiment (including firmware, resident software, microcode, etc.), or a combination of software and hardware embodiments, all of which are referred to herein as a "circuit," "module," or "system." A "circuit," "module," or "system" may include one or more portions of one or more individual physical hardware and / or software components that together perform the functions disclosed by the "circuit," "module," or "system," or the "circuit," "module," or "system" may be a single, independent unit (e.g., hardware and / or software). Furthermore, aspects of the present invention may take the form of a computer program product embodied in one or more computer-readable media having computer-readable program code embodied in the media. This specification may utilize any combination of one or more computer-readable media. The computer-readable media may be a computer-readable signal medium or a computer-readable storage medium. Computer-readable media may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination thereof. Further specific examples of computer-readable storage media (non-exhaustive) include: an electrical connection having one or more wires, a portable computer disk, a hard drive, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the context of this document, a computer-readable storage medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. Computer-readable signal media may include a propagated data signal embodying computer-readable program code in a baseband or partial carrier wave. Such a propagated signal may take any form, including but not limited to electromagnetic, optical, or any suitable combination. Computer-readable signal media may be any computer-readable medium that is not a computer-readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied in a computer-readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing. The computer program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented languages ​​such as Java, Smalltalk, and C++, as well as traditional programming languages ​​such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any network, including a local area network (LAN) or a wide area network (WAN), or connected to an external computer (e.g., via the Internet provided by an Internet service provider). Aspects of the present invention are described above with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to specific embodiments of the present invention. It will be understood that each block in any flowchart and / or block diagram, and any combination of blocks in any flowchart and / or block diagram, may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to generate machine instructions, such as instructions, which are executed by the processor of the computer or other programmable data processing device, or by a device for performing the functions / actions specified in any flowchart and / or block diagram. These computer program instructions may also be stored in a computer-readable medium to direct a computer, other programmable data processing equipment, or other devices to operate in a specific manner, so that the instructions stored in the computer-readable medium produce a manufacturing body, including a device that implements the functions / actions specified in the flowchart and / or block diagram blocks. Computer program instructions may also be loaded into a computer, other programmable data processing device, or other apparatus, causing a series of operating steps to be executed on the computer, other programmable device, or other apparatus to produce a computer-implemented process, such that the instructions executed on the computer or other programmable device provide a process for implementing the functions / actions specified herein. Those skilled in the art will appreciate that any numerical value disclosed herein may be an exact value or may be within a range of values. Furthermore, any approximate terms used herein (e.g., "about," "approximately," "approximately") may indicate that the recited value is within a range. For example, in certain embodiments, the range may be within (plus or minus) 20%, or within 10%, or within 5%, or within 2%, or any other suitable percentage or value understood by those skilled in the art (e.g., for known tolerance limits or error ranges). Unless otherwise indicated, the terms "a," "an," and "the" used herein and in the following claims are used to refer to one or more than one (i.e., at least one) subject matter. For example, "an element" refers to one element or more than one element. As used in the specification and claims, the term "and / or" should be understood to mean "one or both" of the multiple elements being combined, i.e., elements that are present together in some cases and separately in other cases. Multiple elements listed with "and / or" should be interpreted in the same manner, i.e., "one or more" of the multiple elements being combined. In addition to the elements specifically identified by the "and / or" clause, other elements may optionally be present, whether related or unrelated to those specifically identified elements. Thus, as a non-limiting example, when used in conjunction with an open-ended term such as "comprising," a reference to "A and / or B" may refer to only A (optionally including elements other than B) in one embodiment; only B (optionally including elements other than A) in another embodiment; both A and B (optionally including other elements) in yet another embodiment; and so on. As used herein in the specification and claims, "or" should be understood to have the same meaning as "and / or" as defined above. For example, when separating items in a list, "or" or "and / or" should be interpreted as inclusive, i.e., including at least one, but also including more than one of a plurality of listed elements, and optionally may include additional unlisted items. Only when explicitly stated to the contrary, such as "only one" or "exactly one", or when used in the claims, "consisting of" means consisting of exactly one element of a plurality or listed elements. In general, the term "or" as used herein should only be interpreted to indicate exclusive alternatives (i.e., "one or the other, but not both"), provided that exclusive terms such as "either", "one of", "only one of", or "exactly one of" are used. Any suitable combination(s) of any disclosed embodiments and / or any suitable portion(s) thereof are contemplated herein, as would be understood by one of ordinary skill in the art in light of this disclosure. The specific embodiments of the present invention as described above and shown in the accompanying drawings provide improvements in the field to which they belong. Although the present invention includes reference to certain specific embodiments, those skilled in the art will readily appreciate that changes and / or modifications may be made thereto without departing from the spirit and scope of the present invention. 1: Instrument control indicator 2: Finger clutch indicator 4: System clutch indicator 5: Notification indicator 6: Outer cannula translation indicator 7: Outer cannula roll indicator 7A: Supplemental roll indicator 8: Outer cannula bend indicator 9: Endoscope bend indicator 10: Instrument roll indicator 11: Instrument translation indicator 13: Endoscope shape indicator 31: First instrument simulator 32: Second instrument simulator 97a, 97b: Input device 99: User console 100: Display system 101: Display module 103: Display 105: Medical instrument 200: Graphical user interface 201: Image display area 902: Patient cart / console 1100: Outer cannula 1200: Endoscope In order to enable those skilled in the art to easily understand how to make and use the apparatus and method of the present invention without undue experimentation, specific embodiments thereof will be described in detail below with reference to the accompanying drawings, in which: FIG1 is a perspective view of a specific embodiment of a system according to the present invention; FIG2A and 2B illustrate a specific embodiment of a graphical user interface according to the present invention; FIG3A illustrates a specific embodiment of a first instrument simulator according to the present invention; FIG3B illustrates a specific embodiment of a second instrument simulator according to the present invention; FIG4 illustrates a specific embodiment of an instrument control indicator according to the present invention; FIG5A, 5B, 5C and 5D illustrate specific embodiments of one or more instrument control indicators according to the present invention; FIG6 illustrates a specific embodiment of a finger clutch indicator according to the present invention; FIG7 illustrates a specific embodiment of a system clutch indicator according to the present invention; FIG8 illustrates a specific embodiment of a notification indicator according to the present invention, which displays various states; FIG9A illustrates a specific embodiment of an outer sleeve translation indicator according to the present invention; FIG9B illustrates the specific embodiment of FIG9A in the context of a specific embodiment of user input and system output according to the present invention; FIG10A illustrates a specific embodiment of an outer sleeve roll indicator according to the present invention; Figure 10B illustrates a specific embodiment of Figure 10A in a specific embodiment context of clockwise user input and system output according to the present invention; Figure 10C illustrates a specific embodiment of Figure 10A in a specific embodiment context of counterclockwise user input and system output according to the present invention; Figure 11A illustrates a specific embodiment of an outer sleeve bend indicator according to the present invention, which displays various states; Figures 11B and 11C illustrate a specific embodiment of Figure 11A in a specific embodiment context of user input and system output according to the present invention; Figure 12A illustrates a specific embodiment of an endoscope bend indicator according to the present invention, which displays various states; Figures 12B and 12C illustrate a specific embodiment of Figure 12A in a specific embodiment context of user input and system output according to the present invention; Figure 13 illustrates a specific embodiment of an instrument roll indicator according to the present invention, which displays various states; and Figure 14 illustrates a specific embodiment of an endoscope shape indicator according to the present invention, which displays various states. 97a, 97b: Input device 100: Display system 101: Display module 103: Display 105: Medical Equipment 200: Graphical User Interface

Claims

1. A method for controlling a robotic surgical system, comprising: calculating the position and / or orientation of an elongated surgical device controlled by the robot; generating one or more characterization images of the elongated surgical device, wherein the one or more characterization images are simulated representations showing the position and / or orientation of the elongated surgical device relative to an intermediate position; generating a translation indicator indicating whether a translation amount of the elongated surgical device is within one or more translation limits; generating an endoscope shape indicator providing a simulated representation of an endoscope in a bent state, and the endoscope shape indicator indicating a distal segment bend and a proximal segment bend of the endoscope; and generating a color indicator that changes color based on the relative position of the one or more characterization images to one or more limits away from neutrality; wherein the color of the color indicator is encoded according to the percentage of minimum and maximum roll; rendering a single graphical user interface including the one or more characterization images, the translation indicator, the endoscope shape indicator, and the color indicator.

2. The method as described in claim 1, wherein calculating the position and / or orientation includes using instrument controller information to infer the position and / or orientation of the elongated surgical device.

3. The method as described in claim 1, wherein generating the one or more characterization images includes generating a plurality of two-dimensional indicators configured to indicate one or more of pitch, yaw, and roll.

4. The method as described in claim 1, wherein the color indicator is associated with one or more characterizing images of an elongated device.

5. The method as described in claim 4, wherein the position of the color indicator is adjacent to or surrounds the or the characterizing images.

6. The method as described in claim 1, wherein the single graphical user interface further includes a video image from the endoscope.

7. The method as described in claim 1, wherein the one or more characterization images include a first and a second instrument simulator.

8. The method as described in claim 7, wherein: The first instrument simulator is a two-dimensional graphic representation of the real-time three-dimensional structure of the elongated surgical device obtained from a first viewing position; the second instrument simulator is a two-dimensional graphic representation of the real-time three-dimensional structure of the elongated surgical device obtained from a second viewing position; wherein the first viewing position is different from the second viewing position.

9. The method as described in claim 8, wherein one of the first viewing position and the second viewing position is a perspective view.

10. The method as described in claim 8, wherein one of the first viewing position and the second viewing position is a top view.

11. The method as described in claim 8, wherein the first instrument simulator and the second instrument simulator illustrate the elongated surgical device extending from an outer sheath.

12. The method as described in claim 1, further comprising: generating an outer tube shape indicator that provides a simulated representation of a controllable portion of a controllable outer tube.

Images