System and method for interactive point display in a remotely operated component

Abstract

A teleoperation system in a surgical environment includes: a teleoperation assembly including a first teleoperation arm; a first display device coupled to the teleoperation assembly; and a processor. The processor is configured to monitor the positioning of the first display device in the surgical environment and present a first image on the first display device. The first image is presented based on the positioning of the first display device in the surgical environment.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application No. 62 / 543,594, filed August 10, 2017, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to surgical systems and methods for performing minimally invasive teleoperated medical procedures using minimally invasive medical techniques, and more particularly to systems and methods for providing interactive point displays for use by operating room clinicians during medical procedures. Background Art

[0004] Minimally invasive medical techniques are intended to reduce the amount of external tissue damaged during diagnostic or surgical procedures, thereby reducing patient recovery time, discomfort, and harmful side effects. Such minimally invasive techniques can be performed through natural orifices in the patient's anatomy or through one or more surgical incisions. Through these natural orifices or incisions, clinicians can insert medical tools to reach the target tissue location. Minimally invasive medical tools and instruments, such as therapeutic instruments, diagnostic instruments, and surgical instruments. Minimally invasive medical tools may also include imaging devices that provide users with a view within the patient's anatomy, such as endoscopic instruments.

[0005] Certain minimally invasive medical tools may be operated remotely, or with computer assistance. A clinician near a teleoperating system may need to receive guidance in the form of instructions, warnings, confirmations, etc. before, during, or after a medical procedure performed with the teleoperating system. Systems and methods are needed for providing visual display of interaction points for guidance information. Summary of the invention

[0006] Embodiments of the invention are summarized by the appended claims. In one embodiment, a teleoperation system in a surgical environment includes a teleoperation assembly having a first teleoperation arm, a first display device coupled to the teleoperation assembly, and a processor. The processor is configured to monitor a position of the first display device in the surgical environment and present a first image on the first display device. The first image is presented based on the position of the first display device in the surgical environment.

[0007] In another embodiment, a method includes monitoring a position of a first display device in a surgical environment and presenting a first image on the first display device based on the position of the first display device in the surgical environment.

[0008] In another embodiment, a teleoperation system in a surgical environment includes a teleoperation assembly having a first teleoperation arm, a visual projection device coupled to the teleoperation assembly, a sensor, and a processor. The processor is configured to receive first sensor information from the sensor, determine a first visual aid based on the first sensor information, operate the visual projection device to project the first visual aid into the surgical environment, and operate the visual projection device to change the first visual aid to a second visual aid based on second sensor information received from the sensor.

[0009] In another embodiment, a method includes: receiving first sensor information from a sensor of a teleoperation system in a surgical environment; determining a first visual aid based on the first sensor information; and operating a visual projection device to project the first visual aid into the surgical environment. The visual projection device is coupled to a teleoperation component of the teleoperation system. The method also includes: receiving second sensor information from the sensor; determining a second visual aid based on the second sensor information; and operating the visual projection device to change the first visual aid to the second visual aid.

[0010] It should be understood that the above general description and the following detailed description are exemplary and illustrative in nature, and are intended to provide an understanding of the present disclosure without limiting the scope of the present disclosure. In this regard, other aspects, features and advantages of the present disclosure will be apparent to those skilled in the art based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Aspects of the present disclosure may be best understood from the following detailed description when read in conjunction with the accompanying drawings. It is emphasized that, in accordance with standard practice in the industry, various features are not drawn to scale. In fact, the dimensions of various features may be increased or decreased arbitrarily for clarity. In addition, the present disclosure may repeat reference numerals and / or letters in various examples. This repetition is for simplicity and clarity purposes and does not in itself indicate the relationship between the various embodiments and / or configurations discussed.

[0012] Figure 1A is a schematic diagram of a remotely operated medical system according to an embodiment of the present disclosure.

[0013] Figure 1B is a perspective view of a remote operations assembly, according to one example of principles described herein.

[0014] Figure 1C is a perspective view of a surgeon's console for teleoperating a medical system, according to one embodiment.

[0015] Figure 2 A method for providing information in a surgical environment according to an embodiment of the present disclosure is described.

[0016] Figure 3 Another method of providing information in a surgical environment according to an embodiment of the present disclosure is described.

[0017] Figure 4 Describes a surgical environment in which visual aids are used to assist in initial patient approach.

[0018] Figure 5 A surgical environment is described in which a visual aid is used to assist in the positioning of a directional platform.

[0019] Figure 6 A surgical environment is described in which a visual aid is used to assist in the orientation of an orientation platform.

[0020] Figure 7 A surgical environment is described in which another visual aid is used to assist in the orientation of the orientation platform.

[0021] Figure 8 A surgical environment is illustrated in which a visual aid is used to highlight a portion of a teleoperated assembly. DETAILED DESCRIPTION

[0022] For the purpose of promoting understanding of the principles of the present disclosure, reference will now be made to the embodiments shown in the accompanying drawings, and specific language will be used to describe them. However, it will be understood that it is not intended to limit the scope of the present disclosure. In the following detailed description of various aspects of the present invention, many specific details are set forth to provide a thorough understanding of the disclosed embodiments. However, as will be appreciated by those skilled in the art, embodiments of the present disclosure may be practiced without these specific details. In other cases, well-known methods, procedures, components and circuits are not described in detail to avoid unnecessarily obscuring various aspects of embodiments of the present invention.

[0023] Any changes and further modifications to the described devices, apparatuses, methods, and any further applications of the principles of the present invention are fully contemplated, as would be normally expected by a person skilled in the art to which the present invention relates. In particular, it is fully contemplated that the features, components, and / or steps described with respect to one embodiment may be combined with the features, components, and / or steps described with respect to other embodiments of the present disclosure. In addition, the dimensions provided herein are for specific examples, and it is contemplated that different dimensions, sizes, and / or ratios may be utilized to implement the concepts of the present disclosure. In order to avoid unnecessary descriptive repetitions, one or more components or actions described with respect to an illustrative embodiment may be used or omitted according to the applicability of other illustrative embodiments. For the sake of brevity, many iterations of these combinations will not be described separately. For the sake of simplicity, in some cases, the same reference numerals are used in all drawings to refer to the same or similar parts.

[0024] The following embodiments will describe various instruments and parts of instruments according to their state in three-dimensional space. As used herein, the term "positioning" refers to the position of an object or a portion of an object in three-dimensional space (e.g., three translational degrees of freedom along Cartesian X, Y, Z coordinates). As used herein, the term "orientation" refers to the rotational placement of an object or a portion of an object (three rotational degrees of freedom - e.g., roll, pitch, and yaw). As used herein, the term "posture" refers to the positioning of an object or a portion of an object in at least one translational degree of freedom and the orientation of the object or a portion of an object in at least one rotational degree of freedom (up to six degrees of freedom).

[0025] With reference to the attached drawings Figure 1A A teleoperated medical system for use in a medical procedure, such as a diagnostic, therapeutic, or surgical procedure, is generally indicated by reference numeral 10. System 20 is located in a surgical environment 11. As will be described, the teleoperated medical system of the present disclosure is under teleoperated control of a surgeon. In alternative embodiments, the teleoperated medical system may be under partial control of a computer programmed to perform a procedure or subroutine. In other alternative embodiments, a fully automated medical system under full control of a computer programmed to perform a procedure or subroutine may be used to perform the procedure or subroutine. An example of a teleoperated medical system that may be used to implement the systems and techniques described in the present disclosure is the da Surgical System.

[0026] like Figure 1A As shown, the teleoperation medical system 10 generally includes a teleoperation assembly 12, which can be mounted on the operating table O where the patient P is located or is positioned near it. The teleoperation assembly 12 can be referred to as a patient side cart, a surgical cart, a teleoperation arm cart, or a surgical robot. The medical instrument system 14 and the endoscopic imaging system 15 are operably connected to the teleoperation assembly 12. The operator input system 16 allows the surgeon or other type of clinician S to view images of the surgical site or images representing the surgical site and control the operation of the medical instrument system 14 and / or the endoscopic imaging system 15. It should be understood that the medical instrument system 14 may include one or more medical instruments. In an embodiment where the medical instrument system 14 includes multiple medical instruments, the multiple medical instruments may include multiple identical medical instruments and / or multiple different medical instruments. Similarly, the endoscopic imaging system 15 may include one or more endoscopes. In the case of multiple endoscopes, the multiple endoscopes may include multiple identical endoscopes and / or multiple different endoscopes.

[0027] The operator input system 16 may include a surgeon's console and may be located in the same room as the operating table O. However, it should be understood that the surgeon S and the operator input system 16 may be located in different rooms or in a completely different building from the patient P. The operator input system 16 typically includes one or more control devices for controlling the medical instrument system 14. The control device(s) may include one or more of a variety of input devices, such as a handle, a joystick, a trackball, a data glove, a trigger gun, a foot pedal, a manual controller, a voice recognition device, a touch screen, a human motion or presence sensor, and the like. In some embodiments, the control device(s) are provided with the same degrees of freedom as the medical instruments of the medical instrument system 14 to provide telepresence to the surgeon, i.e., the control device(s) are perceived as one piece with the instrument, so that the surgeon has a strong sense of direct control of the instrument, just like being at the surgical site. In other embodiments, the control device(s) may have more or fewer degrees of freedom than the associated medical instrument and still provide telepresence to the surgeon. In some embodiments, one (or more) control devices are manual input devices that move with six degrees of freedom and may also include an actuatable handle for actuating the instrument (e.g., for closing a grasping jaw end effector, applying a potential to an electrode, administering a medication, etc.).

[0028] While the surgeon S views the surgical site through the operator input system 16, the remote operation assembly 12 supports and manipulates the medical device system 14. Images of the surgical site can be obtained by an endoscopic imaging system 15 that can be manipulated via the remote operation assembly 12. The remote operation assembly 12 may include an endoscopic imaging system 15, and similarly may also include multiple medical device systems 14. The number of medical device systems 14 used at one time will generally depend on factors such as the diagnostic or surgical procedure to be performed and the space limitations in the operating room. The remote operation assembly 12 may include a kinematic structure of one or more non-servo-controlled links (e.g., one or more links that can be manually positioned and locked in an appropriate position, generally referred to as a setting structure) and a remote operation manipulator. The remote operation assembly 12 includes multiple motors that drive input members on the medical device system 14. In one embodiment, these motors move in response to commands from a control system (e.g., control system 20). The motor includes a drive system that, when connected to the medical device system 14, can advance the medical device into an anatomical orifice created naturally or by surgery. Other motorized drive systems can move the distal end of the medical device with multiple degrees of freedom, which can include three linear motions (e.g., linear motion along the X, Y, Z Cartesian axes) and three rotational motions (e.g., rotation about the X, Y, Z Cartesian axes). In addition, a motor can be used to actuate an articulated end effector of the medical device to grasp tissue in the jaws of a biopsy device, etc. The medical device of the medical device system 14 may include an end effector having a single working member (such as a scalpel, a blunt knife, an optical fiber, or an electrode). Other end effectors may include, for example, forceps, graspers, scissors, or clip applicators.

[0029] The telemedicine system 10 also includes a control system 20. The control system 20 includes at least one memory 24 and at least one processor 22 for implementing control between the medical device system 14, the operator input system 16, and other auxiliary systems 26, which may include, for example, imaging systems, audio systems, fluid delivery systems, display systems, lighting systems, steering control systems, irrigation systems, and / or suction systems. The clinician C can roam within the surgical environment 11 and can access (for example) the components 12 during the setup procedure or view the display of the auxiliary system 26 from the patient's bedside.

[0030] Despite Figure 1A 12, although in some embodiments, the control system 20 may be completely contained within the remote operation assembly 12. The control system 20 also includes programmed instructions (e.g., stored on a non-transitory computer readable medium) to implement some or all of the methods described in accordance with aspects disclosed herein. Figure 1AThe control system 20 is shown as a single box in the simplified schematic diagram of , but the system may include two or more data processing circuits, where a portion of the processing is optionally performed on or near the remote operation component 12, another portion of the processing is performed on the operator input system 16, etc. Any of a variety of centralized or distributed data processing architectures can be used. Similarly, the programmed instructions can be implemented as multiple separate programs or subroutines, or they can be integrated into multiple other aspects of the remote operation system described herein. In one embodiment, the control system 20 supports wireless communication protocols such as Bluetooth, IrDA, HomeRF, IEEE 802.11, DECT, and wireless telemetry.

[0031] The control system 20 is in communication with a database 27, which may store one or more clinician profiles, a list of patients and patient profiles, a list of procedures to be performed on the patients, a list of clinicians scheduled to perform the procedures, other information, or a combination thereof. The clinician profile may include information about the clinician, including how long the clinician has been in the medical field, the level of education the clinician has obtained, the clinician's level of experience with teleoperating the medical system 10 (or similar system), or any combination thereof.

[0032] The database 27 may be stored in the memory 24 and may be dynamically updated. Additionally or alternatively, the database 27 may be stored on a device such as a server or a portable storage device that is accessible by the control system 20 via an internal network (e.g., a secure network of a medical facility or a remote operating system provider) or an external network (e.g., the Internet). The database 27 may be distributed in two or more locations. For example, the database 27 may exist on multiple devices, which may include devices of different entities and / or cloud servers. Additionally or alternatively, the database 27 may be stored on a portable user-assigned device, such as a computer, a mobile device, a smart phone, a laptop computer, an electronic badge, a tablet computer, a pager, and other similar user devices.

[0033] In some embodiments, the control system 20 may include one or more servo controllers that receive force and / or torque feedback from the medical device system 14. In response to the feedback, the servo controller sends a signal to the operator input system 16. The servo controller(s) may also send a signal to instruct the remote operation assembly 12 to move the medical device system(s) 14 and / or the endoscopic imaging system 15 that extend through an opening in the body to an internal surgical site in the patient's body. Any suitable conventional or specialized servo controller may be used. The servo controller may be separate from or integrated with the remote operation assembly 12. In some embodiments, the servo controller and the remote operation assembly 12 are provided as part of a remote operation arm cart positioned near the patient's body.

[0034] The control system 20 can be coupled to the endoscope 15 and can include a processor to process the captured images for subsequent display to the surgeon, such as on the surgeon's console or another suitable display locally and / or remotely. For example, where a stereo endoscope is used, the control system 20 can process the captured images to present the surgeon with a coordinated stereo image of the surgical site. Such coordination can include alignment between opposing images and can include adjusting the stereo working distance of the stereo endoscope.

[0035] In alternative embodiments, the teleoperated medical system 10 may include more than one teleoperated assembly 12 and / or more than one operator input system 16. The exact number of teleoperated assemblies 12 will depend on the surgical procedure and space limitations in the operating room, among other factors. The operator input systems 16 may be collocated, or they may be placed in separate locations. Multiple operator input systems 16 allow more than one operator to control one or more teleoperated assemblies 12 in various combinations.

[0036] Figure 1B 1 is a perspective view of an embodiment of a teleoperation assembly 12, which may be referred to as a patient side cart, surgical cart, teleoperation arm cart, or surgical robot. The teleoperation assembly 12 shown provides manipulation of three surgical tools 30a, 30b, 30c (e.g., medical instrument system 14) and an imaging device 28 (e.g., endoscopic imaging system 15), such as a stereo endoscope for capturing images of the site of the procedure. The imaging device may transmit signals to the control system 20 via a cable 56. Manipulation is provided by a teleoperation machine having multiple joints. The imaging device 28 and the surgical tools 30a-c may be positioned and manipulated through an incision in the patient, thereby keeping the kinematic remote center at the incision to minimize the size of the incision. When the distal ends of the surgical tools 30a-c are positioned within the field of view of the imaging device 28, the image of the surgical site may include an image of the distal ends of the surgical tools 30a-c.

[0037] The remote operation assembly 12 includes a drivable base 58. The drivable base 58 is connected to a telescoping column 57, which allows the height of the arm 54 to be adjusted. The arm 54 may include a swivel joint 55 that both rotates and moves up and down. Each of the arms 54 may be connected to an orientation platform 53. The arms 54 may be marked to facilitate troubleshooting. For example, each of the arms 54 may be decorated with different numbers, letters, symbols, other identifiers, or combinations thereof. Figure 1B , arms 54 are numbered from one to four. The orientation platform 53 may be capable of rotating 360 degrees. The remote operation assembly 12 may also include a telescopic horizontal boom 52 for moving the orientation platform 53 in a horizontal direction.

[0038] In this example, each of the arms 54 is connected to a manipulator arm 51. The manipulator arm 51 can be directly connected to a medical instrument, such as one of the surgical tools 30a-c. The manipulator arm 51 can be remotely operated. In some examples, the arms 54 connected to the directional platform 53 may not be remotely operated. Instead, these arms 54 can be positioned as needed before the surgeon S begins surgery with the remotely operated components. Throughout the surgical procedure, the medical device can be removed and replaced with other instruments, so that the association of the instrument with the arm can change during the procedure. Displays such as displays 62a-d can help enhance the operating function of each arm based on the currently attached instrument.

[0039] Endoscopic imaging systems (e.g., endoscopic imaging system 15 and imaging device 28) can be provided in various configurations including rigid or flexible endoscopes. Rigid endoscopes include a rigid tube that houses a relay lens system for transmitting images from the distal end of the endoscope to the proximal end. Flexible endoscopes use one or more flexible optical fibers to transmit images. Endoscopes based on digital images have a "chip on the tip" design, in which distal digital sensors such as one or more charge-coupled devices (CCDs) or complementary metal oxide semiconductor (CMOS) devices store image data. Endoscopic imaging systems can provide two-dimensional or three-dimensional images to viewers. Two-dimensional images can provide limited depth perception. Three-dimensional stereoscopic endoscopic images can provide viewers with more accurate depth perception. Stereoscopic endoscopic devices use stereo cameras to capture stereoscopic images of patient anatomical structures. Endoscopic instruments can be fully sterilizable components, with endoscope cables, handles, and shafts all rigidly connected and hermetically sealed.

[0040] A projector 60 (e.g., a type of accessory system 26) can be coupled to or integrated into the teleoperation assembly 12. As shown, the projector 60 can be located on the directional platform 53. In one embodiment, the projector 60 can be centrally located on the bottom side of the directional platform 53. In some cases, the projector 60 can be located elsewhere. For example, the projector 60 can be located on one of the arms 54, on the telescoping column 57, on the drivable base 58, on the telescoping horizontal boom 52, or elsewhere. The location of the projector 60 can be selected based at least in part on the kinematics of the teleoperation assembly 12. The projector 60 can be rigidly mounted or integrated to the teleoperation assembly so that the projector is in a known configuration relative to the kinematically tracked manipulator arm 51. The projector 60 can be positioned so that movement of the manipulator arm 51 during surgery does not change the orientation of the projector 60. Although not affected by the movement of the manipulator arm 51, the projector 60 itself can be rotated, turned, pivoted, or otherwise moved so that images can be projected in different directions without changing the orientation of the teleoperation assembly 12. In Figure 1B Only a single projector 60 is shown in FIG. 5 ; however, the remote operations assembly 12 may include multiple projectors 60 , such as one or more projectors 60 on each of the arms 54 .

[0041] The size and shape of the projector 60 can be designed to be substantially contained within or on a component of the teleoperated assembly 12, such as the arm 54, the directional platform 53, the drivable base 58, the telescopic column 57, the telescopic horizontal cantilever 52, etc. The projector 60 may include a digital light processing (DLP), a liquid crystal on silicon (LCoS) or a laser beam steering (LBS) microprojector or other type of still or motion visual image projector. The projector 60 may project a color image. In order to minimize the effect of ambient light on the image produced by the projector, the projector 60 may produce an image bright enough to be easily perceived despite any ambient light (if any) in the operating room. Projector 60 can project an image having a brightness of approximately 500 lumens, approximately 1,000 lumens, approximately 1,500 lumens, approximately 2,000 lumens, approximately 2,500 lumens, approximately 3,000 lumens, approximately 3,500 lumens, approximately 4,000 lumens, approximately 4,500 lumens, approximately 5,000 lumens, approximately 5,500 lumens, approximately 6,000 lumens, approximately 6,500 lumens, approximately 7,000 lumens, or other brightness.

[0042] Projector 60 may be controlled by teleoperation assembly 12 and / or operator input system 16. In one embodiment, teleoperation assembly 12 may operate projector 60 to provide guidance to a clinician in the operating room in the form of visual aids (which may also be accompanied by audio aids). Visual aids may include graphical indicators, symbols, alphanumeric content, light patterns, or any other visual information.

[0043] The sensor 61 may be located on the orientation platform 53 or elsewhere and may be used to determine whether the patient or the operating table is within the working area of ​​the teleoperation assembly 12. For the purposes of the present disclosure, the working area of ​​the teleoperation assembly 12 may be defined by the range of motion of the surgical tool 30a-c. The sensor 61 may be, for example, a depth sensor or a thermal sensor. The thermal sensor may include an infrared sensor that generates sensor information that is used to determine whether the patient is within the working area by comparing the readings from the sensor with the patient's expected thermal curve. The depth sensor may be, for example, an ultrasonic rangefinder, an infrared rangefinder, a laser rangefinder, a depth camera, or a combination thereof. The depth sensor may measure the distance between the sensor and the surface directly below the sensor. When the teleoperation assembly 12 is not located near the operating table, the surface directly below the sensor may be the floor of the operating room. Conversely, when the teleoperation assembly 12 is located near the operating table, the nearest surface directly below the sensor may be the operating table or a patient located on the operating table.

[0044] The predetermined distance value can be associated with the height of the operating table. If the sensor information from the depth sensor is greater than the predetermined distance, the sensor information indicates that there is no operating table and / or patient in the working area. If the sensor information from the depth sensor is equal to or less than the predetermined distance, the sensor information indicates that there is an operating table and / or patient in the working area. The predetermined distance value can be any value between approximately 30 inches and 60 inches. For example, the predetermined threshold value can be approximately 36 inches, approximately 40 inches, approximately 48 inches, approximately 52 inches, or some other value. In some cases, a second distance value can be set so that when the distance between the sensor and the surface directly below the sensor falls between two predetermined values, the remote operation component 12 determines that it is adjacent to the operating table and / or has a patient located in the working area.

[0045] One or more displays 62a, 62b, 62c, 62d (e.g., an accessory system 26) can be coupled or integrated into the teleoperation assembly 12. As discussed in more detail below, the displays 62a-d can display visual aids including, for example, the status of the arm 54, and can be used as an interface to allow a clinician (e.g., clinician C) to receive guidance from the teleoperation assembly 12 and / or issue instructions to the teleoperation assembly 12. It is contemplated that in certain circumstances, such as in the event of a device failure on one of the arms 54, the displays 62a-d and the projector 60 can work together to maximize the information conveyed to the operating room clinician. As shown, the teleoperation assembly 12 includes displays 62a-d, wherein each display is located on each arm 54. The teleoperation assembly 12 is depicted as including four displays; however, the teleoperation assembly can include more or fewer displays 62. Although the displays 62a-d are shown as being located on a vertical section of the arm 54, it is contemplated that one or more of the displays 62a-d can be located on other sections, such as a horizontal section of the arm 62. The displays 62 may be positioned so that their positioning remains constant or relatively constant during surgery relative to the assembly 12. Additionally, the displays 62 may be located on a portion of the arm 54 that does not move or moves little after an initial setup procedure.

[0046] Displays 62a-d may be located at approximately eye level of clinician C, or between approximately 48 inches and 72 inches above the floor of the surgical environment. Positioning display 62 at approximately eye level may improve visibility of displays 62a-d and may increase the likelihood that an operating room clinician will see information presented on displays 62a-d.

[0047] The display 62a-d can be sized for a position on the arm 54 and can be sized to be accessible around or through a sterile drape. For example, the display 62a-d can be, for example, a square or rectangular shape with a size of about 5 inches to about 9 inches on one side. In various embodiments, the display 62a-d is integrated into the teleoperation assembly 12 and is wired to communicate with other components of the teleoperation assembly 12, such as the control system 20. In other embodiments, the display 62a-d can be removably attached to the arm 54 of the teleoperation assembly 12 and can communicate wirelessly with other components of the teleoperation assembly 12. The display 62a-d can support various wireless protocols, including Bluetooth, IrDA, HomeRF, IEEE802.11, DECT, and wireless telemetry. The display 62a-d can communicate with other accessory systems 26 including cameras and other displays. The display 62a-d can display images from: an endoscopic imaging system 15, other cameras in an operating room or elsewhere, other displays in an operating room or elsewhere, or a combination thereof.

[0048] In various embodiments, each display 62a-d is configured to display information about the arm 54 on which it is located. Figure 1B , display 62a can be configured to display information about arm 54, which is labeled with the number one. As described above, teleoperation assembly 12 can monitor the status of its various components. By being fixed to the arms or by being mounted to teleoperation assembly 12, displays 62a-d can maintain their spatial association with their respective arms so that the content on the displays is spatially arranged in the same order as the arms when viewed by a user.

[0049] The remote operation assembly 12 also includes a dashboard 64. As discussed in more detail below, the dashboard 64 can display the status of the arm 54 and can be used as an interface that allows the clinician to issue instructions to the remote operation assembly 12, etc. It is expected that in some cases, such as when a device on one of the arms 54 fails, the dashboard 64, the display 62a-d, the projector 60, or a combination thereof, can work together to maximize the possibility of conveying information to the clinician in the operating room. As shown, the remote operation assembly 12 includes a single dashboard 64; however, the remote operation assembly can include multiple dashboards 64. Although the dashboard 64 shown is located on the directional platform 53, it is expected that the dashboard 64 can be located elsewhere and can be separated from the remote operation assembly 12. The dashboard 64 can be positioned so that its positioning remains constant or relatively constant during surgery. In other words, the dashboard 64 can be located on a portion of the remote operation assembly 12 that does not move or undergoes little movement during surgery. As viewed by the user, the content on the dashboard 64 can be arranged in the same spatial order as the arm.

[0050] As discussed above with reference to displays 62a-d, during surgery, instrument panel 64 can be located at approximately eye level or above eye level. Similar to the discussion above with reference to displays 62a-d, instrument panel 64 can be sized for positioning on orientation platform 53 and can be sized to be accessible around or through a sterile drape. Instrument panel 64 can be larger than display 62. Thus, instrument panel 64 can be approximately square or rectangular in shape and have a size between about 5 inches and about 15 inches on a side.

[0051] In various embodiments, the dashboard 64 is integrated into the teleoperation assembly 12 and is in wired communication with other components of the teleoperation assembly 12, such as the control system 20. In other embodiments, the dashboard 64 can be surface-attached to the arm of the teleoperation assembly 12 and can communicate wirelessly with components of the teleoperation assembly 12. The dashboard 64 can support various wireless protocols, including Bluetooth, IrDA, HomeRF, IEEE 802.11, DECT, and wireless telemetry. The dashboard 64 is capable of communicating with accessory systems 26 including cameras and other displays. The dashboard 64 can display images from the following: the endoscopic imaging system 15, other cameras in the operating room or elsewhere, the display 62 (and vice versa), other displays in the operating room or elsewhere, or a combination thereof.

[0052] In one embodiment, the dashboard 64 is configured to display the status of each of the arms 54 and / or displays 62a-d. The dashboard 64 can display the status of the arms 54 simultaneously, or can display them in a loop. The loop can display the status of one arm 54 at a time, or display the status of multiple arms 54 at a time. When displaying the status of multiple arms 54, the loop can cause the status of one arm 54 displayed at a time to be removed from the screen and replaced by the status of another arm 54, or the status of multiple arms 54 is removed and replaced at one time. In order to display the status of multiple arms 54, the screen of the dashboard 64 can be divided into multiple sections so that one section displays one status. These sections can be divided by vertical, horizontal or both partitions. A clear spatial association with the arm minimizes the possibility that the user will misassociate the status / prompt with the wrong manipulator or instrument.

[0053] As described above, the teleoperation assembly 12 can monitor the status of its various components. If the teleoperation assembly 12 finds a problem with one or more arms 54, the medical device system 14, the endoscopic imaging system 15, and / or other components, or a combination thereof, the teleoperation assembly 12 can operate the dashboard 64 to display information configured to help solve the problem. The dashboard 64 can display a warning image, a diagnosis of the problem, a suggested solution to the problem, written instructions for identifying and / or solving the problem, an animation describing the location and / or solution of the problem, or some combination thereof.

[0054] In addition to identifying problems with the arm 54, the dashboard 64 and / or arms 62a-d can display other information about the status of the arm 54. As the remote operation assembly 12 continuously monitors the status of the arm 54, the dashboard 64 and / or arms 62a-d can be dynamically updated. The dashboard 64 and / or arms 62a-d can display the following information: which tools or instruments are located on the arm 54, the number of times the tools or instruments have been used, the expiration date of the tools or instruments, other information, or a combination thereof. The dashboard 64 and / or arms 62a-d can display an image of the tool or instrument, an animation of the tool or instrument, a description of the tool or instrument, a usage diagram, a usage timeline, other images, or a combination thereof. The dashboard 64 can also provide higher-level system status and prompts related to the operation of other commonly connected components of the directional platform or manipulator 12.

[0055] Dashboard 64 can be used as an interface that allows clinicians to issue instructions to remote operation assembly 12. In one embodiment, dashboard 64 can be characterized by a capacitive or resistive touch screen and can include a graphical user interface (GUI). In some cases, a resistive touch screen may be required. For example, a resistive touch screen can allow clinicians to interact with the touch screen by using a stylus or other instrument without having to touch the screen directly, thereby ensuring sterility, and direct contact with the screen can destroy the sterility of the clinician's hands. Dashboard 64 can provide clinicians with the same interactive functions provided by device 62. Dashboard 64 can be configured to allow clinicians to issue instructions to any and all arms in arm 54, rather than just one in the case of display 62.

[0056] Figure 1Cis a perspective view of an embodiment of an operator input system 16, which may be referred to as a surgeon's console. The operator input system 16 includes a left eye display 32 and a right eye display 34 for presenting to the surgeon S a coordinated stereoscopic view of the surgical environment that allows depth perception. The operator input system 16 also includes one or more input control devices 36, which in turn enable the remote operation assembly 12 to manipulate one or more instruments in the endoscopic imaging system 15 and / or the medical instrument system 14. The input control device 36 can provide the same degree of freedom as the instrument to which it is associated, so as to provide the surgeon S with a remote presentation or feeling that the input control device 36 is integral with the instrument, so that the surgeon has a strong sense of direct control of the instrument. To this end, position, force and tactile feedback sensors (not shown) can be used to transmit the position, force and tactile sensation from the medical instrument, such as the surgical tool 30a-c or the imaging device 28, back to the surgeon's hand through the input control device 36. The input control device 37 is a foot pedal that receives input from the user's foot. Various aspects of the operator input system 16 , teleoperated assembly 12 , and accessory systems 26 may be adjustable and customizable to meet the surgeon's S physical needs, skill level, or preferences.

[0057] Figure 2 A method 200 for providing information in a surgical environment (e.g., surgical environment 11) is described. Figure 2 The method 200 is described as a set of operations or processes 202 to 206. In all embodiments of the method 200, not all of the described processes 202 to 206 may be performed. In addition, Figure 2 One or more processes not explicitly described in the drawings may be included before, after, between, or as part of processes 202 to 206. In some embodiments, one or more of the processes may be implemented at least in part in the form of executable code stored on a non-transitory tangible machine-readable medium, which, when executed by one or more processors (e.g., processors of a control system), may cause the one or more processors to perform one or more of the processes.

[0058] In process 202, the position of a display device (e.g., display 62a) is monitored or otherwise known in a surgical environment. The position of the display device can be determined in association with or relative to a remotely operated arm. For example, the display device 62 can be mounted to arm 54-1 and fixed relative to arm 54-1 so that a known kinematic position of arm 54-1 provides a known position of the display device. If arm 54-1 moves, for example, during a setup procedure, the monitored change in the kinematic position of arm 54-1 is used to determine the changed position of the display device. The position of arm 54-1 may alternatively be determined by other types of sensors, including electromagnetic positioning sensors or optical sensors. Alternatively, the position of the display device may be monitored by independently tracking the display device using, for example, an electromagnetic position sensor or an optical sensor.

[0059] In process 204, an image is presented on a first display device based on a known or monitored position of the display device in a surgical environment. If the monitored position of the display device is on a teleoperated arm, the image can be associated with the teleoperated arm or an instrument attached to the teleoperated arm. For example, if the teleoperated component 12 finds that one or more of the arms 54, the medical device system 14, the endoscopic imaging system 15, and / or other components have a problem, the teleoperated component 12 can operate the display 62a-d located on the arm 54 that encounters the problem to display information configured to help solve the problem. The display 62a-d can display a warning image, a diagnosis of the problem, a suggested solution to the problem, written instructions for identifying and / or solving the problem, an animation depicting the location and / or solution of the problem, or some combination thereof. The image can display an animation or instruction that reflects the current posture of the teleoperated arm viewed from a clinician's perspective or a commonly used perspective (e.g., from the front of the component 12) so that minimal user interpretation is required to understand the image.

[0060] For example, in the case where the surgical tool 30a has failed, the remote operation assembly 12 can operate the display 62a on the arm 54-1 to display a message indicating that the surgical tool 30a has failed and a replacement should be installed. Providing alarms and instructions on the display 62a located on the arm 54-1 can increase the possibility that the operating room clinician and / or maintenance personnel will correctly identify the device to be replaced and speed up the resolution process. As another example, if a collision between two arms 54-1, 54-2 occurs or is expected by the control system 20, the display devices 62a and 62b can display guidance images. The images displayed on the display devices 62a and 62b can be different. For example, the image on the device 62a can provide instructions for moving the arm 54-1 to prevent collisions, and the image on the device 62b can provide different instructions for moving the arm 54-2 to prevent collisions. In some cases, it may be advantageous to exchange instruments between arms to resolve collisions. An animation depicting the exchange of instruments between arms can be simultaneously displayed on adjacent or non-adjacent arm displays to illustrate the recommended replacement process. As another example, control signals between an operator input system and a medical device system can be monitored to determine whether the device is currently grasping tissue or applying force to tissue above a predetermined threshold level of force. If the device is currently grasping tissue, this status can be displayed on the corresponding arm display device. This can help improve troubleshooting and prevent tissue damage when bedside staff corrects arm collisions or separations of the device.

[0061] The monitored position of the display device can additionally or alternatively provide the position of the display device in the surgical environment. The position of the display device can be compared with the positions of other tracked personnel or devices in the surgical environment. Therefore, a display device within a predetermined range of the tracked clinician can be selected to display guidance information for the tracked clinician. For example, the nearest display device can be used to provide training content based on the skill level or experience of the tracked clinician. The displayed image can be presented from the vantage point of the tracked clinician.

[0062] In optional process 206, the image on the display device changes based on the changed condition of the remotely operated arm to which the display device is attached. For example, the default image on the display device may be the arm or instrument status. The changeable condition may be an error state, an instrument failure state, a collision state, a position, or other conditions related to the state of the remotely operated arm or the attached instrument. For example, during a manual remotely operated arm setup procedure, the image on the display device 62a may change as the position of the arm 54-1 is adjusted to provide the clinician with real-time guidance on adjusting the arm. The displayed image may depict the current posture of the arm and show how to manually reposition the arm. In addition, the changeable condition may indicate arm activity and progress, such as the progress of clamping or firing of the stapler. The changeable condition may also be related to a cable connection, such as sensing that the electrocautery cable is not present.

[0063] In addition to identifying problems with the arm 54, the display 62 may also display other information about the status of the arm 54. As the teleoperation assembly 12 continuously monitors the status of the arm 54, the display 62 may be dynamically updated. The display 62 may display information about which tools or instruments are located on the arm 54, the number of times the tool or instrument has been used, the expiration date of the tool or instrument, other information, or a combination thereof. The display 62 may display an image of the tool or instrument, an animation of the tool or instrument, a description of the tool or instrument, a usage diagram, a usage timeline, other images, or a combination thereof.

[0064] The display 62 can be used as an input interface to allow the clinician to issue instructions to the remote operation assembly 12. In various embodiments, the display 62 can feature a capacitive or resistive touch screen and can include a graphical user interface (GUI). In some cases, a resistive touch screen may be desired. For example, a resistive touch screen can help maintain sterility by allowing the clinician to interact with the touch screen using a stylus or other instrument without having to directly touch the screen, which would compromise the sterility of the clinician's hands.

[0065] Depending on the tool being used with the arm on which the display 62 is mounted, the options available to the clinician interacting with the display 62 may be variable. For example, when the tool is a stapler, the clinician may be able to view the type of stapler recharge that has been installed, view the clamping status of the instrument, view a maintenance report, view the general status of the stapler, and / or order a recharge of the stapler, etc. In contrast, when the tool is an endoscope, the clinician may be able to view images captured by the endoscope, adjust the zoom level of the endoscope, adjust the viewing orientation (e.g., angled up or down), order a snapshot to be taken, view a maintenance report, and / or view the status of the endoscope, etc.

[0066] In various embodiments, the display can be portable within the surgical environment. For example, the display can be a tablet computer carried by a visiting clinician. The portable display can provide context-sensitive troubleshooting guidance that can provide multiple levels of assistance. The assistance can include visual / animated content depending on the state of the remote operating system, a searchable electronic user manual, or messaging or two-way video calling. The display can also provide a barcode scanner for scanning medical instruments or instruments to receive additional information.

[0067] Figure 3 Another method 300 for providing information in a surgical environment according to an embodiment of the present disclosure is described. The method 300 describes using a projector (e.g., projector 60) to provide visual assistance to a clinician in a surgical environment. The method 300 is described in Figure 3 300 as a set of operations or processes 302 to 312. In all embodiments of method 300, not all of the illustrated processes 302 to 312 may be performed. In addition, Figure 3 One or more processes not explicitly described in the process 302 to 312 may be included before, after, between, or as part of the process 302 to 312. In some embodiments, one or more of the processes may be implemented (at least in part) in the form of executable code stored on a non-transitory tangible machine-readable medium, which, when executed by one or more processors (e.g., a processor of the control system), may cause the one or more processors to perform one or more of the processes. Prior to process 302, the system may identify the docking state and control state of the remote operation assembly 12 so that any displayed information is appropriate for the current docking state and control state.

[0068] At process 302, sensor information is received from a sensor (e.g., sensor 61) of a remote operation system. At process 304, a first visual aid is determined based on the sensor information. At process 306, a visual projection device (e.g., projector 60) is operated to project the visual aid into the surgical environment. Figure 4Further illustrating process 302-206, the figure illustrates a surgical environment 400, which includes: a teleoperation assembly 402, which can be substantially similar to assembly 12; and a projector 404, which can be substantially similar to projector 60. Projector 404 is coupled to an orientation platform 406, and a teleoperation arm 408 is coupled to orientation platform 406. A depth sensor 410 (e.g., sensor 61) measures a distance D from the sensor down into the working area to an obstacle. In order to determine a suitable visual aid for projection, the distance D is compared to a predetermined value associated with the height H of the operating table 412. If the distance D is approximately the same as or greater than the predetermined value as the known height of the teleoperation assembly, the sensor information indicates that there is no patient or operating table in the working area. Based on the absence of a patient in the working area, a directional visual aid 414 such as an arrow is projected from projector 404. The arrow is used during the initial approach to the patient and the operating table to confirm the location of the center of the orientation platform 406. The direction of the arrow can provide a direction for transporting the orientation platform to the working area. The orientation of the arrow may be determined to be aligned with the base of teleoperation assembly 402. The orientation of the arrow may be independent of the current direction of the orienteering platform.

[0069] Optionally, the teleoperation assembly 12 can communicate with the operating table 412 in a wired or wireless manner so that the teleoperation assembly 12 can determine its position relative to the operating table. For example, the operating table and / or the teleoperation assembly 12 can include a wireless tracker, such as a global positioning system (GPS) tracker. The teleoperation assembly 12 can receive ongoing positioning updates from the operating table. Positioning updates can be sent continuously, approximately every half second, approximately every second, approximately every three seconds, approximately every five seconds, approximately every 10 seconds in response to changes in the position of the teleoperation assembly 12, in response to a request, in response to a user instruction, at certain intervals, or in response to other stimuli.

[0070] Visual aid 414 can be projected downward onto the floor of the operating room or elsewhere. Aid 414 can be accompanied by an audio prompt, such as a tone, a beep, a buzz, a recording, other audio prompts, or a combination thereof. In an alternative embodiment, the visual aid can include a plurality of arrows aligned with the base of component 402. In an alternative embodiment, the size, direction, color, or other properties of the projected arrows can be adjusted in real time as updated positioning information is received from the operating table. In an alternative embodiment, the visual aid can include an image of footprints, shoe prints, written prompts, alphanumeric aids, colored lines, sidewalks, stepping stones, fingers, human outlines, animated images, or a combination thereof.

[0071] Reference again Figure 3At process 308, additional sensor information is received from the sensor. At process 310, a visual aid is determined based on the additional sensor information. At process 312, the visual projection device changes the visual aid of process 304 to the visual aid of process 310.

[0072] Reference Figure 5 Further explanation of process 308-312. When sensor 410 determines that the patient has entered the working area of ​​the teleoperation assembly or the teleoperation system has entered the directional platform positioning mode, directional visual aid 414 is replaced with visual aid 420 to assist in the next step of the setup process. In this embodiment, visual aid 420 is a directional platform positioning aid, which can be a circle projected onto the patient. The circle is used to position the directional platform 406 above the patient. The circle is resized by projector 404 to have a fixed radius, regardless of the distance between sensor 410 and the patient. The projection radius can be constant for the distance between the projector and the patient. The sensor can be used to determine the projection distance and then calculate the appropriate radius to be projected. The fixed radius can be based on the positioning tolerance of the directional platform, which is used to dock the teleoperation arm to the cannula positioned in the patient's incision. In an example embodiment, the circle can have a radius of approximately 3 inches. Various symbols can be used as directional platform positioning aids, among which the circle is particularly suitable because it does not imply unnecessary orientation directions during platform positioning. In addition to or in lieu of circles, the visual aid may include an image of a crosshair, an 'X', a target featuring concentric rings, a square, a rectangle, a smiling face, a human silhouette, an animated image, or a combination thereof. In an alternative embodiment, an optical element such as a Fresnel lens may be used in front of the projector to achieve orthogonal projection so that the projected visual image does not change size as the height of the orienting platform changes.

[0073] The desired accuracy with which the orientation platform 406 is positioned prior to surgery may vary depending on the procedure to be performed and / or the patient's physical characteristics. As described above, the database 27 may include a list of patient profiles and a list of procedures to be performed on the patients. Thus, the teleoperation assembly 12 may determine the procedure to be performed on the patient currently on the operating table and determine the patient's physical characteristics based on the information contained in the database 27, and may adjust the visual aid based on one or the other or both of these determinations.

[0074] refer to Figure 6 Further explaining process 308-312, in Figure 6, the directional platform positioning aid 420 is replaced by the directional platform orientation aid 422. When the sensor 410 determines that the patient has been correctly positioned in the work area or the remote operation system has entered the directional platform orientation mode, the directional visual aid 414 is replaced by the visual aid 420 to assist in the next step of the setup process. In this embodiment, the directional platform orientation aid 422 is a linear arrow indicating the working direction of the remote operation arm. The aid 422 can minimize confusion about the primary working direction of the directional platform 406 and the arm 408. The arm 408 operates in a tilted forward orientation. The aid 422 is provided to guide the setup operator to position the directional platform and the arm so that the pitch of the arm is in the direction of the aid.

[0075] Figure 7 The orientation platform orientation aid 424 is illustrated, which in this embodiment is a curved arrow displayed when the orientation platform 406 is approaching the rotation limit. The range of motion of the orientation platform can be limited to + / -180 degrees of rotation. If the sensor information indicates that the user is trying to rotate the orientation platform beyond the 180 degree range, the aid 424 can be projected to warn the user that the orientation platform needs to be rotated in the opposite direction.

[0076] After satisfactory positioning and orientation of the teleoperated assembly, the surgeon can begin the planned surgery. The projector can provide one or more visual aids during surgery. The visual aids can be based on information contained in the database 27 and / or based on information received by the teleoperated assembly 12 during surgery, such as from the endoscopic imaging system 15. For example, depending on the procedure to be performed, the projector can project an image representing the incision site onto the patient or elsewhere. For another example, the projector can project a preoperative image of the internal anatomical structure onto the patient or elsewhere based on the information received from the endoscopic imaging system 15.

[0077] Figure 8Description of highlighted visual aid 426. When sensor information indicates that attention needs to be paid to a specific location in the work area, projector 404 projects visual aid 426 onto the location that needs attention. The content of visual aid 426 can be constant or modulated light, symbols, alphanumeric content, or other visual content to draw attention to the highlighted area. For example, visual aid 426 can appear on the patient, on the instrument, on the arm, at the location where the arm collides, or at other locations in the work area. In one example, the visual aid can be a light spot for highlighting the location of interference or collision between one or more arms. Alternatively, the visual aid may include a depiction of arms in contact with each other, a written warning of contact, other images, or a combination thereof. In another example, the visual aid can be information about the error state of the highlighted instrument. In another embodiment, the color or content of the visual aid can change when the arm is manually moved toward the optimal posture. For example, when the arm is moving toward the appropriate position for surgery, the projector can generate a visual aid indicating that the arm is getting closer and closer to the appropriate position. The visual aid can be a light projected onto the moving arm, so that as the arm 54 gets closer to the appropriate position, the light becomes greener and greener, and as the arm moves away from the appropriate position, the light becomes redder and redder. Any other colors may be used in addition or alternatively. When the arm reaches the correct position, a strobe, light spot or other cue may be produced.

[0078] The remote operation assembly 12 can be configured to monitor the condition of its various components (including the medical device system 14, the endoscopic imaging system 15 and the arm 54) and identify maintenance issues. The projector can generate one or more visual aids to help solve such problems. For example, a maintenance problem that may be encountered is the failure or failure of one of the tools such as surgical instruments. The projector can highlight the failed or failed tool as described above and / or can project an image of the identified failed or failed tool onto the patient or elsewhere. The image identifying the failed or failed tool can include a description of the failed or failed tool, a description of the arm in which the failed or failed tool is located, a written warning of the failure or failure, a light spot, an animated image, other images, or a combination thereof. The projected light spot can also highlight a portion of the instrument housing into which the clinician will need to insert an accessory to manually open the instrument jaws before disassembly.

[0079] Depending on the experience level of the clinicians in the operating room, the visual aids generated by the projector may vary. For example, when the experience level of the clinicians in the operating room is low, additional or more detailed visual aids may be used. In one embodiment, for the purpose of visual aid determination, the experience level of the clinicians in the operating room may be limited to the experience level of the least experienced clinician. Alternatively, for the purpose of visual aid determination, the experience level of the clinicians in the operating room may be the average experience level or the experience level of the most experienced clinician. In some cases, the surgeon may be exempt from the calculation of the experience level.

[0080] In various embodiments, images and information displayed to the user may provide safety-related guidance. For example, the information may guide the user through solutions in situations including power failures. A display mounted on the manipulator may include battery backup and high availability isolation to provide instructions for safely withdrawing the device from patient tissue in the event of a power outage or unrecoverable system failure. In another example, if a manipulator arm becomes inoperable, the manipulator mounted display may provide information to properly position the arm in a manner that does not interfere with other arms or other components of the remotely operated assembly.

[0081] In various embodiments, the images and information displayed to the user can provide information about system interruptions related to failed tools, invalid tools, and energy instrument cable connection status. In various embodiments, the images and information displayed to the user can provide information about instrument type, remaining service life on the instrument, endoscope status, manipulator arm status (e.g., in progress, waiting for input), instrument status (e.g., grasping, stapler clamping, busy status), dual console and single site clarity (e.g., an illustration that associates each instrument with a surgeon console, an illustration of left / right hand association), manipulator digital identifier, undocking manipulator arm, management and collision avoidance, and proper robot arm loading guidance. In various embodiments, the images and information displayed to the user can provide a tutorial. Such a tutorial can be provided in response to a user request for help, or such a tutorial can be provided in a training mode of the system. In various embodiments, the images and information displayed to the user can optionally provide optimization information about, for example, bending positioning or patient clearance. Other custom information can also be displayed.

[0082] One or more elements in the embodiments of the present invention may be implemented in software to be executed on a processor of a computer system such as a control processing system. When implemented in software, the elements of the embodiments of the present invention are essentially code segments for performing necessary tasks. The program or code segment may be stored in a processor-readable storage medium or device, which may be downloaded via a transmission medium or communication link as a computer data signal embodied in a carrier wave. The processor-readable storage device may include any medium that can store information, including optical media, semiconductor media, and magnetic media. Examples of processor-readable storage devices include electronic circuits; semiconductor devices, semiconductor storage devices, read-only memories (ROMs), flash memories, erasable programmable read-only memories (EPROMs); floppy disks, CD-ROMs, optical disks, hard disks, or other storage devices. The code segments may be downloaded via a computer network (e.g., the Internet, an Intranet, etc.).

[0083] Note that the processes and displays presented may not be inherently related to any particular computer or other device. Various general purpose systems may be used with programs according to the teachings herein, or it may prove convenient to construct more specialized devices to perform the described operations. The structures required for various of these systems will appear as elements in the claims. In addition, embodiments of the present invention are described without reference to any particular programming language. It should be understood that various programming languages ​​may be used to implement the teachings of the present invention as described herein.

[0084] Although certain exemplary embodiments of the present invention have been described and illustrated in the drawings, it should be understood that such embodiments are merely illustrative of the invention and not limiting of the invention, and that the embodiments of the invention are not limited to the specific constructions and arrangements shown and described, since various other modifications may be made by those skilled in the art.

Claims

1. A remote operating system in a surgical environment, comprising: a teleoperation assembly comprising a first teleoperation arm; a first display device coupled to the first remotely operated arm; and One or more processors configured to: determining a position of the first display device in the surgical environment; and presenting a first image on the first display device, wherein the first image is presented based on the determined position of the first display device in the surgical environment during manual positioning of the first teleoperated arm in the surgical environment, wherein the first image comprises: a representation of a pose of the first teleoperated arm during the manual positioning of the first teleoperated arm in the surgical environment; and instructions for repositioning said first telemanipulator arm during said manual positioning, Wherein the instructions for repositioning include instructions for moving the first teleoperated arm to avoid collision with a second teleoperated arm.

2. The remote operation system according to claim 1, wherein: The one or more processors are further configured to: determining a change in a condition associated with the first remotely operated arm; and A second image is presented on the first display device based on the changed condition.

3. The remote operation system of claim 1, further comprising a second display device coupled to the second remote operation arm; wherein the one or more processors are further configured to present a second image on the second display device, wherein the second image includes a representation of a pose of the second teleoperated arm during manual positioning of the second teleoperated arm in the surgical environment, wherein, during the manual positioning of the second teleoperated arm in the surgical environment, presenting the second image based on the position of the second display device in the surgical environment; and Wherein the first image and the second image are different.

4. The remote operation system according to claim 1, wherein: The one or more processors are configured to determine a position of an operator in the surgical environment, wherein presenting the first image is further based on whether the position of the operator is within a predetermined range of the determined position of the first display device.

5. The remote operation system according to claim 4, wherein: The first image includes training content for the operator.

6. The remote operation system according to claim 1, wherein: The position of the first display device is determined relative to a position of the first remotely operated arm.

7. The remote operation system according to claim 1, wherein: The first display device is integrally formed with the first remote operating arm.

8. The remote operation system according to claim 1, wherein: The first display device is removable from the first remote operating arm.

9. The teleoperation system of claim 1, wherein a first instrument is coupled to the first teleoperation arm and a second instrument is coupled to the second teleoperation arm, and wherein, The first image includes instructions for exchanging the first instrument and the second instrument between the first teleoperated arm and the second teleoperated arm.

10. A non-transitory computer readable medium storing instructions that when executed by one or more processors cause the one or more processors to perform a method of operating a remote operating system, the method comprising: determining a position within the surgical environment of a first display device of a first teleoperation arm coupled to a teleoperation assembly of the teleoperation system; and presenting a first image on the first display device, wherein the first image is presented based on the determined position of the first display device in the surgical environment during manual positioning of the first teleoperated arm in the surgical environment, wherein the first image comprises: a representation of a pose of the first teleoperated arm during the manual positioning of the first teleoperated arm in the surgical environment; and Instructions for repositioning the first teleoperated arm during the manual positioning, wherein the instructions for repositioning include instructions for moving the first teleoperated arm to avoid collision with a second teleoperated arm.

11. The non-transitory computer readable medium of claim 10, wherein the method further comprises: determining a change in a condition associated with the first remotely operated arm; and A second image is presented on the first display device based on the changed condition.

12. The non-transitory computer readable medium of claim 10, wherein the method further comprises: A second image different from the first image is presented on a second display device of the second teleoperated arm connected to the teleoperated assembly, wherein the second image includes a representation of a posture of the second teleoperated arm during manual positioning of the second teleoperated arm in the surgical environment, and wherein the second image is presented based on a position of the second display device in the surgical environment during the manual positioning of the second teleoperated arm in the surgical environment.

13. The non-transitory computer readable medium of claim 10, wherein the method further comprises: An operator's position within the surgical environment is determined, wherein the first image is presented further based on whether the position of the operator is within a predetermined range of the determined position of the first display device.

14. The non-transitory computer readable medium of claim 13, wherein: The first image includes training content for the operator.

15. The non-transitory computer readable medium of claim 13, wherein: The first image includes an image of the teleoperation system from the operator's vantage point.

Images