Systems and methods for onscreen identification of instruments in teleoperational medical system

Abstract

To provide a system comprising a teleoperational assembly which includes an operator control system and a plurality of manipulators configured to control the movement of medical instruments in a surgical environment, and a processing unit.SOLUTION: The manipulators are teleoperationally controlled by the operator control system. The system is further configured to display an image 202 of a field of view of a surgical environment 200, determine association information about the manipulators and the operator control system, and display badges near medical instruments 26a, 26b, 26c in the image of the field of view of the surgical environment. Badges 206 and 207 include the processing unit configured to display the association information for the medical instrument in which they appear associated with each other.SELECTED DRAWING: Figure 2A

Description

[Technical field]

[0001] (Related Applications) This patent application claims the benefit of priority and the filing date of U.S. Provisional Patent Application No. 62 / 134,297, filed March 17, 2015, entitled “SYSTEMS AND METHODS FOR RENDERING ONSCREEN IDENTIFICATION OF INSTRUMENTS IN A TELEOPERATIONAL MEDICAL SYSTEM,” which is incorporated by reference in its entirety.

[0002] FIELD OF THE DISCLOSURE The present disclosure relates to systems and methods for performing remotely operated medical procedures, and more particularly, to systems and methods for displaying information regarding remotely operated instruments used in a surgical environment. [Background technology]

[0003] Minimally invasive medical techniques are intended to reduce the amount of tissue damaged during invasive medical procedures, thereby reducing patient recovery time, discomfort, and adverse side effects. Such minimally invasive medical techniques can be performed through natural openings or one or more surgical incisions in the patient's anatomy. Through these natural openings or incisions, clinicians can insert medical instruments to reach target tissue locations. Minimally invasive medical tools include instruments such as therapeutic, diagnostic, and surgical instruments. Minimally invasive medical tools may also include imaging instruments such as endoscopic instruments. Imaging instruments provide the user with a field of view within the patient's anatomy. Some minimally invasive medical tools and imaging instruments can be remotely operated or computer-assisted. In teleoperated medical systems, it can be difficult to see the instruments, whether due to the small size of the instruments, their proximity to the edge of the field of view, or due to obstruction by tissue. Clinicians operating teleoperated systems can have difficulty keeping track of which instruments are under which operator control. What is needed is a system and method for providing a clinician with information regarding instruments under his or her control to reduce the risk of clinician error. Summary of the Invention

[0004] Embodiments of the invention are summarized in the claims that follow.

[0005] In one embodiment, the teleoperation assembly includes an operator control system and a plurality of manipulators configured to control movement of medical instruments in a surgical environment. The manipulators are remotely controlled by the operator control system. The system further includes a processing unit configured to display an image of the field of view of the surgical environment, determine relevant information related to the manipulators and the operator control system, and display badges in the image of the field of view of the surgical environment near the medical instruments. The badges display relevant information for the medical instruments with which they appear associated.

[0006] In another embodiment, an image of a field of view of the surgical environment is displayed. A location of a medical instrument within the environment is determined, relevant information regarding the medical instrument and an operator control system is determined, and a badge is displayed on the image of the field of view of the surgical environment near the medical instrument. The medical instrument is attached to a manipulator that is remotely controlled by the operator control system. The badge displays relevant information regarding the medical instrument and is displayed near the medical instrument.

[0007] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory in nature and are intended to provide an understanding of the present disclosure without limiting the scope of the disclosure. In this regard, additional aspects, features, and advantages of the present disclosure will become apparent to those skilled in the art from the following detailed description. [Brief description of the drawings]

[0008] Aspects of the present disclosure are best understood from the following detailed description when read in conjunction with the accompanying drawings. It is emphasized that, according to standard industry practice, various features are not drawn to scale. In fact, dimensions of various features may be arbitrarily increased or decreased for clarity of illustration. In addition, the present disclosure may repeat reference numerals and / or letters in various instances. This repetition is for the purposes of brevity and clarity and does not in itself dictate a relationship between the various embodiments and / or configurations discussed.

[0009] [Figure 1A] 1 is a schematic diagram of a remotely operated medical system according to an embodiment of the present disclosure. [Figure 1B] FIG. 1 is a perspective view of a surgeon's control console for a teleoperated medical system, according to many embodiments. [Figure 1C] FIG. 1 illustrates a perspective view of a teleoperated medical system electronics cart, according to many embodiments. [Figure 1D] FIG. 1 is a perspective view of a patient side cart according to one example of principles described herein. [Figure 2A]A view of a surgical workspace is shown with at least one medical instrument visible within the view, the instrument having a badge associated therewith and including pertinent information relating to the instrument. [Figure 2B] 1 is a flow chart illustrating a method for operating a remote medical system. [Figure 3A] 3D illustrates the input control devices and user hand positions corresponding to the related information badge arrangement shown in FIG. 3D. [Figure 3B] 3E shows the input control devices and user hand positions corresponding to the related information badge arrangement shown in FIG. 3E. [Figure 3C] 3C illustrates the input controller and user hand positions corresponding to the related information badge arrangement shown in FIG. 3F. [Figure 3D] 1 illustrates various arrangements of related information badges on an orbital segment. [Figure 3E] 1 illustrates various arrangements of related information badges on an orbital segment. [Figure 3F] 1 illustrates various arrangements of related information badges on an orbital segment. [Figure 4A] 13 illustrates the resize behavior of related information badges as the associated instrument moves towards and away from the imaging device. [Figure 4B] 13 illustrates the resize behavior of related information badges as the associated instrument moves towards and away from the imaging device. [Figure 5A] 13 illustrates how a badge of related information remains visible when the associated instrument is obscured from view. [Figure 5B] 13 illustrates how a badge of related information remains visible when the associated instrument is obscured from view. [Figure 6A] 13 illustrates the adjustment of badges of relevant information when a medical device is in close proximity. [Figure 6B] 13 illustrates the adjustment of badges of relevant information when a medical device is in close proximity. [Figure 7A]The behavior of related information badges is shown as the associated medical instrument approaches an edge of the image of the surgical environment and as the medical instrument moves outside of the image of the surgical environment. [Figure 7B] The behavior of related information badges is shown as the associated medical instrument approaches an edge of the image of the surgical environment and as the medical instrument moves outside of the image of the surgical environment. [Figure 7C] The behavior of related information badges is shown as the associated medical instrument approaches an edge of the image of the surgical environment and as the medical instrument moves outside of the image of the surgical environment. [Figure 8] 13 illustrates how to render a badge within an image of a surgical workspace field of view. [Figure 9] Detailing the process of FIG. 8, the location of the medical instrument distal tip portion is projected into the coordinate space of an image of the field of view of the surgical workspace. [Figure 10] The process of Figure 8 is detailed below, where potential badge overlaps are checked and resolved. In this case, there are at least two medical instruments. [Figure 11] FIG. 8 shows a more detailed view of the process in which the badge is constrained to the visible space of the image in the field of view of the surgical workspace. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] For purposes of promoting an understanding of the principles of the present disclosure, reference will now be made to the embodiments illustrated in the drawings, and specific language will be used to describe the same. It will nevertheless be understood that no limitation of the scope of the present disclosure is intended. In the following detailed description of the embodiments of the present invention, numerous specific details are set forth in order to provide a comprehensive understanding of the disclosed embodiments. However, it will be apparent to one skilled in the art that the embodiments of the present disclosure may be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments of the present invention.

[0011] Any alternatives and further modifications to the described devices, apparatuses, and methods, as well as any further applications of the principles of the present disclosure, are fully contemplated, as would normally occur to one skilled in the art to which the present disclosure pertains. In particular, it is fully contemplated that features, components, and / or steps described for one embodiment may be combined with features, components, and / or steps described for other embodiments of the present disclosure. In addition, it is contemplated that dimensions provided herein are for specific examples, and different sizes, dimensions, and / or ratios may be used to realize the concepts of the present disclosure. In order to avoid unnecessary repetition of description, one or more components or operations described according to one illustrative embodiment may be used or omitted as applicable to other illustrative embodiments. For the sake of brevity, many iterations of these combinations will not be described separately. For the sake of simplicity, in some examples, the same reference numerals are used throughout the drawings to indicate the same or similar parts.

[0012] The following embodiments describe various instruments and instrument parts in terms of states in three-dimensional space. As used herein, the term "position" refers to the location of an object or part of an object in three-dimensional space (e.g., three translational degrees of freedom along Cartesian X, Y, Z coordinates). As used herein, the term "orientation" refers to the rotational configuration of an object or part of an object (three rotational degrees of freedom - e.g., roll, pitch, and yaw). As used herein, the term "pose" refers to the position of an object or part of an object in at least one translational degree of freedom and the orientation of an object or part of an object in at least one rotational degree of freedom (up to a total of six degrees of freedom). As used herein, the term "shape" refers to a set of poses, positions, or orientations measured along an object.

[0013] Referring to FIG. 1 of the drawings, a teleoperated medical system for use in a medical procedure, including, for example, a diagnostic, therapeutic, or surgical procedure, is generally indicated by the reference numeral 10. As described, the teleoperated medical system of the present disclosure is under the remote control of a surgeon. In alternative embodiments, the teleoperated medical system may be under partial control of a computer programmed to perform the procedure or sub-procedure. In yet other alternative embodiments, a fully automated medical system under full control of a computer programmed to perform the procedure or sub-procedure may be used to perform the procedure or sub-procedure. As shown in FIG. 1A, the teleoperated medical system 10 generally includes a teleoperated assembly 12 that is mounted to or near an operating table O on which a patient P is located. The teleoperated assembly 12 may also be referred to as a patient-side cart. A medical instrument system 14 and an endoscopic imaging system 15 are operably coupled to the teleoperated assembly 12. An operator input system 16 allows a surgeon or other type of clinician S to view an image of the surgical site or an image representative of the surgical site and to control the operation of the medical instrument system 14 and / or the endoscopic imaging system 15.

[0014] The operator input system 16 may be located at the surgeon's console, which is typically located in the same room as the operating table O. However, it should be understood that the surgeon S may be located in a different room than the patient P, or in a different building entirely. The operator input system 16 generally includes one or more input control devices for controlling the medical instrument system 14. The input control device(s) may include one or more of any number of different devices, such as hand grips, joysticks, trackballs, data gloves, trigger guns, hand-operated controllers, voice recognition devices, touch screens, pedals, body motion or presence sensors, eye tracking devices, and the like. In some embodiments, the control device(s) have the same degrees of freedom as the medical instruments of the teleoperated assembly to provide the surgeon with telepresence, a perception that the control device(s) are integrated with the instruments such that the surgeon has a strong sense of directly controlling the instruments as if they were present at the surgical site. In other embodiments, the control device(s) may have more or fewer degrees of freedom than the associated medical instruments and still provide telepresence to the surgeon. In some embodiments, the control device(s) is a manual input device that moves in six degrees of freedom and may also include an actuatable handle for actuating the instrument (e.g., to close the grasping jaws, apply an electrical potential to the electrodes, deliver a drug therapy, etc.).

[0015] The teleoperated assembly 12 supports and manipulates the medical instrument systems 14 while the surgeon S views the surgical site through a console 16. Images of the surgical site can be acquired by an endoscopic imaging system 15, such as a stereo endoscope, which can be manipulated by the teleoperated assembly 12 to orient the endoscope 15. Optionally, an electronics cart 18 can be used to process images of the surgical site for later display to the surgeon S through the surgeon's console 16. The number of medical instrument systems 14 used at one time generally depends on the diagnostic or surgical procedure and the spatial constraints within the operating room, among others. The teleoperated assembly 12 can include one or more non-servo controlled links (e.g., one or more links that can be manually positioned and locked in place, commonly referred to as a setup structure) and a teleoperated manipulator. The teleoperated assembly 12 includes multiple motors that drive inputs on the medical instrument systems 14. These motors move in response to commands from a control system (e.g., control system 20). The motor comprises a drive system that, when coupled to the medical instrument system 14, can advance the medical instrument into an orifice in a natural or surgically created anatomical structure. Other motor drive systems can move the distal end of the medical instrument with multiple degrees of freedom, which can include three degrees of linear motion (e.g., linear motion along X, Y, and Z Cartesian coordinate axes) and three degrees of rotational motion (e.g., rotation about X, Y, and Z Cartesian coordinate axes). Additionally, the motor can be used to actuate an articulatable end effector of an instrument for grasping tissue with the jaws of a biopsy device, etc.

[0016] The teleoperated medical system 10 also includes a control system 20. The control system 20 includes at least one processor (not shown), and typically multiple processors and at least one memory, for controlling between the medical instrument system 14, the operator input system 16, and the electronics system 18. The control system 20 also includes programmed instructions (e.g., a computer-readable medium storing instructions) that implement some or all of the methods described in accordance with the embodiments disclosed herein. Although the control system 20 is shown as a single block in the simplified schematic diagram of FIG. 1A, the system may include two or more data processing circuits, where some of the processing is optionally performed at or adjacent to the teleoperated assembly 12 and other parts of the processing are performed at the operator input system 16, etc. Any of a wide variety of centralized or distributed data processing architectures may be used. Similarly, the programmed instructions may be implemented as multiple separate programs or subroutines, or they may be incorporated into many other embodiments of the teleoperated system described herein. In one embodiment, the control system 20 supports wireless communication protocols such as Bluetooth, IrDA, Home RF, IEEE 802.11, DECT, and wireless telemetry.

[0017] In some embodiments, the control system 20 may include one or more servo controllers that receive force and / or torque feedback from the medical instrument system 14. In response to the feedback, the servo controllers send signals to the operator input system 16. The servo controller(s) may also send signals to command the teleoperated assembly 12 to move the medical instrument system(s) 14 and / or the endoscopic imaging system 15 extending through an opening in the patient's body and into an internal surgical site within the patient's body. Any suitable conventional or dedicated servo controller may be used. The servo controller may be separate from or integral with the teleoperated assembly 12. In some embodiments, the servo controller and teleoperated assembly are provided as part of a teleoperated arm cart that is positioned adjacent to the patient's body.

[0018] The teleoperated medical system 10 may further include optional operation and support systems (not shown), such as a lighting system, a steering control system, an irrigation system, and / or an aspiration system. In alternative embodiments, the teleoperated system may include multiple teleoperation assemblies and / or multiple operator input systems. The exact number of manipulator assemblies depends, among other things, on the surgical procedure and the spatial constraints within the operating room. The operator input systems may be collocated, or they may be located in separate locations. Multiple operator input systems allow multiple operators to control one or more manipulator assemblies in various combinations.

[0019] 1B is a perspective view of the surgeon's console 16. The surgeon's console 16 includes a left-eye display 32 and a right-eye display 34 for presenting the surgeon S with a coordinated stereo view of the surgical site that enables depth perception. The console 16 further includes one or more input controls 36, which allow the teleoperation assembly 12 to operate one or more instruments or an endoscopic imaging system. The input controls 36 may provide the same degrees of freedom as their associated instruments 14 to provide the surgeon S with telepresence, or the awareness that the input controls 36 are integral with the instruments 14 such that the surgeon has a strong sense of directly controlling the instruments 14. To this end, position, force, and tactile feedback sensors (not shown) may be used to communicate position, force, and tactile perceptions from the instruments 14 through the input controls 36 and back to the surgeon's hand.

[0020] FIG. 1C is a perspective view of an electronics cart 18. The electronics cart 18 can be coupled to the endoscope 15 and can include a processor that processes the captured images for later display to the surgeon on the surgeon's console or on other suitable displays located locally and / or remotely. For example, if a stereo endoscope is used, the electronics cart 18 can process the captured images to present the surgeon with coordinated stereo images of the surgical site. Such adjustments can include alignment between opposing images and can include adjusting the stereo working distance of the stereo endoscope. As another example, image processing can include the use of pre-determined camera calibration parameters to compensate for imaging errors of the image capture device, such as optical aberrations. The electronics cart 18 can also include a display monitor and components of a control system 20.

[0021] FIG. 1D is a perspective view of one embodiment of the teleoperated assembly 12, which may be referred to as a patient side cart. The patient side cart 12 shown provides for the operation of three surgical tools 26a, 26b, 26c (e.g., instrument system 14) and an imaging device 28 (e.g., endoscopic imaging system 15), such as a stereo endoscope, used for capture of images of the site of the procedure. The imaging device may send signals to the electronics cart 18 via cable 56. The operation is provided by a teleoperated mechanism having multiple joints. The imaging device 28 and surgical tools 26a, 26b, 26c can be positioned and manipulated through the incision in the patient such that a kinematic remote center is maintained at the incision to minimize the size of the incision. The image of the surgical site may include an image of the distal end of the surgical tool 26 when the distal end of the surgical tool 26 is located within the field of view of the imaging device 28. The teleoperated assembly 12 is based on a world coordinate system or frame F. w The distal tip of each medical instrument 26a, 26b, 26c is positioned within a respective instrument coordinate system or frame F. I The distal tip of the imaging device 28 defines a coordinate system or frame F E Determine.

[0022] The patient side cart 12 includes a drivable base 58. The drivable base 58 is connected to a telescoping column 57, which allows for adjustment of the height of the arms 54a, 54b, 54c, and 54d. The arm 54a may be known and / or labeled in the surgical environment as "Arm 1". The arm 54b may be known and / or labeled in the surgical environment as "Arm 2". The arm 54c may be known and / or labeled in the surgical environment as "Arm 3". The arm 54d may be known and / or labeled in the surgical environment as "Arm 4". The arm 54a may include a revolute joint 55 that both rotates and moves up and down. The arm 54a connects to a manipulator arm portion 51. The manipulator arm portion 51 may connect directly to the medical instrument 26a via a manipulator spar 59. Each of the other arms 54b, 54c, 54d may have a similar configuration to the arm 54a. Each of the arms 54a, 54b, 54c, 54d may be connected to an orienting platform 53. The orienting platform 53 may be capable of 360 degree rotation. The patient side cart 12 may also include a telescoping horizontal cantilever 52 that moves the orienting platform 53 horizontally. The manipulator arm portion 51 may be remotely operable. The arms 54a, 54b, 54c, 54d may or may not be remotely operable. In embodiments where the arm 54 is not remotely operable but is a manipulator arm, the arm 54 is positioned as desired before the surgeon begins surgery with the system 10. Although the arms 54 (and associated manipulator arm portion 51) are shown and described as part of a single patient side cart 12 for illustrative purposes, in various other embodiments the arms 54 and / or manipulator arm portion 51 (or additional arms 54 and / or manipulator arm portions 51) may be separate structures (e.g., separate table-mounted, ceiling-mounted, and / or floor-mounted arms).

[0023] Endoscopic imaging systems (e.g., systems 15, 28) may be provided in a variety of configurations, including rigid or flexible endoscopes. Rigid endoscopes include a rigid tube that houses a relay lens system for transmitting images from the distal end of the endoscope to the proximal end. Flexible endoscopes transmit images using one or more flexible optical fibers. Endoscopes may have different viewing angles, including a 0° viewing angle for forward axial viewing or viewing angles between 0° and 90° for forward oblique viewing. Digital image-based endoscopes have a "chip on the tip" design in which a distal digital sensor, such as one or more charge-coupled devices (CCDs) or complementary metal oxide semiconductor (CMOS) devices, stores image data. Endoscopic imaging systems may provide two-dimensional or three-dimensional images to the viewer. Two-dimensional images may provide limited depth perception. Three-dimensional stereoscopic endoscopic images may provide a more accurate depth perception to the viewer. Stereoscopic endoscopic instruments use stereo cameras to capture stereoscopic images of the patient's anatomy within the field of view of the cameras.

[0024] A stereoscopic image of the imaging system's field of view is viewable by the clinician through the eye displays 32, 34. In addition to the patient's tissue, the field of view may also include the distal end of the medical instrument and any accessories used to perform the surgical procedure. To perform this procedure, the clinician makes an association between the instrument in the field of view and the input control device that controls that instrument from the control console. The clinician may make this association, for example, by referencing alphanumeric text, symbols, or other information located on the periphery of the viewed image. However, to see this information, the clinician's focus must move away from the central portion of the display and the distal tip of the instrument to the periphery of the image. Alternatively, the clinician may make the association by moving the input control device and observing the movement of the corresponding instrument in the image. This can be time consuming and disruptive to the flow of the surgical procedure. To help the clinician associate the input control device 36 with the instrument seen in the field of view, information about the input control device may be presented in a graphical format and co-located with the instrument it controls. Co-locating the association information with the instrument allows the clinician to quickly associate the control device with the instrument while maintaining focus on the surgical area of ​​interest.

[0025] FIG. 2A shows a surgical environment 200 within a patient P. An imaging instrument (e.g., imaging instrument 28) is used to display an image 202 of the imaging instrument's field of view within the surgical environment 200. The image 202 may be a three-dimensional image generated as a composite image of images obtained by a stereoscopic endoscope and viewed by a user via right-eye and left-eye displays. The field of view includes a distal end portion of instrument 26a, a distal end portion of instrument 26b, and a distal end portion of instrument 26c. The image 202 also includes an information field 204 located on the periphery of the image that may include instructions to the clinician, warnings, instrument identification information, status information, or other information related to the procedure. Various information is overlaid on the image 202 to assist the clinician performing the procedure using the system 10. In this embodiment, a badge 206 includes relevant information related to the medical instrument 26b, such as a numerical identifier of the arm 54b (e.g., arm "3") to which the instrument is coupled. The badges 206 are displayed in proximity to the medical devices 204 so that a clinician can easily see relevant information about the medical devices 26b. In some embodiments, the badges may also include relevant information including the type of medical device, the status of the medical device (e.g., cautery charging / ready), etc. Methods for creating and displaying the badges 206 are described in more detail below with reference to FIGS. 8-11. In various embodiments, the badges may be round, oval, square, or any suitable shape. In various embodiments, the badges may have a color, outline, shape, or other feature to identify the badges or distinguish them from one another.

[0026] The distal end portion of the medical instrument 26b that is visible in the field of view includes the end effector tip 208, the shaft 210, and the joint region 212 between the shaft and the end effector tip. In some embodiments, the default location for placement of the badge 206 is superimposed over the joint region 212. In other embodiments, the default location for placement of the badge 206 may be on the end effector tip 208 or the shaft 210. In other embodiments, the badge 206 is displayed adjacent to the distal end portion. The default location of the badge may depend on the size of the distal end portion of the medical instrument 26b in the field of view. For example, if the image is zoomed in closely (e.g., FIG. 3F) and the end effector takes up most of the image, the badge may be placed proximal to one of the end effector jaws without obstructing the view at the distal end of the end effector jaws. However, if the image is zoomed out and the end effector jaws are relatively small in the image, the badge may be placed on the joint region or shaft to avoid obscuring the jaws. The placement of the badge 206 allows the clinician to receive relevant information while remaining focused on the instrument and the tissue manipulated by the instrument. The placement of the badge 206 also avoids obscuring objects in the field of view that need to be viewed by the clinician.

[0027] In some embodiments, the badge 206 has a central portion 214 and an orbiting portion 216. The central portion 214 and the orbiting portion 216 may include different associated information. For example, as shown in FIG. 2, the central portion 214 includes a number indicating the arm 54b (e.g., arm "3") to which the medical instrument 26b is attached. In various embodiments, the orbiting portion 216 includes associated information related to a control device used by the clinician to control the medical instrument 26b. This associated information may be, for example, an indication of whether a right (e.g., "R") or left (e.g., "L") hand-operated input device is currently controlling the medical instrument 26b with which the badge 206 is associated.

[0028] An orbital portion 216 of the badge 206 may move circumferentially relative to the badge's central portion 214. The circumferential position of the orbital portion 216 indicates the rotational position of the controls and informs the clinician where the hands on the controls are located relative to the medical device 26b they are controlling. The function of the orbital portion 216 is shown in more detail in Figures 3A-3F.

[0029] Badge 207 is placed in proximity to instrument 26a and has similar features and attributes as badge 206. Badge 207 includes relevant information related to medical instrument 26a, such as a numeric identifier of arm 54a to which the instrument is coupled (e.g., arm "1"). In this embodiment and at this stage of the procedure, instrument 26c is not currently under the control of the input controller. Thus, a badge is not placed in proximity to instrument 26c. In an alternative embodiment, badge information may be provided for instrument 26c indicating the numeric identifier of arm 54c (e.g., arm "4"), but the fact that the instrument is not currently under the control of the input device is indicated by a color, symbol, or other indication.

[0030] 2B illustrates a method 100 for operating a teleoperated system to perform surgery. In process 102, the method includes displaying an image 202 of a field of view of a surgical environment. In process 104, a position of a medical instrument 26a in the surgical environment is determined. In process 106, relevant information about the medical instrument 26a is determined. The relevant information includes information about a teleoperated manipulator to which the medical instrument 26a is coupled and / or about operator input controls that control the medical instrument. In process 108, a badge is displayed adjacent to the image of the medical instrument 26a in the image 202 of the field of view. The badge includes the relevant information.

[0031] 3A-3C show a clinician's right hand 300 controlling an input control device 302 (i.e., input control device 36). The input control device 302 controls a medical instrument 26c. As shown in FIG. 3A, from the clinician's perspective, the input control device 302 extends to the right. Hand-related information including hand identity (right or left) and hand orientation may be conveyed to the clinician via the badge. Although this control-related information is described in terms of a human hand, similar related information may be provided where control is provided by a foot pedal, eye-tracker, voice command, or other clinician control. As shown in FIG. 3D, the hand-related information regarding the control device 302 of FIG. 3A is visually conveyed to the clinician within the image 202 by orienting the orbiting portion 222 of the badge 218 to the right (e.g., at approximately the 3 o'clock position) relative to the center portion 220 of the badge. The orbiting portion 222 displays an "R" indicating that the right hand control device 302 is being used to control the medical instrument 26c. As shown in Figure 3B, from the clinician's perspective, the input control device extends approximately 90 degrees upward from its position in Figure 3A. As shown in Figure 3E, this orientation information for the control device 302 of Figure 3B is visually conveyed to the clinician in the image 202 by orienting the orbital portion 222 of the badge 218 upward (e.g., approximately the 12 o'clock position) relative to the central portion 220 of the badge. As shown in Figure 3C, from the clinician's perspective, the input control device extends approximately 90 degrees downward from its position in Figure 3A. As shown in Figure 3F, this orientation information for the control device 302 of Figure 3C is visually conveyed to the clinician in the image 202 by orienting the orbital portion 222 of the badge 218 downward (e.g., approximately the 6 o'clock position) relative to the central portion 220 of the badge.

[0032] In some alternative embodiments, the hand-related information may be provided as an illustration or 3D model of a hand. The illustration or model depicts a left or right hand depending on whether a left or right hand control device is being used to control the associated medical device. As with orbital segment 222 of FIGS. 3D-3F, when the orientation of the control device changes, the position of the illustration or model changes accordingly to visually convey information about the new orientation of the control device to the clinician. The properties and characteristics described for any of the badges in this disclosure may also be applied to any of the other badges in this disclosure.

[0033] 4A and 4B, further behavior of the badge 206 is shown as the imaging device is moved or zoomed in within the surgical environment. As shown in FIG. 4A, the imaging device is moved or zoomed in toward the medical devices 26a, 26b, 26c. As the medical devices 26a, 26b, 26c appear larger in the field of view and image 202, the badges 206, 207 become larger and change in size to appear as if they were located at a distal depth in the same field of view as the distal end portions of the devices. In FIG. 4B, as the imaging device is moved or zoomed out further away from the medical devices 26a, 26b, 26c, the badges 206, 207 become smaller and change again to appear as if they were located at the same depth in the field of view as the distal end portions of the devices. In some embodiments, there are limits on the maximum and minimum sizes of the badges 206, 207 so that they are not unreadable. Further details regarding the location and size of the badges are provided below in the description for FIG. 8-11. The size of the badges 206, 207 can also provide additional information to the clinician, such as the distance between the medical instruments 26a, 26b and the imaging device, which can be useful because different types of medical instruments vary in size and a small instrument that is closer to the imaging device may appear the same size as a larger instrument that is farther away from the imaging device.

[0034] The badges 206, 207 are superimposed on the image 202 so that they can appear on the display even if the view of the medical instruments 26a, 26b is obstructed by tissue, other instruments, etc. As shown in FIG. 5A, the badge 206 remains visible even if the associated instrument 26b is hidden behind tissue. The presence of the badge 206, which is substantially offset and has a different apparent size and depth than the other instruments 26a, 26c visible in the field of view, indicates to the clinician that the instrument 26b associated with the badge is occluded. The badge 206 allows the occluded instrument 26b to be found as shown in FIG. 5B, where the tissue has been moved and the medical instrument 26b to which the badge 206 is attached is now visible in the field of view.

[0035] The teleoperated medical system 10 may have multiple operating states for each instrument. These may include an active state in which the medical instrument is under the control of the input controller such that the clinician is actively controlling the medical instrument. In a disengaged state, the medical instrument is not under the control of the input controller, remains stationary, and is locked in place by the associated manipulator arm. The system may also have engagement states that prompt the user to engage with or control the medical instrument via the associated input controller. In some embodiments, a badge is displayed when the teleoperated medical system is in an engagement state for a particular instrument. For example, in FIG. 4A, instruments 26a and 26b are in an engagement state in which the badge information prompts the user to control instrument 26a on arm 1 with the left ("L") input controller and instrument 26b on arm 3 with the right ("R") input controller. Since instrument 26c is in a disengaged state, no badge is presented for that instrument. During an active state, the badge may remain visible in the image or may be removed to avoid distracting the clinician. Presenting the badge in the engaged state allows the clinician to recognize the relevant information and location of each instrument in the field of view 202 before controlling any of the medical instruments. For example, the location of the badge may be used to determine if a medical instrument is obscured from view by tissue before controlling the instrument. In addition, the relevant information contained within the badge may provide the clinician with information that allows them to select which medical instrument to control to perform a desired procedure. Furthermore, the information provided by the orbiting portion of the badge may tell the clinician how to position the input control device to match the orientation of the medical instrument before taking control. In some embodiments, the system may require the clinician to match this position before allowing a state change from engaged to active.

[0036] 6A-6B, the distal end portions of medical instrument 26a visible in the field of view include end effector tip 230, shaft 232, and joint area 234 between the shaft and end effector tip. The distal end portions of medical instrument 26c visible in the field of view include end effector tip 240, shaft 242, and joint area 244 between the shaft and end effector tip. As shown in FIG. 6A and previously described, the default location for placement of badge 207 is superimposed over joint area 234. The default location for placement of badge 218 is superimposed over joint area 244. In FIG. 6A, the instrument distal end portions are spaced apart and easily distinguishable from one another, and thus badges 207 and 218 are clearly readable and easily associated with medical instruments 26a and 28c, respectively. FIG. 6B illustrates an instance where the distal end portions of instruments 26a and 26c are overlapping or very close together in image 202. If allowed to remain in the default positions, the badges 207, 218 would appear overlapping or adjacent or superimposed on more than one instrument, causing confusion to the clinician. To remedy this problem, as shown in FIG. 6B, the badge 207 is moved a shift distance along the longitudinal axis A1 of the instrument 26a from its default position near the distal end of the joint region 234. In this embodiment, the badge 207 is relocated to the distal end of the shaft 232 of the instrument 26a. Similarly, the badge 218 is moved a shift distance along the longitudinal axis A2 of the instrument 26c from its default position near the distal end of the joint region 244. In this embodiment, the badge 218 is relocated to the distal end of the shaft 242 of the instrument 26c. The shift distance may correspond to the location of the badge relative to a calibrated kinematic model of the instrument coupled to the arm and spar and / or may correspond to the size of the instrument distal end portion within the field of view 202.For example, a calibrated kinematic model (e.g., based on the manipulator's joint positions, kinematic sensors, and sizes of components in the kinematic chain) provides the position and orientation of the joint region 234 of the instrument 26a in the surgical environment. Uncertainty factors are associated with the determined position orientation of the joint region 234 due to uncertainties in the kinematic model. Based on the kinematic model, the kinematic uncertainty factors of all instruments in the field of view, and the default position of the badges, the teleoperation system can determine a proximity factor (e.g., center-to-center or edge-to-edge distance, percentage of overlap) that indicates whether the badges are overlapping or close enough to each other to create ambiguity to the clinician if the badges were left in the default position. If the proximity factor is less than a predetermined interference distance, the badges are adjusted as necessary to raise the proximity factor above the predetermined interference distance. For example, in some embodiments, the badges can be moved along or across the axis of the instrument by at least a minimum shift distance. In other embodiments, the badges can be resized or reshaped to maintain an acceptable interference distance while still providing the desired information content. In this manner, the badges 207, 218 remain clearly associated with their respective medical instruments 26a, 26c and remain clearly readable.

[0037] 7A-7C, badges with relevant information may be particularly useful when a medical instrument approaches a boundary of the image 202 of the field of view or moves out of the image 202. FIG. 7A shows a displayed image 202 of a field of view of an imaging instrument in a surgical environment 200. The displayed image has a boundary 700. A distal end portion of a medical instrument 26c, specifically the end effector tip 240, is within the boundary 700 and is visible within the displayed image 202. A determination of whether the instrument 26c is inside or outside the field of view of the imaging instrument may be based on a calculated position of the end effector tip 240. Due to small error factors associated with the teleoperation system, the instrument, and / or the imaging system, a determination of the position of the tip 240 relative to the imaging instrument includes an associated cumulative error factor. To avoid providing the clinician with false-positive out-of-view indicators, the determination of whether the instrument tip is out of the imaging system field of view may be biased by estimating a range of possible locations of the distal tip and suppressing the out-of-view indicator if any or a specified percentage of the estimated possible locations of the distal tip are within the field of view. The sensitivity of the bias may be adjusted based on the clinician's tolerance to false-positive out-of-view indicators. A set of error bounding volumes 702 are associated with the tip 240. The bounding volumes 702 may be graphically displayed in the image 202 or may be associated with the tip 240 without being displayed. The error bounding volumes may represent the predicted location of the distal and proximal ends of the tip portion to within a high degree of accuracy, such as 90-99%. The error bounding volumes 702 represent the predicted location of the tip 240 of the instrument 26c.Further explanation of error calculations and bounding volumes is provided in U.S. Provisional Application No. 61 / 954,442, filed March 17, 2014, disclosing “Systems and methods for offscreen indication of instruments in a teleoperational medical system,” which is incorporated by reference in its entirety.

[0038] As shown in FIG. 7A, a bounding volume 702 is within the field of view bounded by a boundary 700. The bounding volumes 702 may or may not be rendered in the image 202, but their location relative to the boundary may be determined by the system regardless of whether they are displayed. Because the medical instrument 26c is visible in the display and the bounding volume is within the boundary 700, the badge 218 is displayed accordingly on the image 202 of the field of view. To maintain visibility and readability, the badge 218 remains completely inside the boundary 700, even if the distal end portion of the instrument 26c to which the badge 206 is attached is partially outside the boundary 700, as shown here. This allows the clinician to clearly locate the medical instrument 26c, even if it is barely visible in the image 202. In FIG. 7B, the medical instrument 26c is not visible in the displayed image 202. In some embodiments, the system may cease displaying the badge 218 when the associated medical instrument 26c is not visible in the image 202. Because the medical instrument 204 may be remotely controlled without being visible in the clinician's field of view, inadvertent movement of the instrument outside the field of view poses a safety hazard. In addition, the clinician may lose track of an instrument located outside the field of view. To minimize these risks, an out-of-field instrument indicator may be presented visually or audibly to increase the clinician's awareness of the location of an instrument that is not visible in the field of view. However, in FIG. 7B, the bounding volume 702 for the medical instrument 26c is within the boundary 700, indicating that the location of the tip 240 relative to the boundary 700 is uncertain. Thus, although the bounding volume 702 remains within the boundary 700, the badge 218 remains on the image 202 even though the image 202 does not appear to include the tip 240.

[0039] When the medical instrument 204 is far enough outside the boundary 700 that the bounding volume 702 is also outside the boundary, an out-of-field instrument indicator 704 is provided along the boundary 700 of the image 202 of the field of view to indicate that the medical instrument 26c is located outside the field of view in the general direction of the indicator, as shown in FIG. 7C. In this embodiment, the indicator 704 is a graphic bar, but in other embodiments, it may be a series of dots, or an icon, an alphanumeric indicator. In addition to or instead of the visual indicator 704, an audible out-of-field indicator, such as a beep or a language-based command, may alert the clinician that the medical instrument 204 is out of the field of view. An audible cue may pan between the left and right speakers of the surgeon's console to reinforce the instrument location relative to the field of view. Alternatively, the audible cue may be emitted from the left or right speaker corresponding to the left or right hand control associated with the instrument. In addition to or instead of the visual indicator 704, textual information 706 associated with the out-of-field instrument may be provided to alert the clinician and / or provide identifying information regarding the instrument or instructions for visualizing the instrument. In various embodiments, the out-of-field indicator 704 is mutually exclusive with the badge 218. When the badge 218 is displayed, the out-of-field indicator 704 is not displayed, and vice versa. This provides the clinician with constant knowledge of the location of the medical instrument 26c.

[0040] In various embodiments, the use of the out-of-field indicator may be limited to avoid being distracting to the clinician. The use of the out-of-field indicator may be context-sensitive such that the out-of-field indicator is only displayed during certain operating states of the remote control system. For example, the out-of-field indicator may be displayed during a state of the system where the operator controls the movement of the imaging system, which state may be known as the camera control state. As another example, the out-of-field indicator may be displayed while the system is waiting for input from the operator to control the movement of the instrument, described above as the disengaged state. As another example, the out-of-field indicator may be displayed for a few seconds after the operator initiates a state of the system where the operator controls the movement of the instrument, described above as the engaged state. In yet other alternative embodiments, the out-of-field indicator may be disabled or selectively enabled when the clinician wants to know the location of the out-of-field instrument. In some embodiments, the clinician must provide confirmation that the instrument tip is outside the field of view before operation of the out-of-field instrument is permitted. Additional warnings or confirmations may be used for energy emitting devices, sharp devices, or devices that pose increased risk to the patient when used without visualization. Further description of the out-of-field indicator system is provided in US Provisional Application No. 61 / 954,442, which is incorporated by reference.

[0041] As discussed above, badges may be positioned and sized to correspond to the image of the instrument distal end with which they are associated. FIGS. 8-11 provide further details for maintaining this physical correspondence. FIG. 8 illustrates a method 800 for rendering a badge within an image 202. In process 802, the position of the distal end portion of the medical instrument is projected into the coordinate space of the image 202. FIG. 9 illustrates a method 900 for performing process 802. In process 902, the forward kinematics of the teleoperated assembly are calculated relative to a world coordinate system (e.g., the surgical environment coordinate space F). W) are evaluated to determine the position and orientation of the medical instruments (including the imaging device) within the surgical system. More specifically, the kinematics associated with each set of links of each medical instrument and teleoperated manipulator are determined in a world coordinate system. The world coordinate system is a set of Cartesian coordinates established with reference to the surgical environment in which the teleoperated system is located.

[0042] In process 904, (e.g., in the instrument coordinate space F I The position and orientation of the distal end portion of each medical instrument (in the endoscope coordinate system (e.g., in the imaging device coordinate space F E ) The endoscope coordinate system is a set of Cartesian coordinates established by its origin at the distal tip of the endoscope. Using the forward kinematics of the endoscope and the forward kinematics of each medical instrument in the world coordinate system, as determined in process 902, the distal end position and orientation of the medical instruments relative to the endoscope tip position and orientation can be calculated in the endoscope coordinate system. In general, the default position of the badge may be at or near the kinematic endpoint of the medical instrument.

[0043] In process 906, a calibrated camera model is obtained for the endoscope. The calibrated camera model is a mapping from the endoscope coordinate space to the right and left eye image coordinate spaces. In one embodiment, the mapping is accomplished with a series of three transformations. First, a modelview transform maps from the endoscope coordinate space (at the distal end of the endoscope) to the eye coordinate space for the endoscope, accounting for the interocular separation of the stereoscopic cameras in the imaging system and the different coordinate system conventions. Second, a perspective projection transform maps from the eye coordinate space to normalized coordinates (e.g., -1, +1), accounting for the view of the endoscope's field of view and convergence distance. Third, a viewport bounds transform maps from the normalized eye coordinates to the right and left eye image coordinate spaces. In one embodiment, these three transformations can be combined into one 4x4 homogenous transform for each of the right and left eye images.

[0044] In process 908, the calibrated camera model is used to project points and vectors from the endoscope coordinate space to each eye's image coordinate space. More specifically, the calibrated camera model projects the medical instrument distal tip position and orientation into right eye image coordinate space and left eye image coordinate space to provide a stereoscopic 3D image to the clinician.

[0045] In process 910, uncertainties in the kinematic model of the teleoperated system that result in corresponding uncertainties in the position of the distal tip portion of the associated medical instrument are estimated for the endoscope tip. In process 912, these uncertainties are mapped into the right-eye and left-eye image coordinate spaces in which the physician views the three-dimensional image 202.

[0046] In process 914, the kinematic remote center (e.g., the incision site around which the instrument pivots) is mapped into endoscope coordinate space using the forward kinematics determined in process 902. In process 916, the calibrated camera model is used to project the remote center into each eye's image coordinate system in a manner similar to that described in process 908.

[0047] Processes 902-916 may be repeated for each medical instrument in the remote operation system such that all medical instruments are properly projected in the stereoscopic 3D image 202. The processes may be performed in sequence or in parallel, and one or more processes may be updated while others are not.

[0048] Referring again to FIG. 8, the rendering of badge graphics is based on consideration of a number of display factors, including, for example, a default location at or near the distal tip of the distal end of the medical instrument, potential ambiguities associated with overlapping or uncertain badge placement, constraints imposed by the endoscope viewport, and system rules for hidden instruments. The location of the badge graphics may be determined in image coordinate space for each eye. In process 804, potential badge overlaps are checked and resolved to reduce the likelihood that a badge is spatially associated with the wrong instrument. FIG. 10 illustrates a method 1000 for performing process 804 of FIG. 8. In this embodiment, at least two medical instruments are positioned in the surgical space. As discussed above, the default location for rendering the badge is the distal end portion of the medical instrument. In process 1002, the distance in image space coordinates between the distal end portions of the two medical instruments is calculated. In process 1004, a minimum allowable inter-badge distance threshold is calculated. The minimum allowable inter-badge distance threshold is the sum of the badge projection radii (or another maximum dimension from the badge center if the badge is not circular) plus the maximum of the sum of the projected uncertainties of the badge positions. In process 1006, the image space distance between the two distal end portions is compared to the calculated threshold distance to determine if their associated badges overlap or can be considered to overlap. If an overlap is detected, in process 1008, a direction to move the badge to resolve the overlap is calculated. The direction to move the badge is determined by the unit vector between the remote center projected into image coordinate space (determined in processes 908 and 916) and the instrument distal end portion. Additionally, if an overlap is detected, in process 1010, a distance to translate the badge to resolve the overlap is determined. The distance the badges are offset is calculated as the difference between the minimum allowable inter-badge distance and the current distance between the distal end portions.This offset prevents badge graphics from overlapping and provides badge positions that are sufficiently spaced to avoid uncertainty about instrument association.

[0049] Referring again to FIG. 8, in process 806, the badge is constrained to a viewable space of the image 202 that corresponds to the boundaries of the endoscopic viewport at the distal tip of the endoscope. This process may place the badge shape within the boundaries of the image 202 without clipping any of the shapes. This process may also control the perceived depth of the badge by preventing modification of horizontal disparity if the badge is shifted to avoid image boundaries or to prevent ambiguity. FIG. 11 shows a method 1100 for performing process 806 of FIG. 8. In process 1102, a right eye boundary of the viewable space of the image 202 that corresponds to the endoscopic viewport is determined. In process 1104, a projection radius of the badge shape is determined for the right eye. In process 1106, for the right eye, the boundary is inset by the projected badge radius to create an inset boundary. In process 1108, the Y position of the badge's center is adjusted to stay within the inset boundary for the right eye. In process 1110, the offset of the X position of the badge's center required to keep the badge completely within the inset boundary is calculated. Processes 1102 through 1110 are repeated for the left eye image. In order to maintain the proper apparent depth in stereoscopic 3D, the badge must be adjusted the same amount in both the right and left eye images. This maintains the horizontal parallax required for the proper perceived depth of the badge. In process 1112, the X offset calculated in process 1110 for one of the right or left eye images is applied to the X position of the badge in both the right and left eye images. In some embodiments, a larger X offset value is selected to prevent the possibility of the badge shape being even partially outside the boundary 700.

[0050] Referring again to FIG. 8, in process 808 the badge is rendered into the left and right eye displays to form image 202 based on the calculations performed in processes 802, 804 and 806.

[0051] One or more elements in an embodiment of the present invention may be implemented in software running on a processor of a computer system such as a control processing system. When implemented in software, the elements of an embodiment of the present invention are essentially code segments performing the necessary tasks. The program or code segments can be stored in a processor-readable storage medium or device, which can be downloaded by a computer data signal embodied in a carrier wave over a transmission medium or communication link. The processor-readable storage device can include any medium capable of storing information, including optical media, semiconductor media, and magnetic media. Examples of the processor-readable storage device include electronic circuits, semiconductor devices, semiconductor memory devices, read-only memories (ROMs), flash memories, erasable programmable read-only memories (EPROMs), floppy disks, CD-ROMs, optical disks, hard disks, or other storage devices. The code segments can be downloaded via a computer network, such as the Internet, an intranet, or the like.

[0052] It should be noted that the presented processes and displays are not inherently related to any particular computer or other apparatus. Various general-purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct a more specialized apparatus to perform the described operations. The required structure for a variety of these systems appears as elements in the claims. Additionally, embodiments of the present invention are not described with reference to any particular programming language. It will be understood that a variety of programming languages ​​may be used to implement the teachings of the present invention as described herein.

[0053] While certain exemplary embodiments of the invention have been described and are illustrated in the accompanying drawings, it is to be understood that such embodiments are merely illustrative of the broad invention and are not intended to be limiting of the broad invention, and that embodiments of the invention are not limited to the specific constructions and arrangements shown and described, since various other modifications may occur to those skilled in the art.

[0054] The following additional note is added: (Supplementary Note 1) A remotely operated assembly including an operator control system and a plurality of manipulators configured for remote operation by the operator control system, wherein a first manipulator of the plurality of manipulators is configured to control movement of a first medical instrument in a surgical environment; and a processing unit including one or more processors; having The processing unit includes: displaying an image of a field of view of the surgical environment; determining first relevant information relating to the first medical device; displaying a first badge within the image of the field of view adjacent an image of the first medical instrument; It is configured as follows: the first associated information includes information regarding at least one of the first manipulator and the operator control system; the first badge includes the first related information; system. (Supplementary Note 2) The first association information includes an identifier for the first manipulator that controls the movement of the first medical instrument. 2. The system of claim 1. (Supplementary Note 3) The operator control system includes a control device, and the first association information includes an identifier for the control device. 2. The system of claim 1. (Supplementary Note 4) The first badge includes a central portion and an orbital portion positioned around the central portion based on an orientation of the control device. 4. The system of claim 3. (Supplementary Note 5) The processing unit is configured to display the first badge superimposed on an image of the distal end portion of the first medical instrument. 2. The system of claim 1. (Supplementary Note 6) The processing unit is further configured to adjust a size of the first badge when the first medical instrument changes depth relative to an imaging instrument, the imaging instrument capturing the field of view of the surgical environment. 2. The system of claim 1. (Supplementary Note 7) The processing unit is configured to terminate display of the first badge in the image of the field of view when the operator control system transitions from a first state to a second state. 2. The system of claim 1. (Supplementary Note 8) A second manipulator of the plurality of manipulators is configured to control movement of a second medical instrument in the surgical environment; The processing unit further comprises: determining second relevant information relating to the second medical device; displaying a second badge within the image of the field of view adjacent an image of the second medical instrument; determining a distance between the first badge and the second badge; moving the first badge in a linear direction aligned with a shaft of the first medical instrument if the determined distance is less than a predetermined interference distance. It is configured as follows: the second associated information includes information regarding at least one of the second manipulator and the operator control system; the second badge includes the second related information; 2. The system of claim 1. (Supplementary Note 9) The processing unit further comprises: determining a tip position of the first medical instrument; determining a position error associated with the tip position; determining at least one bounding volume based on the position error; determining whether the bounding volume is within the field of view; if the bounding volume is within the field of view, displaying the first badge within the image of the field of view proximate the bounding volume; It is configured as follows: 2. The system of claim 1. (Supplementary Note 10) The processing unit further comprises: if the bounding volume is not within the field of view, terminating display of the first badge in the image of the field of view and generating an out-of-field instrument indicator for display in conjunction with the image of the field of view. It is configured as follows: 10. The system of claim 9. (Supplementary Note 11) A method for providing a surgical environment, comprising: displaying an image of a field of view of a surgical environment, wherein a first medical instrument within the field of view is coupled to a first manipulator of a plurality of manipulators of a remote manipulation assembly; determining first relevant information related to the first medical instrument, the first relevant information including information related to at least one of the first manipulator and an operator control device; displaying a first badge in the image of the field of view adjacent an image of the first medical instrument, the first badge including the first relevant information; Including, method. (Supplementary Note 12) The first association information includes an identifier for the first manipulator that controls a movement of the first medical instrument. 12. The method according to claim 11. (Supplementary Note 13) The first association information includes an identifier for the operator control device. 12. The method according to claim 11. (Supplementary Note 14) The first badge includes a central portion and an orbital portion positioned about the central portion based on an orientation of the operator control device. 14. The method according to claim 13. (Supplementary Note 15) The method further includes displaying the first badge superimposed on an image of the distal end portion of the first medical instrument. 12. The method according to claim 11. (Supplementary Note 16) The method further includes adjusting a size of the first badge when the first medical instrument changes depth relative to an imaging instrument, the imaging instrument capturing the view of the surgical environment. 12. The method according to claim 11. (Supplementary Note 17) The method further includes the step of terminating the display of the first badge in the image of the field of view when the remote control assembly transitions from the first state to the second state. 12. The method according to claim 11. (Supplementary Note 18) A method for determining second associated information relating to a second medical instrument, the second associated information including information relating to at least one of a second manipulator and the operator control device; displaying a second badge within the image of the field of view adjacent an image of the second medical instrument, the second badge including the second associated information; determining a distance between the first badge and the second badge; if the determined distance is less than a predetermined interference distance, moving the first badge in a linear direction aligned with a shaft of the first medical instrument; Further comprising: 12. The method according to claim 11. (Supplementary Note 19) A step of determining a tip position of the first medical instrument; determining a position error associated with the tip position; determining at least one bounding volume based on the position error; determining whether the bounding volume is within the field of view; if the bounding volume is within the field of view, displaying the first badge within the image of the field of view proximate the bounding volume; Further comprising: 12. The method according to claim 11. (Supplementary Note 20) If the bounding volume is not within the field of view, terminating display of the first badge in the image of the field of view and generating an out-of-field instrument indicator for display with the image of the field of view. 19. The method according to claim 19.

Claims

[Claim 1] a teleoperated assembly including an operator control system and a plurality of manipulators configured for remote operation by the operator control system, a first manipulator of the plurality of manipulators configured to control movement of a first medical instrument in a surgical environment; a processing unit including one or more processors; having The processing unit comprises: displaying an image of a field of view of the surgical environment; determining a tip position of the first medical instrument by determining a bounding volume associated with the first medical instrument, the bounding volume indicating a range of expected positions of the tip of the first medical instrument, the range of expected positions being based on an error factor; determining whether the bounding volume is within the field of view; if the bounding volume is within the field of view, displaying a first badge associated with the first medical instrument in the image of the field of view proximate to the bounding volume. It is configured as follows: system.

Images