System and method for screen recognition of instruments in a remotely operated medical system

Abstract

The present invention relates to a system and method for screen recognition of instruments in a remotely operated medical system. A remotely operated assembly is disclosed, the remotely operated assembly including an operator control system and a plurality of manipulators configured to control movement of a medical instrument in a surgical environment. The manipulators are remotely controlled by the operator control system. The system also includes a processing unit configured to display an image of a field of view of the surgical environment, determine associated information about the manipulator and the operator control system, and display a marker near the medical instrument in the image of the field of view of the surgical environment. The marker displays associated information of the medical instrument with which the marker is associated.

Description

[0001] This application is a divisional application of Chinese patent application 201680025698X (PCT / US2016 / 022612), whose international application date is March 16, 2016 and whose national phase entry date is November 3, 2017, and whose name is “System and method for screen identification of instruments in a remotely operated medical system”.

[0002] Related Applications

[0003] This patent application claims priority to and the benefit of the filing date of U.S. Provisional Patent Application No. 62 / 134,297, filed on March 17, 2015, entitled “SYSTEMS AND METHODS FOR RENDERING ON SCREEN IDENTIFICATION OF INSTRUMENTS IN A TELEOPERATIONAL MEDICAL SYSTEM,” which is incorporated herein by reference in its entirety. Technical Field

[0004] The present disclosure relates to systems and methods for performing teleoperated medical procedures, and more particularly to systems and methods for displaying information about teleoperated instruments used in a surgical environment. Background Art

[0005] Minimally invasive medical technology aims to reduce the amount of tissue damaged during invasive medical procedures, thereby reducing patient recovery time, discomfort and harmful side effects. Such minimally invasive technology can be implemented through natural orifices in the patient's anatomical structure or through one or more surgical incisions. Through these natural orifices or incisions, clinicians can insert medical tools to reach the target tissue position. Minimally invasive medical tools include instruments such as therapeutic instruments, diagnostic instruments and surgical instruments. Minimally invasive medical tools may also include imaging instruments such as endoscopic instruments. Imaging instruments provide users with a field of view within the patient's anatomical structure. Some minimally invasive medical tools and imaging instruments can be remotely operated or otherwise computer-assisted. In a remotely operated medical system, whether due to a small instrument size, close to the edge of the field of view, or due to being blocked by tissue, the instrument may be difficult to see. It may also be difficult for clinicians operating remote operating systems to track which instruments are under the control of which operators. Systems and methods are needed to provide clinicians with information about instruments under the control of clinicians to reduce the risk of clinician errors. Summary of the invention

[0006] Embodiments of the invention are summarized by the dependent claims.

[0007] In one embodiment, a teleoperation assembly includes an operator control system and a plurality of manipulators configured to control movement of a medical device in a surgical environment. The manipulators are remotely controlled by the operator control system. The system also includes a processing unit configured to display an image of a field of view of the surgical environment, determine associated information about the manipulators and the operator control system, and display a marker near the medical device in the image of the field of view of the surgical environment. The marker displays associated information of the medical device, and the marker appears to be associated with the medical device.

[0008] In another embodiment, an image of a field of view of a surgical environment is displayed. A location of a medical instrument within the environment is determined, associated information of the medical instrument and an operator control system is determined, and a marker is displayed in the image of the field of view of the surgical environment near the medical instrument. The medical instrument is attached to a manipulator that is remotely controlled by the operator control system. The marker displays the associated information of the medical instrument and the marker is displayed near the medical instrument.

[0009] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory in nature, and are intended to provide an understanding of the present disclosure without limiting the scope of the present disclosure. In this regard, additional aspects, features and advantages of the present disclosure will be apparent to those skilled in the art through the following specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Aspects of the present disclosure are best understood from the following detailed description when read in conjunction with the accompanying drawings. It is emphasized that, in accordance with standard practice in the industry, various features are not drawn to scale. In fact, the dimensions of various features may be arbitrarily increased or decreased for clarity of discussion. In addition, the present disclosure may repeat reference numerals and / or letters in various examples. This repetition is for the purpose of simplicity and clarity and does not in itself dictate the relationship between the various embodiments and / or configurations discussed.

[0011] Figure 1A is a schematic diagram of a remotely operated medical system according to an embodiment of the present disclosure.

[0012] Figure 1B is a perspective view of a surgeon's console for teleoperating a medical system, in accordance with many embodiments.

[0013] Figure 1C is a perspective view of a teleoperated medical system electronics cart, in accordance with many embodiments.

[0014] Figure 1D is a perspective view of a patient-side cart, according to one example of principles described herein.

[0015] Figure 2A A field of view of a surgical workspace is shown with at least one medical instrument visible in the field of view, the instrument having a marker associated therewith and containing associated information related to the instrument.

[0016] Figure 2B is a flow chart illustrating a method for operating a teleoperated medical system.

[0017] Figure 3A-3C It shows that the corresponding Figure 3D-Figure 3F The associated information shown marks the arrangement of input controls and the user's hand position.

[0018] Figure 3D-Figure 3F Various arrangements of track portions of associated information marks are shown.

[0019] Figure 4A-4B The size change behavior of the associated information marker is shown as the associated instrument is moved toward and away from the imaging device.

[0020] Figure 5A-Figure 5B It shows how the associated information indicia remains visible when the associated medical instrument is blocked from view.

[0021] Figure 6A-6B The adjustment of the associated information markers when the medical devices are in close proximity is shown.

[0022] Figure 7A-7C The behavior of the associated information marker is shown when the associated medical instrument is close to the edge of the image of the surgical environment and when the medical instrument moves outside the image of the surgical environment.

[0023] Figure 8 A method of presenting a marker within an image of a field of view of a surgical workspace is shown.

[0024] Fig. 9 Shown in detail Figure 8 A process in which the position of a distal portion of a medical device is projected into the coordinate space of an image of a field of view of a surgical workspace.

[0025] Fig.10 Shown in detail Figure 8 A process where potential overlap of markings is examined and resolved. In this case, there are at least two medical devices.

[0026] Fig.11 Shown in detail Figure 8 A procedure in which the marking is confined to the visual space of the image of the field of view of the surgical workspace. DETAILED DESCRIPTION

[0027] For the purpose of promoting an understanding of the principles of the present disclosure, reference will now be made to the embodiments shown in the accompanying drawings, and specific language will be used to describe the embodiments shown in the accompanying drawings. However, it should be understood that it is not intended to limit the scope of the present disclosure. In the following detailed description of aspects of the present invention, a large number of specific details are set forth in order to provide a thorough understanding of the disclosed embodiments. However, it will be apparent to those skilled in the art that embodiments of the present disclosure may be practiced without these specific details. In other cases, well-known methods, processes, components, and circuits are not described in detail to avoid unnecessarily obscuring aspects of embodiments of the present invention.

[0028] As would normally be expected by a person skilled in the art to which the present disclosure relates, any changes and further modifications to the described devices, apparatus, methods, and any further applications of the principles of the present disclosure are fully contemplated. In particular, it is fully contemplated that the features, components, and / or steps described with respect to one embodiment may be combined with the features, components, and / or steps described with respect to other embodiments of the present disclosure. In addition, the dimensions provided herein are for specific examples, and it is contemplated that different sizes, dimensions, and / or ratios may be utilized to implement the concepts of the present disclosure. In order to avoid unnecessary descriptive repetitions, if applicable in other illustrative embodiments, one or more components or actions described according to an illustrative embodiment may be used or omitted. For the sake of brevity, a large number of iterations of these combinations will not be described separately. For simplicity, in some cases, the same reference numerals are used throughout the accompanying drawings to refer to the same or similar parts.

[0029] The following embodiments will describe various instruments and parts of instruments based on the state of the various instruments and parts of instruments in three-dimensional space. As used herein, the term "position" refers to the position of an object or a part of an object in three-dimensional space (e.g., three translational degrees of freedom along Cartesian X, Y, and Z coordinates). As used herein, the term "orientation" refers to the rotational placement of an object or a part of an object (three rotational degrees of freedom—e.g., roll, pitch, and yaw). As used herein, the term "pose" refers to the position of an object or a part of an object in at least one translational degree of freedom and the orientation of an object or a part of an object in at least one rotational degree of freedom (up to six degrees of freedom). As used herein, the term "shape" refers to a set of poses, positions, or orientations measured along an object.

[0030] With reference to the attached drawings Figure 1A, generally indicated by reference numeral 10, is a teleoperated medical system for use in a medical procedure including, for example, a diagnostic, therapeutic, or surgical procedure. As will be described, the teleoperated medical system of the present disclosure is under teleoperated control of a surgeon. In alternative embodiments, the teleoperated medical system may be under partial control of a computer programmed to perform the procedure or sub-procedure. In other alternative embodiments, a fully automated medical system may be used to perform the procedure or sub-procedure under the full control of a computer programmed to perform the procedure or sub-procedure. Figure 1A As shown, the teleoperated medical system 10 generally includes a teleoperated assembly 12 mounted to or near an operating table O on which a patient P is positioned. The teleoperated assembly 12 may be referred to as a patient-side cart. A medical instrument system 14 and an endoscopic imaging system 15 are operably coupled to the teleoperated assembly 12. An operator input system 16 allows a surgeon or other type of clinician S to view an image of the surgical site or an image representing the surgical site and to control the operation of the medical instrument system 14 and / or the endoscopic imaging system 15.

[0031] The operator input system 16 may be located at the surgeon's console, which is typically located in the same room as the operating table O. However, it should be understood that the surgeon S may be located in a different room from the patient P or in a completely different building. The operator input system 16 generally includes one or more input control devices for controlling the medical instrument system 14. The (one or more) input control devices may include one or more of any number of various devices, such as handles, joysticks, trackballs, data gloves, trigger guns, manually operated controllers, voice recognition devices, touch screens, pedals, body motion or presence sensors, eye gaze tracking devices, etc. In some embodiments, the (one or more) control devices will be provided with the same degrees of freedom as the medical instrument of the remote operating assembly to provide the surgeon with telepresence, the perception that the (one or more) control devices are integrated with the instrument, so that the surgeon has a strong sense of directly controlling the instrument as if it were present at the surgical site. In other embodiments, the (one or more) control devices may have more or fewer degrees of freedom than the associated medical instrument and still provide telepresence for the surgeon. In some embodiments, the control device(s) are manual input devices that move with six degrees of freedom, and the manual input device may also include an actuation handle for actuating the instrument (e.g., for closing grasping jaws, applying electrical potentials to electrodes, delivering drug therapy, etc.).

[0032] The teleoperation assembly 12 supports and manipulates the medical instrument system 14 while the surgeon S views the surgical site through the console 16. Images of the surgical site can be obtained through an endoscopic imaging system 15, such as a stereo endoscope, which can be manipulated by the teleoperation assembly 12 to orient the endoscope 15. Optionally, an electronic cart 18 can be used to process images of the surgical site for subsequent display to the surgeon S through the surgeon's console 16. Among other factors, the number of medical instrument systems 14 used at one time will generally depend on the diagnostic or surgical procedure and the space limitations in the operating room. The teleoperation assembly 12 may include a kinematic structure of one or more non-servo control links (e.g., one or more links that can be manually positioned and locked in place, generally referred to as a setup structure), and a teleoperation manipulator. The teleoperation assembly 12 includes a plurality of motors that drive inputs on the medical instrument system 14. These motors move in response to commands from a control system (e.g., the control system 20). The motors include a drive system that, when coupled to the medical instrument system 14, can propel the medical instrument into a natural or surgically created anatomical orifice. Other motorized drive systems can move the distal end of a medical device with multiple degrees of freedom, which can include three degrees of linear motion (e.g., linear motion along the X, Y, Z Cartesian axes) and three degrees of rotational motion (e.g., rotation about the X, Y, Z Cartesian axes). Additionally, the motor can be used to actuate an articulated end effector of the instrument for grasping tissue in the jaws of a biopsy device or the like.

[0033] The teleoperation medical system 10 also includes a control system 20. The control system 20 includes at least one memory and at least one processor (not shown), and typically includes multiple processors, for controlling between the medical device system 14, the operator input system 16, and the electronic system 18. The control system 20 also includes programmed instructions (e.g., a computer-readable medium storing instructions) to implement some or all of the methods described according to the aspects disclosed herein. Figure 1A The control system 20 is shown as a single box in the simplified schematic diagram of , but the system may include two or more data processing circuits, with one portion of the processing optionally being performed on or adjacent to the remote operation component 12, another portion of the processing being performed at the operator input system 16, etc. Any of a variety of centralized or distributed data processing architectures may be employed. Similarly, the programmed instructions may be implemented as many separate programs or subroutines, or the programmed instructions may be integrated into many other aspects of the remote operation system described herein. In one embodiment, the control system 20 supports wireless communication protocols such as Bluetooth, IrDA (Infrared Data Communication), HomeRF, IEEE 802.11, DECT, and wireless telemetry.

[0034] In some embodiments, the control system 20 may include one or more servo controllers that receive force and / or torque feedback from the medical device system 14. In response to the feedback, the servo controller transmits a signal to the operator input system 16. The servo controller(s) may also transmit signals that direct the remote operation assembly 12 to move the medical device system(s) 14 and / or the endoscopic imaging system 15, which extend into an internal surgical site within the patient's body via an opening in the body. Any suitable conventional or application specific servo controller may be used. The servo controller may be separate from the remote operation assembly 12 or integrated with the remote operation assembly 12. In some embodiments, the servo controller and the remote operation assembly are provided as part of a remote operation arm cart positioned adjacent to the patient's body.

[0035] The teleoperated medical system 10 may also include optional operating and support systems (not shown), such as lighting systems, steering control systems, irrigation systems, and / or suction systems. In alternative embodiments, the teleoperated system may include more than one teleoperated assembly and / or more than one operator input system. The exact number of manipulator assemblies will depend, among other factors, on the surgical procedure and the space limitations within the operating room. The operator input systems may be juxtaposed, or the operator input systems may be positioned in separate locations. Multiple operator input systems allow more than one operator to control one or more manipulator assemblies in various combinations.

[0036] Figure 1B is a perspective view of the surgeon's console 16. The surgeon's console 16 includes a left eye display 32 and a right eye display 34 for presenting the surgeon S with a coordinated stereoscopic view of the surgical site with depth perception. The console 16 also includes one or more input controls 36, which in turn cause the teleoperation assembly 12 to manipulate one or more instruments or endoscopic imaging systems. The input controls 36 can provide the same degrees of freedom as the instruments 14 with which they are associated to provide the surgeon S with a telepresence, or a sense that the input controls 36 are integral with the instruments 14 so that the surgeon has a strong sense of direct control of the instruments 14. For this purpose, position, force, and tactile feedback sensors (not shown) may be employed to transmit position, force, and tactile stimuli from the instruments 14 back to the surgeon's hands via the input controls 36.

[0037] Figure 1Cis a perspective view of the electronic cart 18. The electronic cart 18 can be coupled to the endoscope 15 and can include a processor to process the captured images for subsequent display to a surgeon, such as on a surgeon's console, or on another suitable display located locally and / or remotely. For example, in the case of a stereoscopic endoscope, the electronic cart 18 can process the captured images to present the surgeon with a coordinated stereoscopic image of the surgical site. Such coordination can include alignment between relative images and can include adjusting the stereo working distance of the stereoscopic endoscope. As another example, the image processing can include compensating for imaging errors of the image capture device, such as optical aberrations, using previously determined camera calibration parameters. The electronic cart 18 may also include components of a display monitor and control system 20.

[0038] Figure 1D 1 is a perspective view of one embodiment of a teleoperation assembly 12, which may be referred to as a patient-side cart. The patient-side cart 12 shown provides for manipulation of three surgical tools 26a, 26b, 26c (e.g., instrument system 14) and an imaging device 28 (e.g., endoscopic imaging system 15) such as a stereo endoscope for capturing images of the site of the procedure. The imaging device may transmit signals to the electronics cart 18 via cable 56. Manipulation is provided by a teleoperation mechanism having a plurality of joints. The imaging device 28 and the surgical tools 26a, 26b, 26c are positionable and maneuverable through an incision in the patient such that a kinematic telecenter is maintained at the incision to minimize the size of the incision. When the distal end of the surgical tool 26 is positioned within the field of view of the imaging device 28, an image of the surgical site can include an image of the distal end of the surgical tool 26. The teleoperation assembly 12 is located in a world coordinate system or frame F W The distal tip of each medical device 26a, medical device 26b, medical device 26c defines a corresponding device coordinate system or frame F I The distal tip of the imaging device 28 defines a coordinate system or frame F E .

[0039] The patient side cart 12 includes a drivable base 58. The drivable base 58 is connected to a telescopic column 57, which allows adjustment of the height of the arm 54a, arm 54b, arm 54c and arm 54d. Arm 54a can be referred to and / or labeled as "arm 1" in a surgical environment. Arm 54b can be referred to and / or labeled as "arm 2" in a surgical environment. Arm 54c can be referred to and / or labeled as "arm 3" in a surgical environment. Arm 54d can be referred to and / or labeled as "arm 4" in a surgical environment. Arm 54a can include a swivel joint 55 that rotates and moves up and down. Arm 54a is connected to a manipulator arm portion 51. Manipulator arm portion 51 can be directly connected to medical device 26a via a manipulator rod 59. Each of the other arms 54b, arm 54c, and arm 54d can have a configuration similar to arm 54a. Each of arm 54a, arm 54b, arm 54c, and arm 54d can be connected to an orientation platform 53. The orientation platform 53 may be capable of 360-degree rotation. The patient-side cart 12 may also include a telescopic horizontal cantilever 52 for moving the orientation platform 53 in a horizontal direction. The manipulator arm portion 51 may be remotely operable. Arm 54a, arm 54b, arm 54c, arm 54d may be remotely operable or not remotely operable. In an embodiment where the arm 54 is not remotely operable but the manipulator arm is remotely operable, the arm 54 is positioned as needed before the surgeon begins to operate with the system 10. Note that while the arm 54 (and associated manipulator arm portion 51) is depicted and described as part of a single patient-side cart 12 for exemplary purposes, in various other embodiments, the arm 54 and / or manipulator arm portion 51 (or additional arm 54 and / or manipulator arm portion 51) can be a discrete structure (e.g., a separate table-mounted arm, a ceiling-mounted arm, and / or a floor-mounted arm).

[0040] Endoscopic imaging systems (e.g., system 15, system 28) can be provided in various configurations including rigid or flexible endoscopes. A rigid endoscope includes a rigid tube that houses a lens relay system for transmitting images from the distal end of the endoscope to the proximal end of the endoscope. A flexible endoscope uses one or more flexible optical fibers to transmit images. An endoscope can be provided with different viewing angles, including a 0° viewing angle for forward axial viewing or a viewing angle between 0°-90° for forward oblique viewing. An endoscope based on digital images has a "chip on the tip" design in which a distal digital sensor such as one or more charge coupled devices (CCDs) or complementary metal oxide semiconductor (CMOS) devices stores image data. An endoscopic imaging system can provide a two-dimensional or three-dimensional image to an observer. A two-dimensional image can provide limited depth perception. A three-dimensional stereoscopic endoscopic image can provide a more accurate depth perception to an observer. A stereoscopic endoscopic instrument uses a stereo camera to capture a stereoscopic image of a patient's anatomical structure in the camera's field of view.

[0041] A stereoscopic image of the imaging system's field of view can be seen by the clinician through the eye display 32, eye display 34. In addition to patient tissue, the field of view may also include the distal end of the medical instrument and any accessories used to perform the surgical procedure. In order to perform the procedure, the clinician identifies the association between the instrument in the field of view and the input control device that controls the instrument from the console. The clinician can make the association by, for example, referring to alphanumeric text, symbols, or other information located at the periphery of the viewed image. However, in order to view the information, the clinician's attention must be moved from the central portion of the display and the distal tip of the instrument to the periphery of the image. Alternatively, the clinician can make the association by moving the input control device and observing the corresponding instrument movement in the image. This may be time-consuming and disruptive to the flow of the surgical procedure. In order to help the clinician associate the input control device 36 with the instrument visible in the field of view, information about the input control device can be displayed in a graphical form, and the information about the input control device is in the same position as the instrument it controls. Having the associated information in the same position as the instrument allows the clinician to quickly associate the control device with the instrument while maintaining attention on the surgical area of ​​interest.

[0042] Figure 2A A surgical environment 200 within a patient P is shown. An imaging instrument (e.g., imaging instrument 28) is used to display an image 202 of the field of view of the imaging instrument within the surgical environment 200. Image 202 may be a three-dimensional image of a composite image obtained by a stereoscopic endoscope and generated as a user-visible image through a right-eye display and a left-eye display. The field of view includes a distal portion of instrument 26a, a distal portion of instrument 26b, and a distal portion of instrument 26c. Image 202 also includes an information field 204 located at the periphery of the image, which may include instructions to the clinician, warnings, instrument identification information, status information, or other information related to the surgical procedure. In order to assist the clinician in performing the surgical procedure using system 10, various pieces of information are superimposed on image 202. In this embodiment, label 206 contains associated information related to medical instrument 26b, such as a numerical identifier of arm 54b to which the instrument is coupled (e.g., arm "3"). Label 206 is displayed close to medical instrument 204 so that the clinician can easily conclude that the associated information is related to medical instrument 26b. In some embodiments, the tag may also include associated information including the type of medical device, the status of the medical device (eg, cautery device charging / ready), etc. Figure 8-Figure 11 The method for creating and displaying the mark 206 is described in detail. In various embodiments, the mark can be circular, oval, square, or any suitable shape. In various embodiments, the mark can have a color, outline, shape, or other features for identifying the mark or distinguishing the mark.

[0043] The distal portion of the medical device 26b visible in the field of view includes the end effector tip 208, the shaft 210, and the joint area 212 between the shaft and the end effector tip. In some embodiments, the default location for placing the marker 206 is superimposed on the joint area 212. In other embodiments, the default location for placing the marker 206 may be on the end effector tip 208 or the shaft 210. In other embodiments, the marker 206 is displayed adjacent to the distal portion. The default location of the marker may depend on the size of the distal portion of the medical device 26b in the field of view. For example, if the image is zoomed in closely (e.g., Figure 3F ), and the end effector occupies a large portion of the image, the marker can be located on the proximal portion of one of the end effector jaws without blocking the view at the distal end of the jaw. However, if the image is zoomed out and the end effector jaws in the image are relatively small, the marker can be located in the joint area or on the shaft to avoid obscuring the jaws. The placement of the marker 206 allows the clinician to receive contextual information while maintaining focus on the instrument and the tissue being manipulated by the instrument. The placement of the marker 206 also avoids obscuring objects in the field of view that need to be observed by the clinician.

[0044] In some embodiments, the marker 206 has a center portion 214 and a track portion 216. The center portion 214 and the track portion 216 may contain different pieces of associated information. For example, as shown in FIG. 2 , the center portion 214 includes a number indicating the arm 54b to which the medical device 26b is attached (e.g., arm "3"). In various embodiments, the track portion 216 includes associated information related to a control device used by a clinician to control the medical device 26b. The associated information may be, for example, an indication of whether a right (e.g., "R") hand-operated input control device or a left (e.g., "L") hand-operated input control device is currently under the control of the medical device 26b to which the marker 206 is associated.

[0045] The track portion 216 of the marker 206 is movable circumferentially relative to the central portion 214 of the marker. The circumferential position of the track portion 216 indicates the rotational position of the control device, thereby letting the clinician know where their hand on the control device is positioned relative to the medical device 26b they are controlling. Figure 3A-Figure 3F The function of track portion 216 is shown in more detail in FIG.

[0046] Marker 207 is positioned proximate to instrument 26a and has similar features and attributes as marker 206. Marker 207 contains associated information related to medical instrument 26a, such as a numerical identifier of arm 54a to which the instrument is coupled (e.g., arm "1"). In this embodiment, and at this stage of the surgical procedure, instrument 26c is not currently under the control of the input control device. Thus, the marker is not positioned proximate to instrument 26c. In an alternative embodiment, marker information indicating a numerical identifier of arm 54c (e.g., arm "4") may be provided for instrument 26c, but the fact that the instrument is not currently under the control of the input device may be indicated by a color, symbol, or other indication.

[0047] Figure 2B A method 100 of operating a teleoperation system to perform a surgical procedure is shown. At process 102, the method includes displaying an image 202 of a field of view of a surgical environment. At process 104, a location of a medical instrument 26a in the surgical environment is determined. At process 106, associated information of the medical instrument 26a is determined. The associated information includes information about a teleoperation manipulator to which the medical instrument 26a is coupled and / or about operator input controls that control the medical instrument. At process 108, a marker is displayed proximate to the image of the medical instrument 26a in the image 202 of the field of view. The marker includes the associated information.

[0048] Figure 3A-3C The right hand 300 of the clinician is shown controlling the input control device 302 (i.e., the input control device 36). The input control device 302 controls the medical device 26c. Figure 3A As shown, from the clinician's point of view, the input control device 302 extends toward the right. Hand-related information including the (right or left) hand identification code and the orientation of the hand can be conveyed to the clinician via the markings. Although the control-related information is described in terms of a human hand, similar related information can be provided where control is delivered via a foot pedal, eye gaze tracker, voice command, or other clinician control. Figure 3D As shown, by orienting the track portion 222 of the marker 218 to the right (e.g., at approximately the 3 o'clock position) relative to the center portion 220 of the marker, Figure 3A The hand-related information of the control device 302 is visually transmitted to the clinician in the image 202. The track portion 222 displays "R" to indicate that the right hand control device 302 is being used to control the medical device 26c. Figure 3B As shown, from the clinician's point of view, the input control device is Figure 3A The position in the middle extends upward at approximately 90 degrees. Figure 3E As shown, by orienting the track portion 222 of the marker 218 upwardly (e.g., at approximately the 12 o'clock position) relative to the center portion 220 of the marker, Figure 3BThis orientation information of the control device 302 is visually transmitted to the clinician in the image 202. Figure 3C As shown, from the clinician's point of view, the input control device is Figure 3A The position in the middle extends downward at approximately 90 degrees. Figure 3F As shown, by orienting the track portion 222 of the marker 218 downwardly (e.g., at approximately the 6 o'clock position) relative to the center portion 220 of the marker, Figure 3C This orientation information of the control device 302 is visually conveyed to the clinician in the image 202 .

[0049] In some alternative embodiments, the hand-related information may be provided as a diagram or 3D model of the hand. The diagram or model depicts the left or right hand depending on whether the left or right hand control device is used to control the associated medical device. Figure 3D-Figure 3F As the orientation of the control device changes, the position of the illustration or model changes accordingly to visually convey information about the new orientation of the control device to the clinician. The characteristics and features described for any of the markers in the present disclosure may be applied to any of the other markers in the present disclosure.

[0050] Reference now Figure 4A-4B , shows further behavior of the marker 206 when the imaging device is moved or enlarged in the surgical environment. Figure 4A In the illustrated embodiment, the imaging device is moved or zoomed in toward medical device 26a, 26b, 26c. As medical device 26a, 26b, 26c appears larger in the field of view and image 202, markers 206, 207 become larger and change size to appear as if they are located at the same depth in the field of view as the distal end portion of the device. Figure 4B As the imaging device moves further away from or away from the medical device 26a, 26b, 26c, the markers 206, 207 become smaller, again changing to appear as if they are at the same depth in the field of view as the distal portion of the device. In some embodiments, there are limits on the maximum and minimum sizes of the markers 206, 207 to prevent the markers 206, 207 from becoming unreadable. Figure 8-Figure 11 The size of the markers 206, 207 may also inform the clinician of additional information, such as the distance between the medical device 26a, 26b and the imaging device. This may be helpful because different types of medical devices vary in size, and a small device close to the imaging device may appear to be the same size as a large device far away from the imaging device.

[0051] Since the markers 206 and 207 are superimposed on the image 202, the markers 206 and 207 can appear on the display even if the view of the medical device 26a and 26b is blocked by tissue, other devices, etc. Figure 5A As shown, the marker 206 remains visible even when the associated instrument 26b is hidden behind tissue. The presence of the marker 206 that is substantially offset from and at a different apparent size and depth than the other instruments 26a, 26c visible in the field of view indicates to the clinician that the instrument 26b associated with the marker is obscured. Figure 5B As shown, the marker 206 allows the obstructed instrument 26b to be found. Figure 5B In FIG. 2 , tissue has been moved and the medical device 26 b to which the marker 206 is attached has become visible in the field of view.

[0052] The teleoperated medical system 10 may have multiple operational states for each instrument. These may include an active state, in which the medical instrument is under the control of the input control device, allowing the clinician to actively control the medical instrument. In a disengaged state, the medical instrument is not under the control of the input control device and remains stationary, locked in place, via the manipulator arm to which it is coupled. The system may also have an engaged state, in which the user is prompted to engage or control the medical instrument via the associated input control device. In some embodiments, for a particular instrument, an indicia is displayed when the teleoperated medical system is in an engaged state. For example, in Figure 4A , instruments 26a and 26b are in an engaged state, wherein the user is prompted by the marking information to control instrument 26a on arm 1 with the left ("L") input control device, and to control instrument 26b on arm 3 with the right ("R") input control device. Instrument 26c is in a disengaged state, and therefore no marking is displayed for that instrument. During the active state, the marking may remain visible in the image, or may be removed to avoid distracting the clinician. Displaying the markings in the engaged state allows the clinician to identify the associated information and position of each instrument within the field of view 202 before controlling any of the medical instruments. For example, before controlling the instruments, the position of the markings can be used to determine whether any medical instrument is blocked from view by tissue. Additionally, the associated information contained in the markings can give the clinician information that allows him to choose which medical instrument to control to perform the desired procedure. In addition, the information given by the track portion of the markings can tell the clinician how to position the input control device before taking control so that the position matches the orientation of the medical instrument. In some embodiments, the system may require the clinician to match the position before allowing the state to change from engaged to active.

[0053] Reference now Figure 6A-6B, the distal portion of medical device 26a visible in the field of view includes end effector tip 230, shaft 232, and joint region 234 between the shaft and end effector tip. The distal portion of medical device 26c visible in the field of view includes end effector tip 240, shaft 242, and joint region 244 between the shaft and end effector tip. Fig. 6A As shown and previously described, the default location for placing the marker 207 is superimposed over the joint area 234. The default location for placing the marker 218 is superimposed over the joint area 244. Fig. 6A In the embodiment of the present invention, the distal end portions of the instruments are spaced apart from each other and are easily distinguishable from each other, and as a result, the markings 207 and 218 are clearly readable and clearly associated with the medical instruments 26a and 26c, respectively. Figure 6B The distal end portions of instruments 26a and 26c are shown overlapping or in close proximity in image 202. If allowed to remain in the default positions of markers 207, 218, markers 207, 218 would overlap or appear adjacent or superimposed on more than one instrument, thereby confusing the clinician. Figure 6BAs shown, in order to correct this problem, the mark 207 is moved by a displacement distance from its default position near the far end of the joint area 234 along the longitudinal axis A1 of the instrument 26a. In this embodiment, the mark 207 is relocated to the far end of the shaft 232 of the instrument 26a. Similarly, the mark 218 is moved by a displacement distance from its default position near the far end of the joint area 244 along the longitudinal axis A2 of the instrument 26c. In this embodiment, the mark 218 is relocated to the far end of the shaft 242 of the instrument 26c. The displacement distance can correspond to the position of the kinematic model of the calibration of the instrument coupled to the arm and the round rod, and / or can correspond to the size of the instrument distal part in the field of view 202. For example, the kinematic model of calibration (for example, based on the knowledge of the size of the parts in the manipulator joint position, kinematic sensor, and kinematic chain) provides the position and orientation of the joint area 234 of the instrument 26a in the surgical environment. Due to the uncertainty in the kinematic model, the uncertainty factor is associated with the determined position orientation of the joint area 234. Based on the kinematic model, the kinematic uncertainty of all instruments in the field of view, and the default position of the markers, the teleoperation system can determine a proximity factor (e.g., center-to-center or edge-to-edge distance, percentage of overlap) that indicates whether the markers overlap or whether the markers are close enough to each other so that if the markers remain in the default position, it will cause ambiguity to the clinician. If the proximity factor is less than the predetermined interference distance, the marker is adjusted as needed to increase the proximity factor to above the predetermined interference distance. For example, in some embodiments, the marker can be moved at least a minimum displacement distance along or across the axis of the instrument. In other embodiments, the marker can be resized or reshaped to maintain an acceptable interference distance while still providing the desired information content. In this way, the markers 207, 218 remain clearly associated with their corresponding medical instruments 26a, 26c, and also remain clearly readable.

[0054] Reference now Figure 7A-7C , a marker with its associated information may be particularly useful when the medical device approaches a boundary of the image 202 of the field of view or moves out of the image 202 . Fig. 7AThe displayed image 202 of the field of view of the imaging instrument in the surgical environment 200 is shown. The displayed image has a boundary 700. The distal portion of the medical instrument 26c, particularly the end effector tip 240, is within the boundary 700 and is visible in the displayed image 202. Determining whether the instrument 26c is inside or outside the field of view of the imaging instrument can be based on the calculated position of the end effector tip 240. Due to small error factors associated with the remote operation system, the instrument and / or the imaging system, determining the position of the tip 240 relative to the imaging instrument has an associated cumulative error factor. In order to avoid providing a false positive out of view indicator to the clinician, the determination of whether the instrument tip is outside the field of view of the imaging system can be biased by estimating a range of possible positions of the distal tip and suppressing the out of view indicator if any or a specified percentage of the estimated possible positions of the distal tip is within the field of view. The sensitivity of the bias can be adjusted based on the clinician's tolerance for false positive out of view indicators. A set of error bounding volumes 702 is associated with the tip 240. The bounding volume 702 may be graphically displayed in the image 202, or the bounding volume 702 may be associated with the tip 240 and not displayed. The error bounding volume may indicate that the predicted position of the distal and proximal ends of the tip portion is within a high degree of certainty (such as 90%-99%). The error bounding volume 702 represents the predicted position of the tip 240 of the instrument 26c. Additional description of error calculations and bounding volumes is provided in U.S. Provisional Application No. 61 / 954,442, filed on March 17, 2014, disclosing “Systems and methods for offscreen indication of instruments in a teleoperational medical system,” which is incorporated herein by reference in its entirety.

[0055] like Fig. 7A As shown, the bounding volume 702 is within the field of view defined by the boundary 700. The bounding volume 702 may or may not be presented in the image 202, but the position of the bounding volume 702 relative to the boundary may be determined by the system regardless of whether the bounding volume 702 is displayed. Because the medical device 26c is visible in the display and the bounding volume is within the boundary 700, the marker 218 is correspondingly displayed on the image 202 of the field of view. In order to maintain visibility and readability, even if the distal portion of the device 26c to which the marker 206 is attached is partially outside the boundary 700, the marker 218 remains completely inside the boundary 700, as shown here. This allows the clinician to clearly locate the medical device 26c, even if the medical device 26c is barely visible within the image 202. Figure 7B, the medical device 26c is not visible in the displayed image 202. In some embodiments, the system may terminate the display of the marker 218 when the associated medical device 26c is not visible in the image 202. Because the medical device 204 may be controlled by remote operation and not visible to the clinician in the field of view, unintentional movement of the device outside the field of view poses a safety risk. Additionally, the clinician may lose track of the device located outside the field of view. To minimize these risks, an indicator of the device outside the field of view may be presented visually or audibly to increase the clinician's awareness of the location of the device that is not visible in the field of view. However, in Figure 7B , bounding volume 702 for medical device 26c is within boundary 700, indicating that the position of tip 240 relative to boundary 700 is uncertain. Thus, while bounding volume 702 remains within boundary 700, marker 218 remains on image 202 even though image 202 does not appear to include tip 240.

[0056] Once the medical device 204 is sufficiently outside the boundary 700, and the boundary volume 702 is also outside the boundary, an out-of-view device indicator 704 is provided along the boundary 700 of the image 202 of the field of view, such as Figure 7C , to indicate that the medical device 26c is outside the field of view in the general direction of the indicator. In this embodiment, the indicator 704 is a graphical bar, but in other embodiments it may be a series of dots or icons, alphanumeric indicators. In addition to or in place of the visual indicator 704, an audible out-of-view indicator such as a beep or language-based instruction may alert the clinician that the medical device 204 is outside the field of view. The audible prompt may be swayed between the left or right speakers of the clinician's console to reinforce the position of the device relative to the field of view. Alternatively, the audible prompt may be emitted from the left or right speaker consistent with the left or right hand control associated with the device. In addition to or in place of the visual indicator 704, text information 706 related to the device outside the field of view may be provided to alert the clinician and / or provide identification information about the device or instructions for visualizing the device. In various embodiments, the out-of-view indicator 704 is mutually exclusive with the mark 218. When the marker 218 is displayed, the out of view indicator 704 is not displayed, and vice versa. This provides the clinician with constant awareness of the position of the medical device 26c.

[0057] In various embodiments, the use of the out of field of view indicator can be limited to avoid distracting the clinician. The use of the out of field of view indicator can be context-dependent, so that the out of field of view indicator can be displayed only during certain states of operation of the remote operation system. For example, the out of field of view indicator can be displayed during a state of the system in which the operator controls the movement of the imaging system (a state that can be referred to as the camera control state). As another example, the out of field of view indicator can be displayed when the system waits for input from the operator to control the associated instrument (described as the disengaged state above). As another example, the out of field of view indicator can be displayed within a few seconds after starting the state of the system in which the operator controls the movement of the instrument (described as the engaged state above). In other alternative embodiments, the out of field of view indicator can be disabled or selectively enabled when the clinician wants to understand the position of the out of field of view instrument. In some embodiments, the clinician must provide confirmation that the instrument tip is outside the field of view before enabling the operation of the out of field of view instrument. Additional warnings or confirmations can be used for energy emitting devices, sharp devices, or devices that provide increased patient risks if used without visualization. Further description of the out of field of view indicator system is provided in US Provisional Application No. 61 / 954,442, which is incorporated herein by reference above.

[0058] As explained above, the marker may be positioned and sized to correspond to the image of the distal end of the instrument with which the marker is associated. Figure 8-Figure 11 Further details for preserving this physical correspondence are provided. Figure 8 A method 800 of presenting a marker within an image 202 is shown. At process 802, the position of a distal portion of a medical device is projected into the coordinate space of the image 202. Fig. 9 A method 900 for implementing process 802 is shown. At process 902, the forward kinematics of the teleoperated assembly are evaluated to determine a world coordinate system (eg, surgical environment coordinate space F W ) within a surgical environment. More specifically, kinematic characteristics associated with a corresponding set of links and teleoperated manipulators for each medical instrument are determined in a world coordinate system. The world coordinate system is a set of Cartesian coordinates established with reference to a surgical environment in which the teleoperation system is located.

[0059] At process 904, the position and orientation of the distal portion of each medical device (eg, in the device coordinate space F I ) is mapped to the endoscope coordinate system (e.g., the imaging device coordinate space F E). The endoscope coordinate system is a set of Cartesian coordinates whose origin is established at the distal tip of the endoscope. By using the forward kinematic properties of the endoscope and the forward kinematic properties of each medical instrument in the world coordinate system, as determined in process 902, the distal position and orientation of the medical instrument relative to the position and orientation of the endoscope tip can be calculated in the endoscope coordinate system. Generally, the default position of the marker can be at or near the kinematic endpoint of the medical instrument.

[0060] At process 906, a calibrated camera model is obtained for the endoscope. The calibrated camera model is a mapping from the endoscope coordinate space to the right eye and left eye image coordinate spaces. In one embodiment, the mapping is accomplished through a series of three transformations. First, a model view transform maps from the endoscope coordinate space (at the distal tip of the endoscope) to the eye coordinate space of the endoscope, taking into account the binocular separation of the stereo camera in the imaging system, and taking into account differences in coordinate system specifications. Second, a perspective projection transform maps from the eye coordinate space to normalized coordinates (e.g., -1, +1), and takes into account the perspective view of the endoscope field of view and convergence distance. Third, a viewport bounds transform maps from the normalized eye coordinates to the right eye image coordinate space and the left eye image coordinate space. In one embodiment, these three transforms can be combined into a 4×4 uniform transform for each of the right eye image and the left eye image.

[0061] At process 908, the calibrated camera model is used to project points and vectors from the endoscope coordinate space to the image coordinate space of each eye. More specifically, the calibrated camera model projects the medical device distal end position and orientation to the right eye image coordinate space and the left eye image coordinate space to provide a stereoscopic 3D image to the clinician.

[0062] At process 910, uncertainties in the kinematic model of the teleoperation system are estimated relative to the endoscope tip, which leads to corresponding uncertainties in the position of the distal portion of the associated medical instrument. At process 912, these uncertainties are mapped to the right eye image coordinate space and the left eye image coordinate space from which the clinician views the three-dimensional image 202.

[0063] At process 914, the kinematic remote center (e.g., the incision site about which the instrument pivots) is mapped to the endoscope coordinate space using the forward kinematic properties determined at process 902. At process 916, the calibrated camera model is used to project the remote center to the image coordinate system of each eye in a manner similar to that described in process 908.

[0064] Processes 902-916 may be repeated for each medical instrument in the teleoperation system so that all medical instruments are correctly projected into the stereoscopic 3D image 202. The processes may be performed sequentially or in parallel, and one or more processes may be updated while others are not.

[0065] Reference again Figure 8 The presentation of the marker graphic is based on consideration of multiple display factors, including, for example, a default position at or near the distal tip of the distal end of the medical instrument, potential ambiguity associated with overlapping or uncertain marker placement, constraints imposed by the endoscope window, and system rules for hiding instruments. The position of the marker graphic can be determined in the image coordinate space of each eye. At process 804, potential overlap of the markers is checked and resolved to reduce the likelihood that the marker will be spatially associated with the wrong instrument. Fig.10 Shown for implementation Figure 8 Method 1000 of process 804 of the present invention. In this embodiment, at least two medical instruments are positioned in a surgical space. As described above, the default position for presenting a marker is at the distal portion of the medical instrument. At process 1002, the distance in image space coordinates between the distal portions of the two medical instruments is calculated. At process 1004, a threshold value for the minimum allowed inter-marker distance is calculated. The threshold value for the minimum allowed inter-marker distance is the maximum value of the sum of the projected radii of the markers (or, if the markers are not circular, another maximum dimension from the center of the markers) plus the maximum value of the sum of the projection uncertainties of the marker positions. At process 1006, the image space distance between the two distal portions is compared with the calculated threshold distance to determine whether their associated markers overlap or can be considered to overlap. If an overlap is detected, the direction of moving the marker to resolve the overlap is calculated at process 1008. The direction of moving the marker (as determined in process 908 and process 916) is determined by the unit vector between the distal portion of the instrument and the remote center projected into the image coordinate space. Additionally, if an overlap is detected, a distance to translate the markers to resolve the overlap is determined at process 1010. The distance of the marker offset is calculated as the difference between the minimum allowed inter-marker distance and the current distance between the distal portions. The offset prevents marker graphics from overlapping and provides marker positions that are sufficiently spaced apart to avoid ambiguity in instrument association.

[0066] Reference again Figure 8 At process 806, the marker is constrained to the viewable space of image 202, which corresponds to the boundaries of the endoscope window at the distal tip of the endoscope. The process can place the marker graphics within the boundaries of image 202 without clipping any of the graphics. The process can also control the perceived depth of the marker by preventing modification of horizontal parallax if the marker is shifted to avoid image boundaries or prevent ambiguity. Fig.11Shown for implementation Figure 8 Method 1100 of process 806 of the embodiment of the present invention. At process 1102, the right eye boundary of the visual space of the image 202 corresponding to the endoscope window is determined. At process 1104, the projected radius of the marker graphic is determined for the right eye. At process 1106, for the right eye, the boundary is inset by the projected marker radius to create an inset boundary. At process 1108, the Y position of the center of the marker is adjusted to remain within the inset boundary of the right eye. At process 1110, the offset of the X position of the center of the marker necessary to keep the marker completely within the inset boundary is calculated. For the left eye image, processes 1102 to 1110 are repeated. In order to maintain the proper apparent depth in stereoscopic 3D, the marker must be adjusted by the same amount in the right eye image and the left eye image. This maintains the horizontal parallax required for the proper perceived depth of the marker. At process 1112, the X offset calculated at process 1110 for one of the right eye image or the left eye image is applied to the X position of the marker in both the right eye image and the left eye image. In some embodiments, a larger X offset value is chosen to prevent the possibility that the marker graphic is even partially outside of the border 700 .

[0067] Reference again Figure 8 At process 808 , based on the calculations performed in process 802 , process 804 , and process 806 , the marker is presented within the left eye display and the right eye display to form the image 202 .

[0068] One or more elements of the embodiments of the present invention may be implemented in software to be executed on a processor of a computer system such as a control processing system. When implemented in software, the elements of the embodiments of the present invention are essentially code segments for performing the necessary tasks. The program or code segment can be stored in a processor-readable storage medium or device, and the program or code segment can have been downloaded by means of a computer data signal embodied in a carrier through a transmission medium or a communication link. The processor-readable storage device may include any medium capable of storing information, including optical media, semiconductor media, and magnetic media. Examples of processor-readable storage devices include electronic circuits; semiconductor devices, semiconductor memory devices, read-only memories (ROMs), flash memories, erasable programmable read-only memories (EPROMs); floppy disks, CD-ROMs, optical disks, hard disks, or other memory devices. The code segments may be downloaded via a computer network such as the Internet, an intranet, or the like.

[0069] Note that the processes and displays presented are not inherently related to any particular computer or other device. Various general purpose systems may be used with the programs according to the teachings herein, or it may prove convenient to construct more specialized devices to perform the described operations. The structures required for a variety of these systems will appear as elements in the claims. In addition, embodiments of the present invention are described without reference to any particular programming language. It should be understood that a variety of programming languages ​​may be used to implement the teachings of the present invention as described herein.

[0070] Although certain exemplary embodiments of the present invention have been described and shown in the accompanying drawings, it is to be understood that such embodiments are merely illustrative of the broad invention and not limiting thereof, and that the embodiments of the present invention are not limited to the specific constructions and arrangements shown and described, since various other modifications may occur to one of ordinary skill in the art.

Claims

1. A system for screen recognition, comprising: a teleoperation assembly comprising a first manipulator configured to control movement of a first medical device; as well as A processing unit, the processing unit comprising one or more processors, wherein the processing unit is configured to: an image showing a field of view of a surgical environment; displaying a first mark in the image of the field of view at a default position relative to the image of the first medical device, the first mark including first associated information for the first medical device; Determining a display factor associated with the first mark, wherein determining the display factor comprises: determining a distance between the first marker and a second marker displayed in the image of the field of view, the second marker comprising second association information for a second medical device; and identifying that the determined distance is less than a predetermined interference distance, wherein the determined distance is less than the predetermined interference distance at least before the first marker and the second marker overlap; In response to the display factor and the determined distance being less than the predetermined interference distance, determining a shift distance between the default position and the shifted position; and The first mark is displayed at the shifted position. 2 . The system of claim 1 , wherein determining the display factor further comprises determining whether the first marker overlaps with the second marker displayed in the image of the field of view. 3 . The system of claim 1 , wherein determining the display factor further comprises determining whether the first marker overlaps with an image of the second medical device in the image of the field of view.

4. The system of claim 1 , wherein determining the display factor further comprises determining whether there is a window constraint for an imaging device to capture the field of view of the first medical device or whether there is an obstruction of the image of the first medical device in the image of the field of view.

5. The system of claim 1, wherein the displacement distance is along a linear direction aligned with an axis of the first medical device.

6. The system of claim 1, wherein the shift distance is based on the predetermined interference distance between the first marker and the second marker displayed in the image of the field of view.

7. The system according to claim 1, wherein the processing unit is further configured to: Responsive to the display factor, determining a modified size for the first indicia; and The first marker is displayed using the modified size.

8. The system of claim 1, wherein the first marker comprises a static central portion and a track portion configured to move circumferentially about the static central portion.

9. The system according to claim 1, wherein the remote operation assembly further comprises a second manipulator configured to control movement of the second medical device; wherein the processing unit is further configured to display the second marker in the image of the field of view proximate to the image of the second medical device; and Wherein the displacement distance is along a linear direction aligned with an axis of the first medical device.

10. The system of claim 9, wherein the processing unit is further configured to shift the second marker in a linear direction aligned with an axis of the second medical device if the determined distance is less than the predetermined interference distance.

11. The system of claim 9, wherein determining the distance between the first marker and the second marker comprises determining a proximity factor corresponding to the first marker and the second marker.

12. The system of claim 11, wherein determining the proximity factor comprises at least one of determining a distance between a center of the first marker and a center of the second marker or determining a distance between an edge of the first marker and an edge of the second marker.

13. The system of claim 11, wherein determining the proximity factor comprises determining a percentage of overlap between the first marker and the second marker.

14. The system of claim 9, wherein the processing unit is further configured to adjust a size of the first marker if the determined distance is less than the predetermined interference distance.

15. The system of claim 9, wherein displaying the first marker comprises displaying the first marker superimposed on an image of a distal portion of the first medical device, and wherein displaying the second marker comprises displaying the second marker superimposed on an image of a distal portion of the second medical device.

Images