Methods and devices for manipulating tissue

Abstract

A method and apparatus for manipulating tissue. Displaying a tissue control point above an image of the tissue in a user interface. Receive input to move the tissue control point within the user interface. Operate a first instrument physically associated with the tissue based on the received input, thereby manipulating the tissue.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application No. 62 / 671,945, filed May 15, 2018, which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates to medical procedures and methods for manipulating tissue during medical procedures. More specifically, the present disclosure relates to systems and methods for manipulating tissue by controlling virtual tissue control points displayed in a user interface. Background Art

[0004] Minimally invasive medical techniques are intended to reduce the amount of extracorporeal tissue damaged during diagnostic or surgical procedures, thereby reducing the patient's recovery time, discomfort, and harmful side effects. Such minimally invasive techniques can be performed through natural orifices in the patient's anatomical structure or through one or more surgical incisions. Through these natural orifices or incisions, clinicians can insert medical tools to reach the target tissue location. Minimally invasive medical tools include instruments such as therapeutic instruments, diagnostic instruments, and surgical instruments. Minimally invasive medical tools can also include imaging instruments, such as endoscopic instruments, which provide users with a field of view within the patient's anatomical structure.

[0005] Some minimally invasive medical tools may be remotely controlled, or teleoperated, or computer-assisted. During a medical procedure, a clinician may need to manipulate tissue to retract tissue, expose a target area, inspect a hidden area of ​​tissue, or perform some other action. When manipulating tissue, the clinician may need to consider various parameters, including direction of motion, amplitude of motion, operator-instrument orientation, avoiding instrument collisions, multi-instrument interference, and range of motion limits. Systems and methods are needed to simplify and improve the clinician's experience in manipulating tissue. Summary of the invention

[0006] Embodiments of the invention are outlined by the appended claims following the detailed description.

[0007] In one example embodiment, a method for manipulating tissue is provided. In a user interface, a tissue control point is displayed above an image of the tissue. An input to move the tissue control point within the user interface is received. A first instrument physically associated with the tissue is operated based on the received input to manipulate the tissue.

[0008] In another example embodiment, another method for manipulating tissue is provided. In a user interface, a virtual object representing a tissue control point is displayed in juxtaposition with the tissue over an image of the tissue. The virtual object is offset from the positioning of a first instrument. A second instrument is operated to move a proxy geometry representing the second instrument into contact with the virtual object and to apply a force to the virtual object. The first instrument is operated based on the force applied to the virtual object to manipulate the tissue. A haptic feedback response is generated at the second instrument in response to the operation of the first instrument.

[0009] In yet another example embodiment, an apparatus includes a display system and a control system. The display system displays an image of tissue in a user interface. The control system is communicatively coupled to the display system. The control system is configured to display tissue control points above the image; receive input to move the tissue control points within the user interface; and operate an instrument physically associated with the tissue based on the received input to manipulate the tissue.

[0010] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory in nature, and are intended to provide an understanding of the present disclosure, but not to limit the scope of the present disclosure. In this regard, additional aspects, features and advantages of the present disclosure will be apparent to those skilled in the art based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] When read in conjunction with the accompanying drawings, various aspects of the present disclosure can be best understood from the following detailed description. It is emphasized that, in accordance with standard practice in the industry, various features are not drawn to scale. In fact, for clarity of discussion, the sizes of various features can be increased or reduced at will. In addition, the present disclosure may repeat reference numerals and / or letters in various examples. This repetition is for simplicity and clarity purposes and does not in itself indicate the relationship between the various embodiments and / or configurations discussed.

[0012] Figure 1A is a schematic diagram of a medical system according to an embodiment.

[0013] Figure 1B is a perspective view of an assembly according to an embodiment.

[0014] Figure 1C is a perspective view of a surgeon's console for a medical system according to an embodiment.

[0015] Figure 2 is a representation of tissue control points juxtaposed with tissue to be controlled using the tissue control points, according to an embodiment.

[0016] Figure 3 is according to an embodiment after the tissue has been manipulated based on the movement of the tissue control points Figure 7 A representation diagram of the organization.

[0017] Figure 4 is a representation of tissue control points represented as virtual objects or fixtures juxtaposed with tissue on which a medical procedure is performed, according to an embodiment.

[0018] Figure 5 is a flow chart illustrating a method for manipulating tissue, according to an embodiment.

[0019] Figure 6 is a flow chart illustrating a method for manipulating tissue, according to an embodiment.

[0020] Figure 7 is a flow chart illustrating a method for virtually juxtaposing a tissue control point with a tissue to be controlled, according to an embodiment.

[0021] Figure 8 is a flow chart illustrating a method for generating a haptic feedback response, according to an embodiment. DETAILED DESCRIPTION

[0022] In order to promote the understanding of the principles of the present disclosure, reference will now be made to the embodiments illustrated in the accompanying drawings, and specific language will be used to describe these embodiments. However, it will be understood that it is not intended to limit the scope of the present disclosure. In the following detailed description of various aspects of the present invention, many specific details are set forth to provide a thorough understanding of the disclosed embodiments. However, as will be appreciated by those skilled in the art, embodiments of the present disclosure may be practiced without these specific details. In other cases, well-known methods, procedures, components and circuits are not described in detail to avoid unnecessarily making various aspects of the embodiments of the present invention unclear.

[0023] As those skilled in the art involved in the present disclosure would normally expect, any changes and further modifications to the described equipment, apparatus, methods of the principles of the present disclosure and any further applications of the principles of the present disclosure are fully contemplated. In particular, it is fully contemplated that the features, components and / or steps described with respect to one embodiment can be combined with the features, components and / or steps described with respect to other embodiments of the present disclosure. In addition, the dimensions provided herein are for specific examples, and it is contemplated that different dimensions, sizes and / or ratios can be utilized to implement the concepts of the present disclosure. In order to avoid unnecessary descriptive repetitions, one or more components or actions described with respect to an illustrative embodiment can be used or omitted according to the applicability of other illustrative embodiments. For the sake of brevity, many iterations of these combinations will not be described separately. For the sake of simplicity, in some cases, the same reference numerals are used in all drawings to refer to the same or similar parts.

[0024] The following embodiments will describe various instruments and parts of instruments according to their states in three-dimensional space. As used herein, the term "positioning" refers to the position of an object or a portion of an object in three-dimensional space (e.g., three translational degrees of freedom along Cartesian X, Y, and Z coordinates). As used herein, the term "orientation" refers to the rotational placement of an object or a portion of an object (three rotational degrees of freedom - e.g., roll, pitch, and yaw). As used herein, the term "posture" refers to the positioning of an object or a portion of an object in at least one translational degree of freedom and the orientation of the object or a portion of an object in at least one rotational degree of freedom (up to six total degrees of freedom).

[0025] Now referring to the accompanying drawings, Figure 1A , Figure 1B and Figure 1C Together, an overview of a medical system 10 that can be used in a medical procedure, including, for example, a diagnostic, therapeutic, or surgical procedure, is provided. The medical system 10 is located in a medical environment 11. Figure 1A 1 and 2 depict medical environment 11 as an operating room. In other embodiments, medical environment 11 may be an emergency room, a medical training environment, a medical laboratory, or some other type of environment in which any number of medical procedures or medical training procedures may occur. In other embodiments, medical environment 11 may include an operating room and a control area located outside of the operating room.

[0026] In one or more embodiments, the medical system 10 may be a telesurgical medical system under the telesurgical control of a surgeon. In alternative embodiments, the medical system 10 may be under partial control of a computer programmed to perform a medical procedure or subroutine. In other alternative embodiments, the medical system 10 may be a fully automated medical system under the full control of a computer programmed to perform a medical procedure or subroutine with the medical system 10. One example of a medical system 10 that may be used to implement the systems and techniques described in the present disclosure is the DA manufactured by Intuitive Surgical, Inc. of Sunnyvale, California. Surgical system.

[0027] like Figure 1AAs shown, the medical system 10 generally includes an assembly 12, which can be mounted on an operating table O where a patient P is located or positioned near the operating table O. The assembly 12 can be referred to as a patient side cart, a surgical cart, or a surgical robot. In one or more embodiments, the assembly 12 can be a remotely operated assembly. The remotely operated assembly can be referred to as, for example, a remotely operated arm cart. A medical device system 14 and an endoscopic imaging system 15 are operably coupled to the assembly 12. An operator input system 16 allows a surgeon S or other type of clinician to view an image of the surgical site or an image representing the surgical site and control the operation of the medical device system 14 and / or the endoscopic imaging system 15.

[0028] The medical device system 14 may include one or more medical devices. In embodiments where the medical device system 14 includes multiple medical devices, the multiple medical devices may include multiple identical medical devices and / or multiple different medical devices. Similarly, the endoscopic imaging system 15 may include one or more endoscopes. In the case of multiple endoscopes, the multiple endoscopes may include multiple identical endoscopes and / or multiple different endoscopes.

[0029] The operator input system 16 may be located at a surgeon's console, which may be located in the same room as the operating table O. In some embodiments, the surgeon S and the operator input system 16 may be located in a different room than the patient P or in a completely different building. The operator input system 16 generally includes one or more control devices for controlling the medical instrument system 14. The one or more control devices may include any number or more of a variety of input devices, such as a handle, a joystick, a trackball, a data glove, a trigger gun, a foot pedal, a hand controller, a voice recognition device, a touch screen, a human motion or presence sensor, and other types of input devices.

[0030] In some embodiments, one or more control devices are provided with the same degrees of freedom as one or more medical devices of the medical device system 14 to provide the surgeon with a sense of telepresence that the one or more control devices are integral with the device, so that the surgeon has a strong sense of direct control of the device, just as if he were at the surgical site. In other embodiments, one or more control devices may have more or fewer degrees of freedom than the associated medical device and still provide telepresence to the surgeon. In some embodiments, one or more control devices are manual input devices that move with six degrees of freedom and may also include an actuatable handle for actuating the device (e.g., a jaw end effector for closing a grasp, applying an electrical potential to an electrode, providing medication, and actuating other types of devices).

[0031] When the surgeon S observes the surgical site through the operator input system 16, the assembly 12 supports and manipulates the medical device system 14. Images of the surgical site can be obtained by the endoscopic imaging system 15, which can be manipulated by the assembly 12. The assembly 12 may include the endoscopic imaging system 15, and may also include multiple medical device systems 14. The number of medical device systems 14 used at one time will generally depend on factors such as the diagnostic or surgical procedures to be performed and the space limitations in the operating room. The assembly 12 may include a kinematic structure of one or more non-servo-controlled links (e.g., one or more links that can be manually positioned and locked in place, generally referred to as mounting structures) and a manipulator. When the manipulator takes the form of a remote manipulator, the assembly 12 is a remote assembly. The assembly 12 includes multiple motors that drive inputs on the medical device system 14. In an embodiment, these motors move in response to commands from a control system (e.g., control system 20). The motor includes a drive system that, when coupled to the medical device system 14, can advance the medical device into an anatomical orifice created naturally or surgically. Other electric drive systems can move the distal end of a medical device in multiple degrees of freedom, which can include three degrees of linear motion (e.g., linear motion along the X, Y, Z Cartesian axes) and three degrees of rotational motion (e.g., rotation about the X, Y, Z Cartesian axes). In addition, the motor can be used to actuate an articulated end effector of a medical device to grasp tissue in the jaws of a biopsy device, etc. The medical devices of the medical device system 14 can include an end effector having a single working member, such as a scalpel, a blunt knife, an optical fiber, or an electrode. Other end effectors can include, for example, forceps, graspers, scissors, or clip applicators.

[0032] The medical system 10 also includes a control system 20. The control system 20 includes at least one memory 24 and at least one processor 22 for implementing control between the medical device system 14, the operator input system 16, and other auxiliary systems 26, which may include, for example, imaging systems, audio systems, fluid delivery systems, display systems, lighting systems, steering control systems, irrigation systems, and / or suction systems. The clinician can circulate within the medical environment 11 and can access, for example, the assembly 12 during the setup process or view the display of the auxiliary system 26 from the patient's bedside.

[0033] Despite Figure 1A While the control system 20 is depicted as being external to the assembly 12, in some embodiments, the control system 20 may be completely contained within the assembly 12. The control system 20 also includes programmed instructions (e.g., stored on a non-transitory computer readable medium) to implement some or all of the methods described in accordance with various aspects disclosed herein. Figure 1AThe control system 20 is shown as a single box in the simplified schematic of , but the control system 20 may include two or more data processing circuits, where part of the processing is optionally performed on or near the assembly 12, another part of the processing is performed on the operator input system 16, and so on.

[0034] Any of a variety of centralized or distributed data processing architectures may be employed. Similarly, the programmed instructions may be implemented as a plurality of separate programs or subroutines, or they may be integrated into many other aspects of the systems described herein, including remote operating systems. In one embodiment, the control system 20 supports wireless communication protocols such as Bluetooth, IrDA, HomeRF, IEEE 802.11, DECT, and wireless telemetry.

[0035] The control system 20 is in communication with a database 27, which may store one or more clinician profiles, a list of patients and patient profiles, a list of procedures to be performed on the patients, a list of clinicians scheduled to perform the procedures, other information, or a combination thereof. The clinician profile may include information about the clinician, including how long the clinician has worked in the medical field, the level of education the clinician has obtained, the clinician's level of experience with the medical system 10 (or similar systems), or any combination thereof.

[0036] The database 27 may be stored in the memory 24 and may be dynamically updated. Additionally or alternatively, the database 27 may be stored on a device such as a server or a portable storage device that may be accessed by the control system 20 via an internal network (e.g., a secure network of a medical facility or a remote operating system provider) or an external network (e.g., the Internet). The database 27 may be distributed across two or more locations. For example, the database 27 may be present on multiple devices that may include devices of different entities and / or cloud servers. Additionally or alternatively, the database 27 may be stored on a portable user-assigned device such as a computer, mobile device, smart phone, laptop, electronic badge, tablet computer, pager, and other similar user devices.

[0037] In some embodiments, the control system 20 may include one or more servo controllers that receive force and / or torque feedback from the medical device system 14. In response to the feedback, the servo controller sends a signal to the operator input system 16. The one or more servo controllers may also send a signal to instruct the assembly 12 to move the medical device system 14 and / or the endoscopic imaging system 15, which extends to an internal surgical site in the patient's body via an opening in the body. Any suitable conventional or dedicated servo controller may be used. The servo controller may be separated from the assembly 12 or integrated with the assembly 12. In some embodiments, the servo controller and the assembly 12 are provided as part of a remote control arm cart positioned near the patient's body.

[0038] The control system 20 can be coupled to the endoscopic imaging system 15 and can include a processor to process the captured images for subsequent display, such as to a surgeon at a surgeon's console, or on other suitable displays located locally and / or remotely. For example, where a stereoscopic endoscope is used, the control system 20 can process the captured images to present a coordinated stereoscopic image of the surgical site to the surgeon. Such coordination can include alignment between relative images and can include adjusting the stereo working distance of the stereoscopic endoscope.

[0039] In alternative embodiments, the medical system 10 may include more than one assembly 12 and / or more than one operator input system 16. The exact number of assemblies 12 will depend on the surgical procedure and the space limitations in the operating room, as well as other factors. The operator input systems 16 may be collocated, or they may be located in separate locations. Multiple operator input systems 16 allow more than one operator to control one or more assemblies 12 in various combinations. The medical system 10 may also be used for training and rehearsing medical procedures.

[0040] Figure 1B1 is a perspective view of an embodiment of an assembly 12, which can be referred to as a patient side cart, a surgical cart, a remote operating arm cart or a surgical robot. The assembly 12 shown provides manipulation of three surgical tools 30a, 30b and 30c (e.g., medical instrument system 14) and an imaging device 28 (e.g., an endoscopic imaging system 15), which is, for example, a stereo endoscope for capturing images of the surgical site. The imaging device can send a signal to the control system 20 via a cable 56. Manipulation is provided by a remote operating mechanism with multiple joints. Imaging device 28 and surgical tools 30a-30c can be positioned and manipulated through an incision in the patient, thereby keeping the kinematic remote center at the incision to minimize the size of the incision. When the distal end of the surgical tool 30a-30c is positioned in the field of view of the imaging device 28, the image of the surgical site can include the image of the distal end of the surgical tool 30a-30c.

[0041] The assembly 12 includes a drivable base 58. The drivable base 58 is connected to a telescopic column 57, which allows the height of the arm 54 to be adjusted. The arm 54 may include a rotary joint 55 that both rotates and moves up and down. Each arm 54 may be connected to an orientation platform 53. The arm 54 may be marked to facilitate troubleshooting. For example, each arm 54 may be decorated with different numbers, letters, symbols, other identifiers or combinations thereof. The orientation platform 53 may be able to rotate 360 ​​degrees. The assembly 12 may also include a telescopic horizontal cantilever 52 for moving the orientation platform 53 in the horizontal direction.

[0042] In this example, each arm 54 is connected to a manipulator 51. The manipulator 51 can be directly connected to a medical instrument, such as one of the surgical tools 30a-30c. The manipulator 51 can be remotely controlled. In some examples, the arms 54 connected to the orientation platform 53 may not be remotely controlled. Instead, these arms 54 can be positioned as needed before the surgeon S begins to operate with the remotely operated components. Throughout the surgical procedure, medical instruments may be removed and replaced with other instruments, so that the association of instruments with arms may change during the procedure.

[0043] Endoscopic imaging systems (e.g., endoscopic imaging system 15 and imaging device 28) can be provided in various configurations including rigid or flexible endoscopes. Rigid endoscopes include a rigid tube that houses a relay lens system for transmitting images from the distal end of the endoscope to the proximal end. Flexible endoscopes use one or more flexible optical fibers to transmit images. Endoscopes based on digital images have a "chip-on-the-tip" design in which a distal digital sensor (e.g., one or more charge-coupled devices (CCDs) or complementary metal oxide semiconductor (CMOS) devices) stores image data. Endoscopic imaging systems can provide two-dimensional or three-dimensional images to a viewer. Two-dimensional images can provide limited depth perception. Three-dimensional stereoscopic endoscopic images can provide a more accurate depth perception for the viewer. Stereoscopic endoscopic instruments use stereo cameras to capture stereoscopic images of patient anatomical structures. Endoscopic instruments can be fully sterilizable assemblies with endoscope cables, handles, and shafts that are securely coupled and hermetically sealed.

[0044] Figure 1C is a perspective view of an embodiment of an operator input system 16 at a surgeon's console. The operator input system 16 includes a left-eye display 32 and a right-eye display 34 for presenting a coordinated stereoscopic view of the surgical environment enabling depth perception to the surgeon S. The left-eye display 32 and the right-eye display 34 may be components of a display system 35. In other embodiments, the display system 35 may include one or more other types of displays.

[0045] The operator input system 16 also includes one or more input control devices 36, which in turn enable the assembly 12 to manipulate one or more instruments of the endoscopic imaging system 15 and / or the medical instrument system 14. The input control devices 36 can provide the same degrees of freedom as the instruments they are associated with, to provide telepresence to the surgeon S, or the perception that the input control devices 36 are integral with the instruments, so that the surgeon has a strong sense of direct control of the instruments. To this end, position sensors, force sensors, and tactile feedback sensors (not shown) can be used to transmit the position, force, and tactile sensation of medical instruments such as surgical tools 30a-30c or imaging devices 28 back to the surgeon's hands through the input control device 36. The input control device 37 is a foot pedal that receives input from the user's foot. Various aspects of the operator input system 16, the assembly 12, and the auxiliary system 26 can be adjustable and customizable to meet the physical needs, skill level, or preferences of the surgeon S.

[0046] During a medical procedure performed using the medical system 10, the surgeon S or other clinician may need to manipulate tissue to retract the tissue, expose a target area, inspect a hidden area of ​​the tissue, or perform some other action. For example, the surgeon S may need to use the surgical tool 30a to retract the tissue, but the instrument may be partially or completely out of the field of view of the endoscopic imaging system 15. Therefore, the surgeon S may not be able to easily observe whether the movement of the surgical tool 30a will result in a collision with one of the surgical tools 30b, 30c, or the manipulator 51. In addition, the surgeon S may need to twist his or her wrist in order to control the surgical tool 30a. Therefore, it may be desirable to have a method and system that can improve the surgeon S's experience in manipulating tissue during a medical procedure.

[0047] The various embodiments described below provide methods and systems that allow surgeon S to more easily and directly manipulate tissue within the field of view of endoscopic imaging system 15 using an instrument (e.g., one of surgical tools 30a, 30b, or 30c). In one or more embodiments, display system 35 may display tissue control points (e.g., the following control points) above an image representing the field of view of endoscopic imaging system 15. Figure 2 The surgeon S may manipulate the tissue control points 200 using the operator input system 16, and the control system 20 may process the input to control the operation of the instrument. The use of the tissue control points 200 simplifies the steps required by the surgeon S to manipulate tissue in a desired manner. In addition, the use of the tissue control points 200 allows the surgeon S to control the operation of the instrument, thereby manipulating tissue in the field of view of the endoscopic imaging system 15 even when the instrument is partially in the field of view or completely out of the field of view.

[0048] Figure 2 is a representation of a tissue control point 200 juxtaposed with tissue 202 to be controlled using the tissue control point 200. The diagram depicts a boundary 203 representing the field of view of the endoscopic imaging system 15. The field of view will be displayed in a user interface on the display system 35 as an image of the tissue 202. The image may be an image of the endoscopic imaging system 15 or an image representing the field of view of the endoscopic imaging system 15.

[0049] The figure further depicts instrument 204, instrument 206, and instrument 208, each of which is engaged with tissue. Each of instrument 204, instrument 206, and instrument 208 may be an example of a type of instrument that may be in a medical device system, such as medical device system 14 in FIG. 1. For example, in one embodiment, instrument 204, instrument 206, and instrument 208 may be Figure 1B204 can be used to manipulate tissue 202 during a medical procedure. Instrument 204 can be used to perform tasks such as retraction, withdrawal, or a combination thereof. For example, instrument 204 can be implemented as a retractor, grasper, forceps, clamps, or some other type of auxiliary instrument.

[0050] The tissue control point 200 can be located relative to the image of the tissue. Figure 1C The organization control point 212 can be moved within a user interface displayed on the display system 35 of the display system 35 and can be moved with a selected number of degrees of freedom. For example, the organization control point 200 can be translatable, rotatable, or both. In one or more embodiments, the organization control point 200 is implemented as a graphical element (e.g., a movable indicator) indicating the number of degrees of freedom in which the organization control point 212 can move. In some embodiments, the organization control point 212 can be translated or rotated in any direction relative to the user interface 210.

[0051] In one embodiment, the tissue control point 200 is represented using a four-headed arrow cursor. The four-headed arrow cursor indicates that the tissue control point 200 can be moved in four translation directions (e.g., left, right, up, and down). The tissue control point 200 can be used to control an instrument 204. As shown, the instrument 204 can be located outside the boundary 203 representing the field of view of the endoscopic imaging system 15. The tissue control point 200 enables the surgeon S or other clinician to control the operation of the instrument 204 even when the instrument 204 is not visible in the field of view and therefore not displayed in the user interface. There is an offset 210 between the tip 212 and the tissue control point 200. The offset 210 is represented by a dotted line extending between the tip 212 of the instrument 204 and the tissue control point 200.

[0052] Figure 3 After the tissue 202 has been manipulated based on the movement of the tissue control point 200 Figure 7 202 of tissue. As shown, tissue control point 200 has been translated upward. Based on this movement, control system 20 operates instrument 204 to cause corresponding movement of instrument 204, thereby manipulating tissue.

[0053] Figure 4 is a representation of a tissue control point 200 represented as a virtual object 400 (or virtual fixture) juxtaposed with a tissue 202 to be controlled. Figure 4, the virtual object 400 is depicted as a graphical line element. The proxy geometry 402 of the instrument 206 may be known to the control system 20. The proxy geometry 402 may indicate the geometry of the jaws of the instrument 206, the tip of the instrument 206, or some other portion of the instrument 206. In this embodiment, the proxy geometry 402 indicates the geometry of the jaws of the instrument 206. The distance 404 is represented by a dashed line extending between the proxy geometry 402 and the virtual object 400. When the proxy geometry 402 and the virtual object 400 are not in contact, the distance 404 represents the closest distance between the proxy geometry 402 and the virtual object 400. When the proxy geometry 402 is in contact with the virtual object 400, the distance 404 represents the penetration depth that can be used to express the restoring force.

[0054] Figure 5 is a flow chart of a method 500 for manipulating tissue. The method 500 Figure 3 It is described as a set of operations or processes 502 to 508, and continues to refer to Figure 1A , Figure 1B , Figure 1C and Figure 2 In all embodiments of the method 500, not all of the illustrated processes 502 to 508 are performed. Figure 3 One or more processes not explicitly described in the description may be included before, after, during, or as part of processes 502 to 508. In some embodiments, one or more of processes 502 to 508 may be implemented at least in part in the form of executable code stored on a non-transitory tangible machine-readable medium, which when executed by one or more processors (e.g., processors of a control system) may cause the one or more processors to perform the one or more processes.

[0055] In process 502, tissue control points 200 are displayed over an image of tissue 202 in a user interface. Tissue control points 200 may be graphical indicators that allow a user (e.g., surgeon S) to control an instrument 204 engaged with a patient's tissue 202 during a medical procedure, thereby manipulating tissue 202. Instrument 204 engages tissue 202 by being physically associated with tissue 202. For example, the instrument may be touching tissue 202, grasping tissue 202, retracting tissue 202, applying force to tissue 202, otherwise engaging tissue 202, or a combination thereof. The image of tissue 202 may be displayed by, for example, Figure 1A The image may be provided by the endoscopic imaging system 15. In other words, the image may present the field of view of the endoscopic imaging system 15.

[0056] The tissue control point 200 is displayed above the image at a selected location above the tissue 202 in the image. In this manner, the tissue control point 200 can be considered to be virtually juxtaposed with the tissue 202. The selected location for the tissue control point 200 can be established based on the positioning of the instrument 204 relative to the tissue. In some cases, the instrument 204 may be visible in the image. In other cases, the instrument 204 may not be visible in the image. In other words, the instrument 204 may not be within the field of view of the endoscopic imaging system 15. In the event that the instrument 204 is not within the field of view, the selected location for the tissue control point 200 relative to the image can be offset from the positioning of the instrument 204 or the tip 212 of the instrument 204 relative to the tissue 202.

[0057] In process 504, an input is received that moves the tissue control point 200 within the user interface. The input may be received by the input control device 36 and processed by the control system 20. In some embodiments, the input device 208 may be a joystick. In other embodiments, the input control device 36 may include at least one of a touch screen, a gesture tracking system, a gaze tracking system, a hand control device, a remote control device, a mouse, or some other type of input device such as for input devices 36, 37.

[0058] Based on the received input, the control system 20 can move the tissue control point 200. The tissue control point 200 can have a selected number of degrees of freedom. The number of degrees of freedom selected can be task specific based on the type of corresponding movement that needs to be provided for the instrument 204. The human arm is considered to have seven degrees of freedom. In one embodiment, the number of degrees of freedom selected for the tissue control point 200 is less than the seven degrees of freedom provided by the human arm, thereby simplifying the user interaction required to achieve the desired movement of the instrument 204.

[0059] In one embodiment, the movement of the tissue control point 200 may include a left translation, a right translation, an upward translation, a downward translation, an inward translation, an outward translation, a rotation, or a combination thereof of the tissue control point. Figure 3 The translation of the tissue control point 200 depicted in is an example of one type of movement of the tissue control point 200 based on the received input. In some embodiments, the movement of the tissue control point 200 can include multiple translations in varying directions and / or in the same direction. In other embodiments, the tissue control point 200 can be only translatable, only translatable along one axis, only rotatable, or limited in movement in some other way. In one embodiment, when the instrument 204 has jaws that grasp the tissue 202, the movement of the tissue control point 200 may be limited so that the tissue control point 200 cannot cause the jaws to open and release the tissue 202.

[0060] In process 506, an instrument 204 physically associated with the tissue 202 is operated based on the received input, thereby manipulating the tissue 202. In one embodiment, operating the instrument 204 includes transforming the movement of the tissue control point 200 into a corresponding movement of the instrument 204, thereby manipulating the tissue 202. In addition to the movement of the tissue control point 200, this transformation may also take other factors into account.

[0061] For example, the transformation may include optimizing the movement of the instrument 204 based on one or more secondary goals. These secondary goals may include optimizing the speed of movement, ensuring that the movement is within the selected motion limits of the instrument 204, avoiding collisions with one or more other instruments or structures, creating sufficient working space between the instrument 204 and any adjacent instruments, avoiding selected areas or zones (e.g., prohibited areas), or a combination thereof. In one or more embodiments, the control system 20 determines the positioning of the instrument 204 and a manipulator (e.g., manipulator 51) connected to the instrument relative to other instruments and manipulators. The control system 20 can calculate a movement path for the instrument 204 that corresponds to the movement of the tissue control point 200 and prevents interaction between different instruments and manipulators. The movement path may include any number of translational movements, rotational movements, or combinations thereof.

[0062] In some embodiments, the control system 20 can identify operating parameters of the instrument 204. These operating parameters can include, for example, the geometry of the instrument 204, the minimum movement speed, the maximum movement speed, the range of motion, the number of degrees of freedom, other types of parameters, or combinations thereof. The control system 20 can take these optional parameters into account to calculate the movement path of the instrument 204. In addition, the control system 20 can calculate the movement path of the instrument 204 to ensure that the instrument 204 does not enter a selected area or zone (e.g., a prohibited area).

[0063] In some embodiments, the control system 20 may impose a limit on the amount of force that the instrument 204 is allowed to apply. For example, the instrument 204 is typically capable of applying approximately 2 pounds of force at the tip 212 of the instrument 204. However, the control system 20 may limit the amount of force that can be applied at the tip 212 of the instrument 204 to approximately 0.5 pounds of force.

[0064] In one or more embodiments, the operation of the instrument 204 in the process 506 manipulates the tissue 202 by causing a corresponding movement of the tissue 202, which achieves the intended movement of the instrument 204 as well as a secondary goal. The corresponding movement of the tissue 202 can be, for example, a retraction of the tissue 202, a translation of the tissue 202, a twisting of the tissue 202, a rotation of the tissue 202, a deformation of the tissue 202, or a combination thereof. In this manner, the user, via the input control device 36, can transform the movement of the tissue control point 200 into a corresponding movement of the instrument 204, which causes the tissue 202 engaged with or proximate to the instrument 204 to be retracted, twisted, rotated, lifted, pushed down, pulled down, lifted up, moved to the side, moved upward, deformed, and / or otherwise manipulated.

[0065] In some embodiments, the control system 20 can use imaging data or sensor data to observe the movement of the tissue 202 around the tissue control point 200, and can update one or more control laws used to control the movement of the instrument 204 based on the movement of the tissue control point 200 to reduce errors in the observed movement of the tissue 202. In other words, the control system 20 can use feedback in the form of imaging data or sensor data to reduce errors in the actual movement of the tissue 202 relative to the expected movement of the tissue 202 based on the movement of the tissue control point 200.

[0066] In process 508, which may be optional, a tactile feedback response is generated in response to the operation of the instrument 204. The tactile feedback response, which may also be referred to as tactile communication or a kinesthetic response, may be a physical or mechanical stimulation by applying force, vibration, motion, or a combination thereof to the user. The tactile feedback response may be generated based on the physical effects of the operation of the instrument 204 on the tissue 202, and may allow the user to receive information from the control system 20 in the form of sensations on certain parts of the body. The tactile feedback response may, for example, allow the user to experience the stiffness, rigidity, or deformability of the tissue 202. In some embodiments, the tactile feedback response may allow the user to experience the traction and resistance of the tissue 202 to movement.

[0067] A tactile feedback response may be generated using a haptic device that generates a sense of touch that can be felt by a user. The tactile device may include at least one of a remote control device, a joystick, a glove, some other type of hand control device, some other type of tactile sensation generating device, or a combination thereof. In some embodiments, the tactile device may be an input control device 36. For example, the input control device 36 may reflect forces and torques generated from virtual-physical interactions by physical forces. Virtual-physical interactions may be constraints encountered, for example, due to contact with a virtual surface, a virtual line, a virtual point, or a combination thereof.

[0068] Figure 6600 is an illustration of a method 600 for manipulating tissue. The method 600 is illustrated as a set of operations or processes 602 to 608 and continues with reference to Figure 1A , Figure 1B , Figure 1C and Figure 2 In all embodiments of method 600, not all of the illustrated processes 602 to 608 are performed. Figure 6 One or more processes not explicitly described in the description may be included before, after, between, or as part of processes 602 to 608. In some embodiments, one or more of processes 602 to 608 may be implemented at least in part in the form of executable code stored on a non-transitory tangible machine-readable medium, which, when executed by one or more processors (e.g., a processor of a control system), may cause the one or more processors to perform one or more of the processes.

[0069] In process 602, an initial position of a tissue control point 200 is virtually determined. Determining the initial position includes establishing a relationship between the tissue 202 to be controlled, the tissue control point 200, and the instrument 204. Specifically, determining the initial position includes establishing a relationship between the tissue 202 to be controlled, the tissue control point 200, and the tip or end effector of the instrument 204.

[0070] In one embodiment, the initial position of the tissue control point 200 can be determined or selected by the user. For example, the user can enter an initial input for determining the tissue control point 200. The user can select the initial position using, for example, the tip 212 of the instrument 204 or some other remotely operated instrument or tool to contact a position on the surface of the tissue 202, and then engage the control. Engaging the control can include, for example, pressing a button on the input control device 36. The initial position of the tissue control point can then be determined based on the position on the surface where the tip 212 of the instrument 204 has engaged with the tissue 202. In some embodiments, the instrument 204 can be used to grasp the tissue 202. The control system 20 can use the grasping force that commands the instrument 204 to grasp the tissue 202 as a signal to establish and lock the offset between the tip 212 of the instrument 204 and the tissue control point 200.

[0071] In response to the control being engaged, the control system 20 can then create and display the tissue control point 200 at the corresponding virtual position above the image. In other embodiments, the user can use the input control device 36 to manually move the tissue control point 200 in the user interface to a selected position on the image. In other embodiments, the input control device 36 can track or detect the user's gaze to determine where to position the tissue control point 200. For example, the control system 20 can establish the tissue control point 200 at a location on the surface of the tissue 202 where it is detected that the user's gaze has been directed for a selected period of time. The gaze input from the user's left and right eyes can be triangulated to produce a three-dimensional fixed position.

[0072] In one embodiment, the initial position or tissue control point 200 can be determined by the control system 20. For example, the control system 20 can generate a sparse or dense three-dimensional surface reconstruction of the surface of the tissue 202 based on imaging data received from an imaging system (e.g., a laser imaging system). The control system 20 can then identify a location on the surface of the tissue 202 that is connected to or engaged by an instrument 204 (e.g., a tip or end effector of the instrument 204). The control system 20 can identify a candidate point on the surface of the tissue 202 that is, for example, centrally located and furthest forward in the view, and can then select an initial position for the tissue control point 200 based on the candidate point. The tissue control point 200 can be selected at a location that is visible in the field of view of the endoscopic imaging system 15, is not obscured by other instruments, and is in a position that is close to or optimally centered relative to the control instrument 204.

[0073] In other embodiments, the three-dimensional surface reconstruction of the surface of the tissue 202 described above can be used in conjunction with the gaze detected from a single eye. For example, the vector of the gaze detected from a single eye can be intersected with the three-dimensional surface reconstruction to determine the location of the tissue control point 200.

[0074] In some embodiments, the initial position of the tissue control point 200 can be calculated from the color / depth segmentation of the image provided by the endoscopic imaging system 15. For example, the control system 20 can segment the region of the image using color image segmentation or depth image segmentation. The control system 20 can then calculate the centroid of the segmented region of the image as the initial position of the tissue control point 200.

[0075] In other embodiments, a two-dimensional position may be determined for the tissue control point 200 within the image. The two-dimensional position may then be mapped to a three-dimensional position relative to the field of view of the endoscopic imaging system 15 or other imaging device providing the image. For example, stereo images displayed in the left eye display 32 and the right eye display 32 may be processed to determine matching pixel positions in the left view and the right view. The depth may then be calculated from the parallax between the left eye pixels and the right eye pixels, and the depth is used to determine the three-dimensional position of the tissue control point 200. Thus, the initial position of the tissue control point 200 may be determined with reference to a two-dimensional coordinate system or a three-dimensional coordinate system.

[0076] In process 604, a selected mode is activated, which locks the position of the instrument 206 relative to the reference coordinate system. In one or more embodiments, the same input control device 36 (e.g., the same joystick) can be used to control the instrument 206 and the tissue control point 200. Activating the selected mode configures the input control device 36 to ensure that received input is used to control the tissue control point 200, rather than the instrument 206. When the selected mode is not activated, input received through the input control device 36 is used to control the instrument 206, rather than the tissue control point 200.

[0077] In other words, if the user has activated the selected mode, the input control device 36 or certain portions of the input control device 36 used to remotely operate the instrument 206 can be reconfigured so that input received through the input control device 36 or certain portions of the input control device 36 can be used to control the tissue control point 200 instead of the instrument 206. The user can then use the input control device 36 to move the tissue control point 200, which in turn causes corresponding operation of the instrument 204.

[0078] In process 606, input is received via the input control device 36 that moves the tissue control point 200 within the user interface. Figure 3 The translation of the tissue control point 2020 described in 606 is an example of the movement of the tissue control point 200 that occurs in process 606. In process 608, the movement of the tissue control point 200 is transformed into a corresponding movement of the instrument 204 to manipulate the tissue 202.

[0079] Figure 7 700 is an illustration of a method 700 for virtually juxtaposing an organization control point 200 with an organization to be controlled 202. The method 700 is illustrated as a set of operations or processes 702 to 708 and continues with reference to Figure 1A , Figure 1B , Figure 1C and Figure 2 In all embodiments of method 700, not all of the illustrated processes 702 to 708 are performed. Figure 7One or more processes not explicitly described in the description may be included before, after, between, or as part of processes 702 to 708. In some embodiments, one or more of processes 702 to 708 may be implemented at least in part in the form of executable code stored on a non-transitory tangible machine-readable medium, which, when executed by one or more processors (e.g., a processor of a control system), may cause the one or more processors to perform one or more of the processes.

[0080] In process 702, the positioning of the instrument 204 and the positioning of the imaging device providing the tissue image are calculated. In one embodiment, in process 702, the positioning of the instrument 204 can be the positioning of the tip 212 of the instrument 204 calculated using the kinematic equations of the manipulator controlling the instrument 204. Similarly, the positioning of the imaging device (e.g., the endoscopic imaging system 15) can be the positioning of the tip of the imaging device calculated using the kinematic equations of the manipulator controlling the imaging device. The manipulator can be implemented as, for example, Figure 1B The robot 51 of the assembly 12 is shown. The positioning of the tip 212 of the instrument 204 can be determined by T tip Represented by T, and the positioning of the imaging device can be represented by T imd Indicates. tip and T imd It can be a 3D transformation matrix consisting of 3D positioning and 3D rotation components.

[0081] In process 704, a reference position of the tissue control point 200 is calculated based on the position and offset transform calculated for the instrument 204. The offset transform can be used to maintain an offset between the tip 212 of the instrument 204 and the tissue control point 200, thereby allowing the tissue control point 200 to be an extension of the kinematic chain of the manipulator 51 controlling the instrument 204. Using the offset transform allows the tissue control point 200 to be related to the instrument 204 in the reference coordinate system of the instrument 204 or the manipulator 51 controlling the instrument 204. The reference position of the tissue control point 200 can be determined by T TCP Represents and can be three-dimensional.

[0082] In process 706, the reference position of the tissue control point 200 is transformed into an imaging device-based position based on the position calculated for the imaging device. The imaging device-based position can be in an imaging device coordinate system (e.g., a coordinate system for the endoscopic imaging system 15). The imaging device-based position can be two-dimensional or three-dimensional. The imaging device-based position can be determined by T TCP_IM In process 708, the imaging device-based positioning is transformed into a display positioning. The display positioning can be in the display coordinate system of the user interface. The display positioning can be represented by T TCP_DCIn some embodiments, the display location may be or may be used to calculate the position of the tissue control point 200 relative to the image displayed in the user interface. Thus, the tissue control point 200 may be related to the instrument 204 in many different relative coordinate systems.

[0083] Figure 8 is an illustration of a method 800 for generating a tactile feedback response based on movement of a tissue control point 200. The method 800 is illustrated as a set of operations or processes 802 to 824 and continues with reference to Figure 1A , Figure 1B , Figure 1C and Figure 2 In all embodiments of method 800, not all of the illustrated processes 802 to 824 are performed. Figure 8 One or more processes not explicitly described in the description may be included before, after, during, or as part of processes 802 to 824. In some embodiments, one or more of processes 802 to 824 may be implemented at least in part in the form of executable code stored on a non-transitory tangible machine-readable medium, which, when executed by one or more processors (e.g., a processor of a control system), may cause the one or more processors to perform one or more of the processes. In one or more embodiments, processes 802 to 824 may be performed by the medical system 10.

[0084] In process 802, the position of the tip of the first instrument 204 and the tip of the second instrument 206 are calculated. The input control device 36 can be used to control the second instrument 206. In one embodiment, calculating the position in process 802 includes calculating the position of the tips of the two instruments by calculating, for example, the forward kinematics of the manipulator 51 connected to these instruments. In one or more embodiments, the input control device 36 includes a tactile device that is combined with or connected to provide tactile feedback. The position of the tip of the first instrument 204 can be determined by T tip1 and the positioning of the tip of the second instrument 206 can be represented by T tip2 In one or more embodiments, T tip1 and T tip2 Both can be three-dimensional positioning calculated in the manipulator coordinate system using the kinematic equations.

[0085] In process 804, a reference position of the tissue control point 200 is calculated based on the calculated tip position of the first instrument 204 and the offset transform. The offset transform can be used to maintain an offset between the tip of the instrument 204 and the tissue control point 200, thereby allowing the tissue control point 200 to become an extension of the kinematic chain of the manipulator 51 controlling the instrument 204. The reference position of the tissue control point 200 can be determined by T TCP Represents and can be three-dimensional.

[0086] In process 806, the reference location of the tissue control point 200 and the location of the tip of the second instrument 206 are transformed relative to a common coordinate system of the medical environment 11 in which the medical procedure is performed. In one embodiment, the common coordinate system may be referred to as a world coordinate system. The world location for the tissue control point 200 within the world coordinate system may be represented as T TCP_WC , and the world position of the tip of the second input control device 36 in the world coordinate system can be expressed as T tip2_WC The world location of the tissue control point 200 may be the world location of the virtual object 400 representing the tissue control point 200 .

[0087] The proxy geometry 402 of the second instrument 206 and the geometry of the virtual object 400 representing the tissue control point 200 can be known. The proxy geometry 402 can be moved so that the proxy geometry 402 contacts or otherwise engages with the virtual object 400 representing the tissue control point 200. The proxy geometry 402 can be used to nudge, poke, or otherwise manipulate the virtual object 400. The forces exerted on the virtual object 400 by the proxy geometry 402 can affect the movement of the first instrument 204, resulting in manipulation of the tissue 202.

[0088] In other examples, the second instrument 206 can be used to directly grasp and manipulate the virtual object 400. This type of control provides a natural and direct way to affect the posture of the retracting instrument without changing the control mode of the remote operation interface, for example. In this way, the user can perceive that they are using the second instrument 206 to directly move the tissue 202.

[0089] In process 808, the distance 404 between the world position of the virtual object 400 and the world position of the proxy geometry 402 of the instrument 206 is calculated. Figure 4 As shown, the proxy geometry 402 can be the geometry of the jaws of the instrument 206. In process 810, it is determined whether the distance 404 is less than zero. When there is a virtual penetration or interpenetration indicating a virtual contact between the proxy geometry 402 and the virtual object 400, the distance 404 is less than zero. When there is such a virtual contact, the operation of the second instrument 206 affects the virtual object 400, which in turn affects the first instrument 204, resulting in the manipulation of the tissue 202. When the contact between the proxy geometry 402 and the virtual object 400 does not exist, the distance 404 is not less than zero. Without such a virtual contact, the operation of the second instrument 206 does not affect the virtual object 400, and therefore does not affect the first instrument 204. When the distance 404 is less than zero, the distance 404 can be referred to as the penetration depth. When the distance 404 is not less than zero, the distance 404 represents the closest distance between the proxy geometry 402 and the virtual object 400.

[0090] Referring again to process 808, if the distance 404 is not less than zero, the method 800 returns to process 802 as described above. However, if the distance 404 is less than zero, then in process 811, a virtual force / torque is calculated. The virtual force / torque can be a virtual force / torque of the proxy geometry 402 on the tissue control point 200. Haptic rendering can be used to provide stable virtual contact with the tissue control point 200 and avoid slippage problems that occur when the penetration depth exceeds the thickness of the virtual object 400.

[0091] After process 811 has been executed, submethod 812 and submethod 814 are executed. Submethod 812 includes processes 816-820 for manipulating tissue 202 via interaction with tissue control points 200, and submethod 814 includes processes 822-826 for providing tactile feedback to a user when interacting with tissue control points 200.

[0092] In sub-method 812, in process 816, the calculated virtual force / torque is applied to the virtual object 400 representing the tissue control point 200. The calculated virtual force / torque is applied as a reaction force to move the virtual object 400. As previously described, the virtual object 400 geometrically represents the tissue control point 200 and can be virtually juxtaposed with the tissue 202 in the user interface. The virtual object 400 may also be referred to as a simulated tissue control point body or a simulated TCP body.

[0093] In process 818, the virtual object 400 transformation is updated. The transformation can be a transformation that determines how the movement of the virtual object 400 through the proxy geometry 402 will affect the first instrument 204. In process 820, the first instrument 204 is operated based on the updated virtual object transformation. The operation of the first instrument 204 manipulates the tissue 202. In particular, in process 820, commands for the first instrument 204 can be generated based on the updated virtual object transformation and then applied to (e.g., sent to) the first instrument 204. In this way, the second instrument 206 can be operated to move the proxy geometry 402, thereby engaging the virtual object 400 and applying a force / torque thereto. The force / torque applied to the virtual object 400 can then result in the movement of the first instrument 204, which results in the manipulation of the tissue 202. The force / torque can be integrated by the mass / damper virtual model to calculate the corresponding velocity and positioning changes.

[0094] In process 822, the virtual force / torque is transformed relative to the instrument / manipulator coordinate system. In process 824, the virtual force / torque is then transformed from the instrument / manipulator coordinate system to the input coordinate system. The input coordinate system is used for the input control device 36. In process 826, the force / torque is then applied to the input control device 36. Applying the force / torque to the input control device 36 produces a tactile feedback response that can be experienced by the user. For example, based on the virtual movement of the virtual object 400 by the proxy geometry 402, the user can feel a physical response to the pushing, stabbing, nudging, or other movement of the tissue 202 caused by the operation of the first instrument 204.

[0095] In this manner, the tissue control point 200 can support familiar physical interactions by representing the tissue control point 200 as a visual and tactile virtual object 402. Movement of the second instrument 206 can move the proxy geometry 402 to exert forces on the virtual object 400 representing the tissue control point 200. The forces exerted on the virtual object 400 can in turn cause movement of the first instrument 204. The determination of the distance 404 in process 810 can ensure that the simulated movement of the virtual object 400 occurs only when the proxy geometry 402 is in contact with the virtual object 400. This ensures that tissue manipulation is continuously controlled by the second instrument 206 and stops when contact is lost between the proxy geometry 402 and the virtual object 400.

[0096] The method 800 described above provides a method in which a surgeon S or other clinician can control the operation of a first instrument 204 without switching modes on an input control device 36 used to control a second instrument 206. Instead, the operation of the second instrument 206 can be used to virtually contact a virtual object 400 and exert a force thereon, which in turn causes movement of the first instrument 204 or some other type of operation.

[0097] Thus, the above-described embodiments provide a method and apparatus for manipulating tissue using tissue control points 200. The control system 20 establishes a relationship between the tissue control points 200 and the instrument 204 such that translational movement of the tissue control points 200 results in corresponding movement of the instrument 204, which optionally also takes into account other degrees of freedom to achieve secondary goals, such as avoiding collisions with adjacent instruments. The use of tissue control points 200 simplifies the steps required by the surgeon to manipulate tissue in a desired manner. In addition, the use of tissue control points 200 enables the surgeon to control the operation of the instrument 204 to manipulate tissue within the field of view of the endoscopic imaging system 15, even when the instrument 204 is partially or completely out of view.

[0098] One or more elements in the embodiments of the present invention may be implemented in software to be executed on a processor of a computer system such as a control processing system. When implemented in software, the elements of the embodiments of the present invention are essentially code segments that perform necessary tasks. The program or code segment may be stored in a processor-readable storage medium or device, which may be downloaded via a computer data signal embodied in the form of a carrier wave on a transmission medium or communication link. The processor-readable storage device may include any medium that can store information, including optical media, semiconductor media, and magnetic media. Examples of processor-readable storage devices include electronic circuits; semiconductor devices, semiconductor storage devices, read-only memories (ROMs), flash memories, erasable programmable read-only memories (EPROMs); floppy disks, CD-ROMs, optical disks, hard disks, or other storage devices. The code segment may be downloaded via a computer network (e.g., the Internet, an intranet, etc.).

[0099] Note that the processes and displays presented may not be inherently related to any particular computer or other device. Various general purpose systems may be used with programs according to the teachings herein, or it may prove convenient to construct more specialized devices to perform the described operations. The structures required for various of these systems will appear as elements in the claims. In addition, embodiments of the present invention are described without reference to any particular programming language. It should be understood that various programming languages ​​may be used to implement the teachings of the present invention as described herein.

[0100] Although certain exemplary embodiments of the present invention have been described and shown in the accompanying drawings, it should be understood that such embodiments are merely exemplary and not limiting of the present invention, and that since various other modifications may be made by a person skilled in the art, the embodiments of the present invention are not limited to the specific constructions and arrangements shown and described.

Claims

1. A method for manipulating tissue, the method comprising: In a user interface, displaying a tissue control point graphical element above the image of the tissue; receiving input for moving the organizational control point graphical element within the user interface; and A first instrument physically associated with the tissue is operated based on the received input of moving the tissue control point graphical element, thereby manipulating the tissue.

2. The method according to claim 1, wherein the movement of the organization control point graphic element based on the input includes at least one of left translation, right translation, upward translation, downward translation, inward translation, outward translation or rotation of the organization control point graphic element relative to a display coordinate system.

3. The method of claim 1 , wherein operating the first instrument comprises: Movement of the tissue control point graphical element is transformed into corresponding movement of the first instrument, thereby manipulating the tissue.

4. The method of claim 1 , wherein receiving the input comprises: The input is received via one of a touch screen and a hand control device.

5. The method of claim 1, wherein operating the first instrument comprises: A movement path of the first instrument is calculated that prevents at least one of the first instrument or a manipulator holding the first instrument from unintentionally interacting with another object.

6. The method of claim 1, wherein the movement of the tissue control point graphic element comprises at least one of a translation of the tissue control point graphic element or a rotation of the tissue control point graphic element relative to the image.

7. The method of claim 1, wherein the manipulation of the tissue comprises at least one of retraction of the tissue, translation of the tissue, or twisting of the tissue.

8. The method according to claim 1, further comprising: A selected mode is activated, the selected mode locking the positioning of the second instrument relative to the reference coordinate system and enabling the input of the tissue control point graphical element to be received.

9. The method of claim 8, wherein receiving the input comprises: When the selected mode has been activated, the input for moving the organizational control point graphic element within the user interface is received through an input device, wherein when the selected mode is activated, the input device controls the movement of the first instrument, and when the selected mode is not activated, the input device controls the movement of the second instrument.

10. The method according to claim 1, further comprising: An initial position within the image is calculated for the tissue control point graphical element.

11. The method of claim 10, wherein calculating the initial position comprises: performing segmentation of a region of the image, wherein the segmentation comprises at least one of color image segmentation or depth image segmentation; and The centroid of the area of ​​the image is calculated as the initial position of the tissue control point graphic element.

12. The method according to claim 1, further comprising: An initial position of the tissue control point graphical element is virtually determined based on an initial input received through an input device.

13. The method of claim 12, wherein the input device comprises at least one of a touch screen, a gesture tracking system, a gaze tracking system, a hand control device, a remote control device, or a mouse.

14. The method according to claim 1, further comprising: receiving an initial position input via an input device after the tip of the first instrument has engaged the tissue; and In response to receiving the initial position input, an initial position of the tissue control point graphical element is determined based on the position at which the tip of the instrument has engaged the tissue.

15. The method according to claim 1, further comprising: determining a two-dimensional position of the tissue control point graphical element within the image; and The two-dimensional position is mapped to a three-dimensional position relative to a field of view of an imaging device providing the image.

16. The method according to claim 1, further comprising: A tactile feedback response is generated in response to operation of the first instrument.

17. The method of claim 16, wherein generating the tactile feedback response comprises: The tactile feedback response is generated by at least one of a hand control device or a remote control device based on a physical impact of the manipulation of the tissue by the first instrument.

18. The method of claim 1, wherein operating the first instrument comprises: A movement path of the first instrument is calculated, the movement path both corresponding to the movement of the tissue control point graphical element and satisfying a predetermined criterion.

19. A method for manipulating tissue, the method comprising: displaying, in a user interface, a virtual object representing a tissue control point juxtaposed with the tissue over the image of the tissue, wherein the virtual object is offset from the positioning of the first instrument; operating a second instrument to move a proxy geometry representing the second instrument into contact with the virtual object and to exert a force on the virtual object; operating the first instrument based on the force applied to the virtual object to manipulate the tissue; and In response to operation of the first instrument, a tactile feedback response is generated at the second instrument.

20. An apparatus comprising: a display system that displays an image of the tissue in a user interface; and a control system communicatively coupled to the display system, wherein the control system is configured to: displaying a tissue control point graphical element above the image; receiving input for moving the organizational control point graphical element within the user interface; and An instrument physically associated with the tissue is operated based on the received input of moving the tissue control point graphical element, thereby manipulating the tissue.

21. The apparatus of claim 20, wherein the movement of the organizational control point graphic element based on the input comprises at least one of left translation, right translation, upward translation, downward translation, inward translation, outward translation, or rotation of the organizational control point graphic element.

22. The apparatus according to claim 20, further comprising: An input device, wherein the input is received through the input device.

23. The apparatus of claim 22, wherein the input device comprises at least one of a touch screen or a hand control device.

24. The apparatus of claim 20, wherein the control system is further configured to calculate a movement path of the instrument that both corresponds to the movement of the tissue control point graphic element and satisfies a predetermined criterion.

25. The apparatus of claim 20, wherein the movement of the tissue control point graphic element comprises at least one of a translation of the tissue control point graphic element or a rotation of the tissue control point graphic element relative to the image.

26. The device of claim 20, wherein manipulation of the tissue comprises at least one of retraction of the tissue, translation of the tissue, or twisting of the tissue.

27. The apparatus of claim 20, wherein the control system is further configured to generate a tactile feedback response in response to operation of the device.

28. The apparatus of claim 20, wherein the control system is further configured to calculate a movement path for the instrument that prevents the instrument from accidentally interacting with another object.

Images