Configuring surgical systems using surgical procedure atlases

Abstract

This application discloses configuring a surgical system using a surgical procedure atlas. A surgical method for use with a remotely controlled surgical system (surgical system) is provided, the method comprising: during the occurrence of a surgical procedure, recording surgical instrument kinematic information indicating the movement of surgical instruments generated within the surgical system; determining corresponding kinematic signatures associated with the corresponding surgical instrument movements; generating an information structure in a computer-readable storage device that associates the corresponding kinematic signatures with corresponding control signals; during the execution of the surgical procedure, comparing the surgical instrument kinematic information during the execution period with at least one corresponding kinematic signature; and during the execution of the surgical procedure, transmitting the associated corresponding control signal in response to a match between the surgical instrument kinematics and the corresponding kinematic signature during the execution period.

Description

[0001] This application is a divisional application of Chinese patent application 201680038267.7 (PCT / US2016 / 036733) filed on June 9, 2016 and entitled "Configuring a surgical system using a surgical process atlas".

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims priority to provisional application serial number 62 / 173,077 filed on June 9, 2015, the contents of which are incorporated herein by reference in their entirety, and the benefit of such priority is hereby claimed. Technical Field

[0004] Aspects of the present invention are associated with medical devices used during surgical procedures. More specifically, the aspects are associated with controlling surgical instruments in a robotic-assisted surgical system based on kinematic information and anatomical tissue image information from previous surgical procedures. Background Art

[0005] Surgeons usually conduct extensive research before performing a surgical procedure. Traditionally, surgeons have been limited to studying general anatomical models, such as photographs or pictures. In recent years, various preoperative diagnostic procedures (e.g., x-ray, CT, MRI, etc.) have made patient-specific anatomical information available.

[0006] In some cases, it is desirable to make additional, relevant anatomical and surgical procedure information available to the surgeon. In one aspect, it is desirable to provide a surgeon planning to operate on a particular patient with a surgical site video record of an earlier surgical procedure performed on that particular patient. In another aspect, it is desirable to provide the surgeon with one or more surgical site video records of surgical procedures on other patients that are similar to the surgical procedure planned for the particular patient. In one aspect, it is desirable to provide such information to the surgeon before the surgeon performs a particular surgical procedure. And in another aspect, it may be desirable to provide this information to the surgeon intraoperatively.

[0007] In one aspect, it is desirable to configure a video database that includes intraoperative surgical site video recordings of various procedures undergone by various patients. In one aspect, it is desirable to configure a medical device capable of video recording that also includes inputs that enable a surgeon to use the medical device to highlight and annotate the video recording in real time as the video recording is being recorded. In one aspect, it is desirable to configure a computer-based pattern matching algorithm to search through various records of the video database, identify relevant video recordings, and provide the surgeon with this relevant information for a particular surgical procedure. Summary of the invention

[0008] The following summary of the invention introduces certain aspects of the present subject matter in order to provide a basic understanding. This summary of the invention is not an extensive overview of the present subject matter and is not intended to identify key or important elements and define the scope of the present subject matter. Although this summary of the invention contains information related to various aspects and embodiments of the present subject matter, its sole purpose is to present certain aspects and embodiments in a general form as a prelude to the detailed description below.

[0009] In one aspect, a method for use with a teleoperated surgical system is provided. Surgical instrument kinematic information indicating the movement of a surgical instrument is recorded for multiple occurrences of a surgical procedure. A kinematic signature is determined based on the recorded kinematic information representing the movement of the surgical instrument. An information structure is generated in a computer-readable storage device that associates a corresponding kinematic signature with a corresponding electronic control signal for the surgical system. During the performance of a surgical procedure using the surgical system, surgical instrument kinematic information generated by the system during the procedure is compared with at least one kinematic signature. In response to a match between the kinematic information generated during the surgical procedure and the corresponding kinematic signature, an electronic control signal associated with at least one kinematic signature is emitted within the surgical system. More specifically, some system behaviors are triggered based on kinematic analysis.

[0010] On the other hand, a method for use with a teleoperated surgical system is provided. Motion picture images of a surgical scene generated during a robotic-assisted surgical procedure are recorded for multiple occurrences of the surgical procedure. A surgical image signature is determined based on the recorded motion picture images. An information structure is generated in a computer-readable storage device that associates a corresponding surgical image signature with a corresponding electronic control signal for the surgical system. During the performance of a surgical procedure using the surgical system, the motion picture images generated during the procedure are compared with at least one surgical image signature. In response to a match between a surgical image generated during the surgical procedure and a corresponding surgical image signature, an electronic control signal associated with at least one surgical image signature is emitted within the surgical system. More specifically, some system behaviors are triggered based on video analysis.

[0011] On the other hand, a training method for use with a teleoperated surgical system is provided. A motion map image is recorded that shows anatomical tissue within a surgical scene displayed within a viewer of the surgical system during a surgical procedure. Surgical system control tactile information imparted to surgical instrument control in response to forces imparted to the surgical instrument during contact with the displayed anatomical tissue is recorded. During a simulation of the surgical procedure, the recorded motion map image is replayed within the surgical system viewer. During the replay of the recorded motion map image during the simulation of the surgical procedure, the recorded surgical instrument control tactile sensations are imparted to the surgical instrument control. In some embodiments, the surgical control tactile sensations are replayed through vibrotactile simulation of the control input.

[0012] On the other hand, a training method for use with a remote surgical system is provided. For each of the many occurrences of a surgical procedure in a robot-assisted surgical system, (one or more) diagnostic data information instances are recorded. Each diagnostic data information instance includes: during the surgical procedure, a corresponding motion map image of an anatomical tissue in a surgical scene displayed in a viewer of the surgical system. Each diagnostic data information instance also includes: during the surgical procedure, in response to the force imparted to the surgical instrument during contact with the displayed anatomical structure, a surgical instrument control tactile sense given to the surgical instrument control. An information structure is generated in a computer-readable storage device, which associates the corresponding diagnostic data information instance with the corresponding diagnosis. Select the corresponding diagnostic data information instance. During the simulation of the surgical procedure, the recorded motion map image from the selected corresponding recorded diagnostic data information instance is replayed in the surgical system viewer. During the replay of the recorded motion map image during the simulation of the surgical procedure, the recorded surgical instrument control tactile information from the selected corresponding recorded diagnostic data information instance is given to the surgical instrument control.

[0013] In another aspect, a teleoperated surgical system includes an information structure in a computer readable storage device that associates a surgical image signature with a control signal. A processor is configured to compare a surgical image generated within a surgical scene during a surgical procedure with at least one surgical image signature. The processor is configured to transmit a control signal in response to a match between the surgical image and the at least one surgical image signature. The instrument is configured to adjust its movement in response to the control signal.

[0014] In another aspect, a teleoperated surgical system includes an information structure in a computer readable storage device that associates respective surgical image signatures with respective control signals. A processor is configured to compare a surgical image within a surgical scene during a surgical procedure with at least one surgical image signature. The processor is configured to transmit a control signal in response to a match between the surgical image during the surgical procedure and the at least one surgical image signature. The instrument is configured to adjust its movement in response to the control signal. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a plan view of a minimally invasive teleoperated surgical system.

[0016] Figure 2 is a perspective view of the surgeon's console.

[0017] Figure 3 is a perspective view of an electronics cart.

[0018] Figure 4 is a schematic diagram of a teleoperated surgical system.

[0019] Figure 5A is a schematic diagram of a teleoperated surgical system.

[0020] Figure 5B is a perspective view of a patient side cart of a surgical system.

[0021] Figure 5C This is a schematic diagram of a surgical scene.

[0022] Figure 6 It is a front view of the surgical instrument.

[0023] Figure 7 is a perspective view of an instrument manipulator.

[0024] Figure 8 is a schematic diagram of a surgical planning tool.

[0025] Fig. 9 is a flow chart of a method for using a surgical planning tool.

[0026] Fig.10 is a schematic diagram illustrating storage of an atlas in a computer-readable storage device according to some embodiments.

[0027] Fig.11 is a schematic diagram representing an illustrative example of a first data information structure included in an atlas that includes information about various surgical procedures in accordance with some embodiments.

[0028] Fig.12 is a schematic diagram representing an exemplary instance of a second data information structure included in an atlas according to some embodiments, which associates recorded motion picture image segments from various surgical procedures, corresponding surgical instrument kinematic information segments, corresponding surgical system actuation states, and corresponding annotations.

[0029] Figures 13A-13C is a schematic diagram illustrating an exemplary surgical instrument and actuator assembly according to some embodiments, wherein the surgical instrument is shown in three different exemplary instrument states.

[0030] Fig.14 is a schematic diagram showing an illustrative example of a control signal rule information structure including rules for associating kinematics, images, and system actuation states with control signals according to some embodiments.

[0031] Fig.15 is a schematic flow chart representing a process for generating control signals based at least in part on anatomical image information, instrument kinematics, and system status according to some embodiments.

[0032] Fig.16 is a diagram showing an illustrative example of a third data information structure included in an atlas that associates recorded motion picture image segments with haptic information segments according to some embodiments.

[0033] Fig.17 is a schematic flow chart representing a process of configuring a telerobotic-assisted surgical system to replay a surgical experience, according to some embodiments.

[0034] Fig.18 is a diagram representing an illustrative example of a diagnosis rule information structure included in an atlas, including rules for associating image signature information and haptic feedback information with a diagnosis, according to some embodiments.

[0035] Fig.19 is a schematic flow chart representing a process for generating a diagnosis based at least in part on anatomical image information and tactile feedback information in accordance with some embodiments. DETAILED DESCRIPTION

[0036] This specification and drawings illustrating aspects, embodiments, implementations or applications of the invention should not be considered restrictive - the claimed inventions are defined. Various mechanical, compositional, structural, electrical and operational changes may be made without departing from the scope of this specification and claims. In some instances, well-known circuits, structures or techniques are not shown and described in detail to avoid obscuring the invention. Like numbers in two or more drawings represent the same or similar elements.

[0037] Elements described in detail with reference to one embodiment, implementation or application may be included in other embodiments, implementations or applications that are not specifically shown and described whenever implemented. For example, even if an element is described in detail with reference to one embodiment without describing with reference to a second embodiment, the element may be considered to be included in the second embodiment. Therefore, in order to avoid unnecessary repetition in the following description, unless otherwise specifically described, one or more elements shown and described associated with one embodiment, implementation or application may be incorporated into other embodiments, implementations or aspects, unless one or more elements render a certain embodiment or implementation inoperative or unless two or more elements provide conflicting functions.

[0038] Although based primarily on the use of da Surgical system (specifically, Model IS4000, which is sold as da Xi TM HD TMAlthough various aspects of the present invention are described in terms of embodiments of a robotic surgical system, those skilled in the art will appreciate that aspects of the present invention disclosed herein may be embodied and implemented in a variety of ways, including robotic and, if applicable, non-robotic embodiments and implementations. Surgical systems (e.g., Model IS4000 da Xi TM Surgical System, Model IS3000 da Vinci The embodiments described herein are merely exemplary and are not to be considered as limiting the scope of the inventive aspects disclosed herein.

[0039] According to various aspects, the present disclosure describes a surgical planning tool that includes a medical device configured to video record the performance of a surgical procedure. The video record can be embedded with various metadata, such as markings made by medical personnel. In addition, the video record can be identified with various metadata, such as text annotations describing a certain subject of the video, the identity of the patient to whom the video record corresponds, archival or medical information about the patient, etc. In one aspect, the identified metadata is embedded in the video record.

[0040] According to a further aspect, an information modality is identified within motion map images and surgical instrument kinematic information collected from a number of teleoperated surgical procedures. For example, the motion map information can indicate the geometry and color of anatomical tissue. The kinematic information can indicate surgical instrument motion characteristics, such as, for example, the direction of instrument movement, the speed and acceleration of instrument movement, and the sequence of instrument movement. The information modality can be identified based on the recorded motion map, and the kinematic information can serve as a basis for managing or regulating control of surgical instruments during surgery. The information modality can be used as a basis for providing intraoperative guidance to a surgeon.

[0041] According to a further aspect, motion graphics consistent with tactile feedback can be used as a basis for surgical training. For example, a surgeon can re-experience a previous surgical procedure performed by the surgeon by replaying motion graphics and corresponding tactile feedback generated by the same surgeon during a previous surgical procedure through surgical simulation. Alternatively, for example, a surgeon can experience a previous surgical procedure performed by a different surgeon by replaying motion graphics generated by another surgeon during a previous surgical procedure and corresponding tactile feedback through surgical simulation. Therefore, the surgeon can use surgical simulation as an opportunity to improve surgical skills by replaying the actual surgical experience of the relationship between visual cues and tactile cues by the surgeon or another surgeon through simulated practice surgery.

[0042] According to a further aspect, based on the information morphology identified in the kinematic images and surgical instrument kinematic information collected from several teleoperated surgical procedures, a recommended intraoperative diagnosis is developed. A skilled surgeon is generally able to assess tissue disease states and tissue damage states based at least in part on tissue geometry and color. The recorded kinematic images provide information about the tissue geometry and tissue color of anatomical tissues within the surgical scene within the surgical system. Furthermore, a skilled surgeon is able to assess tissue disease states and tissue damage states based at least in part on the palpation of tissues. In a teleoperated surgical system, the palpation of tissues can be achieved by using an instrument control operated by a surgeon to touch the tissue structure, and the instrument control operated by the surgeon provides tactile feedback to the surgeon operating the control, which indicates the reaction force imparted to the instrument in response to the instrument touching the tissue structure. The expert surgeon evaluation of the collected kinematic images and surgical instrument kinematic information is used to identify images and kinematics of different morphologies indicating different intraoperative diagnoses. Video recording and the information structure of associated motion map image and surgical instrument kinematic information can be archived on the electronic medical record database implemented locally or remotely (for example, on a remote computer system, on a LAN or WAN, or on a cloud data storage server). Similarly, in some embodiments, for example, the information structure of associated motion map image and control tactile feedback information and corresponding diagnosis suggestion can be archived on the electronic medical record database implemented locally or remotely, for surgeon training. This video recording and information structure can be used for interested health care providers. In some embodiments, the information structure of storage can be used for use with a remote robot-assisted surgical system, to generate control signal information provided to the surgical system to produce intraoperative surgical guidance to the surgeon during the surgical procedure and provide the robot-assisted surgical control of the instrument. In some embodiments, the information structure of storage can be used for use with a surgical simulation system, to replay surgical scenes and corresponding tactile feedback, to be used in the technology of surgeon training operation surgical system. In some embodiments, the stored information structure can be used with a surgical simulation system to replay surgical scenarios and corresponding tactile feedback for use in surgeon training to diagnose tissue structure disease states and tissue damage states when performing surgery using the surgical system.

[0043] For example, a health care provider can search a medical device database based on one or more of the surgical procedure to be performed, tissue structure characteristics, and surgical instrument kinematics for the video and information structure relationships of interest using the above-mentioned metadata identification. In addition, in one aspect, the surgical planning tool includes a computer-based morphology matching and analysis algorithm. In one aspect, the morphology matching algorithm selects through the video stored on the electronic medical record database to identify correlations between visual characteristics in the video record and the associated metadata identification made by the medical staff. The surgical planning tool can apply these correlations to newly encountered anatomical structures, thereby helping the medical staff performing the procedure determine the patient's anatomy, preferred surgical approach, disease state, potential complications, etc.

[0044] In another aspect, for example, a morphology matching algorithm picks through the recorded motion map image information and optional kinematic information to identify correlations between anatomical tissue features (such as geometry) and instrument motion. This morphology can be useful, for example, to identify various anatomical features associated with various instrument motions. This morphology can also be useful, for example, to identify various anatomical features that are not associated with various instrument motions. For example, this morphology information can be used as a basis for generating a surgical guide for presentation to a surgeon during a surgical procedure. For example, this morphology information can be used as a basis for restraining or imparting certain surgical instrument motions to a surgeon during a surgical procedure.

[0045] In another aspect, for example, a morphology matching algorithm selects through recorded motion map image information and tactile feedback information to identify correlations between anatomical tissue features (such as geometric shapes) and reaction forces imparted by the tissue structure in response to touch by a surgical instrument. This morphology can be useful, for example, to identify correlations between visible anatomical tissue structures and tactile feedback imparted by the tissue structure in response to palpation by a robot-assisted instrument. In some embodiments, the relevant motion map image morphology and tactile feedback information are associated with expert surgeon diagnostic assessments used in surgeon training.

[0046] Minimally invasive remote surgical system

[0047] Teleoperation refers to operating a machine at a distance. In a minimally invasive telemedicine system, a surgeon can use an endoscope that includes a camera to view the surgical site inside the patient. In some embodiments, stereoscopic images can be captured, which allows for the perception of depth during the surgical procedure.

[0048] Referring now to the drawings, wherein like reference numerals refer to like parts throughout, Figure 11 is a plan view of a minimally invasive telesurgery system 10, which is generally used to perform minimally invasive diagnostic or surgical procedures on a patient 12 lying on an operating table 14. The system includes a surgeon's console 16 for use by a surgeon 18 during the procedure. One or more assistants 20 may also participate in the procedure. The minimally invasive telesurgery system 10 also includes (one or more) patient side carts 22 and an electronics cart 24. The patient side cart 22 is capable of manipulating at least one surgical instrument 26 through a minimally invasive incision in the patient 12 while the surgeon 18 views the surgical site through the surgeon's console 16. An image of the surgical site can be obtained through an endoscope 28 (e.g., a stereo endoscope), and the patient side cart 22 is capable of manipulating the endoscope to orient the endoscope 28. A computer processor located on the electronics cart 24 can be used to process the image of the surgical site for subsequent display to the surgeon 18 through the surgeon's console 16. NOTE: Although separate system components (i.e., the patient side cart 22, the electronics cart 24, and the surgeon's console 16) are depicted and described for exemplary purposes, in various embodiments, the elements contained therein can be combined and / or separated. For example, in some embodiments, the computer processor of the electronics cart 24 can be incorporated into the surgeon's console 16 and / or the patient side cart 22. The number of surgical instruments 26 used at one time generally depends on factors such as the diagnostic or surgical procedure and the space constraints within the surgical site. If it is necessary to replace one or more of the surgical instruments 26 being used in the procedure, the assistant 20 can remove the surgical instrument 26 from the patient side cart 22 in the operating room and replace it with another surgical instrument 26 from the tray 30.

[0049] Figure 2 1 is a perspective view of the surgeon's console 16. The surgeon's console 16 includes a viewer 31, which includes a left-eye display 32 and a right-eye display 34 for presenting a coordinated stereoscopic image of the surgical site to the surgeon 18 that enables depth perception. The console 16 further includes one or more control inputs 36. The console 16 is mounted for use on a patient side cart 22 (e.g., Figure 1 One or more surgical instruments on the operative table 26 (shown) are moved in response to the surgeon 18 manipulating one or more control inputs 36. The control inputs 36 can provide a surgical instrument 26 (such as a Figure 126) to provide the surgeon 18 with a remote presentation or perception of the control input 36 as being integrated with the instrument 26, so that the surgeon has a strong sense of directly controlling the instrument 26. To this end, position, force, and tactile feedback sensors (not shown) may be employed to transmit position, force, and tactile perception from the surgical instrument 26 back to the surgeon's hand through the control input 36, subject to communication latency constraints. It should be noted that while a physical console 16 with a fixed viewer 31 and mechanically coupled control input 36 is depicted and described for exemplary purposes, in various other embodiments, an "ungrounded" control input and / or display structure can be used. For example, in some embodiments, the viewer 31 can be a head-mounted display and / or the control input 36 can be mechanically independent of any base structure (e.g., wired, wireless, or gesture-based, such as Microsoft's Kinect).

[0050] The surgeon's console 16 is typically located in the same room as the patient, enabling the surgeon to directly monitor the procedure, be physically present if necessary, and speak directly to a patient-side assistant rather than via a telephone or other communication medium. However, the surgeon can be located in a different room from the patient, an entirely different building, or other remote location that allows for a telesurgical procedure.

[0051] Figure 3 28 is a perspective view of the electronics cart 24. The electronics cart 24 can be coupled to the endoscope 28, and the electronics cart 24 includes a computer processor to process the captured images for subsequent display, such as on the surgeon's console or on another suitable display located locally and / or remotely to the surgeon. For example, if a stereoscopic endoscope is used, the computer processor on the electronics cart 24 can process the captured images so as to present the surgeon with a coordinated stereoscopic image of the surgical site. Such coordination can include alignment between opposing images, and can include adjusting the stereo working distance of the stereoscopic endoscope. As another example, the image processing can include using previously determined camera calibration parameters to compensate for imaging errors of the image capture device, such as optical aberrations. Optionally, the equipment in the electronics cart can be integrated into the surgeon's console or patient side cart, or it can be distributed in various other locations in the operating room.

[0052] Figure 4 The teleoperated surgical system 50 (such as Figure 1 The minimally invasive telesurgery system 10 of FIG. 1 ). The surgeon's console 52 (such as Figure 1 The surgeon's console 16 in FIG. 1 can be used by the surgeon to control the patient side cart 54 (such as Figure 1The patient side cart 54 is capable of using an imaging device, such as a stereo endoscope, to capture images of the surgical site and output the captured images to an electronic cart 56 (e.g., Figure 1 The computer processor is typically comprised of one or more data processing boards that are intended to execute computer readable code stored in a non-volatile memory device of the computer processor. In one aspect, the computer processor is capable of processing the captured image in various ways prior to any subsequent display. For example, the computer processor is capable of overlaying the captured image with a virtual control interface and then displaying the combined image to the surgeon via the surgeon's console 52.

[0053] Additionally or alternatively, the captured image can be image processed by a computer processor located external to the electronics cart 56. In one aspect, the teleoperated surgical system 50 includes an optional computer processor 58 (shown in dashed lines) similar to the computer processor located on the electronics cart 56, and the patient side cart 54 outputs the captured image to the computer processor 58 for image processing before displaying it on the surgeon's console 52. In another aspect, the captured image is first image processed by the computer processor on the electronics cart 56, and then additional image processing is performed by the computer processor 58 before displaying it on the surgeon's console 52. The teleoperated surgical system 50 can include an optional display 60, as indicated by dashed lines. The display 60 is coupled to the computer processor located on the electronics cart 56 and to the computer processor 58, and the captured images processed by these computer processors can be displayed on the display 60 in addition to being displayed on the display of the surgeon's console 52.

[0054] Figure 5A 1 is a schematic simplified block diagram showing the arrangement of components of a teleoperated surgical system 10 for performing a surgical procedure using one or more mechanical support arms 510 according to some embodiments. Aspects of the system 10 include features of robotic-assisted and autonomous surgery. These mechanical support arms 510 often support surgical instruments. For example, a mechanical surgical arm (e.g., a central mechanical surgical arm 510C) can be used to support an endoscope having a stereoscopic or three-dimensional surgical image capture device 101C. The mechanical surgical arm 510C can include a sterile adapter, or forceps, clamps, screws, slots / grooves, or other fastening mechanisms to mechanically secure an endoscope including an image capture device 101C to the mechanical arm.

[0055] A user or operator O (typically a surgeon) performs a surgical procedure on a patient P by manipulating control input devices 36, such as handles and foot pedals at the main control console 16. The operator is able to view video frames of images of the surgical site within the patient through a stereoscopic display viewer 31. The computer processor 58 of the console 16 directs the movement of the remotely controlled endoscopic surgical instruments 101A-101C via control lines 159, thereby affecting the movement of the instruments using a patient-side system 24 (also referred to as a patient-side cart).

[0056] The patient-side system 24 includes one or more mechanical support arms 510. Typically, the patient-side system 24 includes at least three mechanical surgical arms 510A-510C (generally referred to as mechanical surgical support arms 510), which are supported by corresponding positioning setting arms 156. The central mechanical surgical arm 510C can support an endoscopic camera 101C, which is suitable for capturing images within the field of view of the camera. The central left and right mechanical surgical support arms 510A and 510B can support instruments 101A and 101B that can manipulate tissue, respectively.

[0057] Figure 5B 5 is a perspective view of a patient side cart 500 of a minimally invasive teleoperated surgical system 10 according to an embodiment. The patient side cart 500 includes one or more support arm assemblies 510. A surgical instrument manipulator 512 is mounted at the end of each support arm assembly 510. In addition, each support arm assembly 510 can optionally include one or more setting joints (e.g., unpowered and / or lockable) for positioning the attached surgical instrument manipulator 512 relative to the patient for surgical operation. As described, the patient side cart 500 is placed on the floor. In other embodiments, the operating portion of the patient side cart can be mounted to a wall, mounted to a ceiling, mounted to an operating table 526 that also supports a patient's body 522, or mounted to other operating room equipment. Further, although the patient side cart 500 is shown as including four surgical instrument manipulators 512, more or fewer surgical instrument manipulators 512 can be used.

[0058] A functional teleoperated surgical system will generally include a visual system portion that enables a user of the teleoperated surgical system to view the surgical site from outside the patient's body 552. The visual system typically includes a camera instrument 528 for capturing video images and one or more video displays for displaying the captured video images. In some surgical system configurations, the camera instrument 528 includes an optical device that transmits images from the distal end of the camera instrument 528 to one or more imaging sensors (e.g., CCD or CMOS sensors) outside the patient's body 522. Alternatively, the imaging sensor(s) can be positioned at the distal end of the camera instrument 528, and the signals generated by the sensor(s) can be transmitted along the leads or wirelessly for processing and display on one or more video displays. An example of a video display is a stereoscopic display on a surgeon's console in a surgical system commercialized by Intuitive Surgical Operations, Inc. of Sunnyvale, California.

[0059] refer to Figure 5A-5B , mounted to each surgical instrument manipulator 512 is a surgical instrument 520 that operates at a surgical site within a patient's body 522. Each surgical instrument manipulator 512 can be provided in a variety of forms to allow the associated surgical instrument to move in one or more mechanical degrees of freedom (e.g., all six Cartesian degrees of freedom, five or fewer Cartesian degrees of freedom, etc.). Typically, mechanical or control constraints restrict each manipulator 512 from moving its associated surgical instrument about a center of motion on the instrument that is at rest relative to the patient, and this center of motion is typically located at the location where the instrument enters the body.

[0060] In one aspect, the surgical instrument 520 is controlled by computer-assisted telecontrol. The functional minimally invasive teleoperated surgical system includes a control input that receives input from a user of the teleoperated surgical system (e.g., a surgeon or other medical personnel). The control input communicates with one or more computer-controlled teleoperated actuators, such as the one or more motors to which the surgical instrument 520 is coupled. In this manner, the surgical instrument 520 moves in response to the movement of the medical personnel controlling the input. In one aspect, the one or more control inputs are contained in a surgeon's console (such as a Figure 2 The surgeon can manipulate the control input device 36 of the surgeon's console 16 to operate the remote actuator of the patient side cart 500. The force generated by the remote actuator is transferred via a power transmission mechanism, which transmits the force from the remote actuator to the surgical instrument 520.

[0061] refer to Figure 5A-5B In one aspect, the surgical instrument 520 and the cannula 524 are removably coupled to the manipulator 512, wherein the surgical instrument 520 is inserted through the cannula 524. One or more teleactuators of the manipulator 512 move the surgical instrument 512 as a whole. The manipulator 512 also includes an instrument bracket 530. The surgical instrument 520 is removably connected to the instrument bracket 530. In one aspect, the instrument bracket 530 houses one or more teleactuators therein, thereby providing a number of controller movements, which the surgical instrument 520 translates into various movements of the end effector on the surgical instrument 520. Therefore, the teleactuators in the instrument bracket 530 only move one or more components of the surgical instrument 520 without moving the instrument as a whole. The input of controlling the instrument as a whole or controlling a component of the instrument enables the input provided to the control input ("master" command) by the surgeon or other medical personnel to be translated into a corresponding action ("slave" response) of the surgical instrument.

[0062] According to some embodiments, the surgical system 10 can have multiple system actuation states, including docked, following, instrument type, and head-in. During the docked system state, one or more manipulators 512 have been coupled to the cannula 524. During the following state, the surgical instrument ("slave") is tracking the control input ("master" command). During the instrument type system state, the system has installed therein a set of instruments suitable for performing a specific surgical procedure or suitable for performing a specific surgical activity during a surgical procedure. During the head-in system state, the system is waiting for the surgeon to indicate that he / she has taken control of the "master" control input device.

[0063] In an alternative embodiment, the instrument bracket 530 does not accommodate a remote actuator. The remote actuator that can realize various movements of the end effector of the surgical instrument 520 is accommodated in a position away from the instrument bracket 530, such as elsewhere on the patient side cart 500. A cable-based force transmission mechanism or the like is used to transfer the movement of each of the remotely located remote actuators to the corresponding actuator output of the instrument interface located on the instrument bracket 530. In some embodiments, the surgical instrument 520 is mechanically coupled to a first actuator that controls a first movement of the surgical instrument, such as longitudinal (z axis) rotation. The surgical instrument 520 is mechanically coupled to a second actuator that controls a second movement of the surgical instrument, such as two-dimensional (x, y) movement. The surgical instrument 520 is mechanically coupled to a third actuator that controls a third movement of the surgical instrument, such as the opening and closing of the end effector of the clamping jaw.

[0064] Figure 5C is a schematic diagram of an endoscope 101C showing a surgical scene 550 and also showing a camera 528 mounted for recording the scene, according to some embodiments. The scene 550 is disposed within a body cavity of a patient. The scene 550 includes an exemplary hypothetical spherical anatomical structure 552, which includes a geometric outline 554. The scene 550 contains a surgical instrument 556. The camera 528 mounted on the endoscope 101C captures the scene, which is displayed within the viewer 31 and recorded for later playback.

[0065] Figure 6 6 is a side view of a surgical instrument 520, which includes a distal portion 650 and a proximal control mechanism 640 coupled by an elongated tube 610 having an elongated tube central axis 611. The surgical instrument 520 is configured to be inserted into a patient's body and used to perform a surgical or diagnostic procedure. The distal portion 650 of the surgical instrument 520 can provide any of a variety of end effectors 654, such as the forceps shown, a needle driver, a cauterization device, a cutting tool, an imaging device (e.g., an endoscope or an ultrasound probe), or the like. The surgical end effector 654 can include functional mechanical degrees of freedom, such as a jaw that opens or closes, or a knife that translates along a path. In the illustrated embodiment, the end effector 654 is coupled to the elongated tube 610 by a wrist 652, thereby allowing the end effector to be oriented relative to the elongated tube central axis 611. The surgical instrument 520 can also include stored information (e.g., on a semiconductor memory associated with the instrument), which can be permanent or updateable by a surgical system configured to operate the surgical instrument 520. Thus, the surgical system can provide one-way or two-way information communication between the surgical instrument 520 and one or more components of the surgical system.

[0066] Figure 7 5 is a perspective view of a surgical instrument manipulator 512. The instrument manipulator 512 is shown without a surgical instrument mounted thereon. The instrument manipulator 512 includes an instrument holder 530 to which a surgical instrument (e.g., surgical instrument 520) can be removably attached. The instrument holder 530 houses a plurality of remote actuators. Each remote actuator includes an actuator output 705. When a surgical instrument is mounted on the instrument manipulator 512, the instrument proximal control mechanism (e.g., Figure 6One or more instrument inputs (not shown) of the proximal control mechanism 640 at the instrument manipulator 512 are mechanically coupled to corresponding actuator outputs 705. In one aspect, such mechanical coupling is direct, wherein the actuator outputs 705 directly contact the corresponding instrument inputs. In another aspect, such mechanical coupling occurs through an intermediate interface, such as a component of a drape configured to provide a sterile barrier between the instrument manipulator 512 and the associated surgical instrument.

[0067] In one aspect, movement of one or more instrument inputs via corresponding remote actuators results in movement of the mechanical degrees of freedom of the surgical instrument. For example, in one aspect, the surgical instrument mounted on instrument manipulator 512 is surgical instrument 520, such as Figure 6 Reference Figure 6 In one aspect, movement of one or more instrument inputs to the proximal control mechanism 640 via a corresponding tele-actuator rotates the elongated tube 610 (and the attached wrist 652 and end effector 654) about the elongated tube central axis 611 relative to the proximal control mechanism 640. In another aspect, movement of one or more instrument inputs via a corresponding tele-actuator results in movement of the wrist 652, thereby orienting the end effector 654 relative to the elongated tube central axis 611. In another aspect, movement of one or more instrument inputs via a corresponding tele-actuator results in movement of one or more movable elements (e.g., a jaw member, a knife member, etc.) of the end effector 654. Thus, various mechanical degrees of freedom of surgical instruments mounted on the instrument manipulator 512 can be moved via operation of the tele-actuators of the instrument holder 530.

[0068] Annotate recorded video

[0069] Figure 8 A schematic diagram of an exemplary surgical planning tool 800 is shown. In one aspect, the surgical planning tool 800 includes a teleoperated surgical system 50 in data communication with an electronic medical device record database 830. The teleoperated surgical system 50 shown here is similar to Figure 4 8. The teleoperated surgical system 50 shown in FIG. 8. In one aspect, the electronic medical record database 830 includes medical records of patients treated at a particular hospital or multiple hospitals. The database 830 can be implemented on a server located on-site at the hospital. The medical record entries contained in the database 830 can be accessed from a hospital computer via an internal network. Alternatively, the database 830 can be implemented on a remote server located outside the hospital, such as using one of a variety of cloud data storage servers. In this case, the medical record entries of the database 830 are stored on a cloud server and can be accessed by a computer using Internet access.

[0070] In one aspect, a surgical procedure is performed on a first patient using a teleoperated surgical system 50. An imaging device associated with the teleoperated surgical system 50 captures images of the surgical site and displays the captured images as video frames on a display of a surgeon's console 52. In one aspect, medical personnel at the surgeon's console 52 use an input device of the surgeon's console 52 to mark or annotate a certain patient anatomical structure shown in the displayed video. An example of such an input device is Figure 2 A control input 36 is shown at the top, which is coupled to a cursor that operates in conjunction with a graphical user interface overlaid on the displayed video. The graphical user interface can include a QWERTY keyboard, a pointing device (such as a mouse) and an interactive screen display, a touch screen display, or other means for data or for text entry or voice annotations / or speech to text conversion via a microphone and processor. Thus, medical personnel can mark certain tissues of interest in the displayed image or enter text annotations.

[0071] In one aspect, the surgical site video is additionally displayed on a display located on the electronic cart 56. In one aspect, the display of the electronic cart is a touch screen user interface that can be used by medical personnel to mark or annotate certain portions of the patient's anatomy shown on the image displayed for viewing on the display on the electronic cart. The user can mark portions of the displayed image by touching portions of the patient's anatomy displayed on the touch screen user interface. In addition, a graphical interface including a QWERTY keyboard can be overlaid on the displayed image. The user can enter text annotations using the QWERTY keyboard.

[0072] In one aspect, the surgical site video captured by the imaging device associated with the teleoperated surgical system 50 is recorded by the teleoperated surgical system 50 and stored on the database 830 in addition to being displayed to the user in real time or near real time. Marks and / or annotations made by the user associated with the recorded video can also be stored on the database 830. In one aspect, the marks made by the user are embedded with the recorded video before it is stored on the database 830. At a later time, the recorded video can be retrieved for viewing. In one aspect, the person viewing the recorded video can choose whether to display or hide the marks from the view. Similarly, the annotations associated with the recorded video can also be stored on the database 830. In one aspect, the annotations made by the user are used to identify the recorded video and can be used to provide a means of identifying the subject matter contained in the recorded video. For example, a note can describe the condition of a certain disease state. This annotation is used to identify the recorded video. At a later time, a person who wishes to view the recorded process associated with this disease state can use a keyword search to locate the video.

[0073] Stored video retrieval

[0074] In some cases, it is desirable for medical personnel to be able to review video records of past surgical procedures performed on a given patient. In one aspect, a patient who previously underwent a first surgical procedure to treat a medical condition subsequently requires a second surgical procedure to treat a recurring same medical condition or to treat an anatomical structure adjacent to the surgical site of the first surgical procedure. In one aspect, the surgical site events of the first surgical procedure are captured in a surgical site video record, and the video record is archived in database 830 as part of the patient's electronic medical record. Prior to performing a second surgical procedure on the patient, medical personnel can perform a search of database 830 to locate the video record of the patient's earlier surgical procedure.

[0075] In some cases, it is desirable that medical personnel planning to perform a surgical procedure on a patient can view video records of similar surgical procedures performed on people with certain characteristics similar to the patient. In one aspect, before each video record is archived in the database 830, the surgical site video record of the surgical procedure can be identified with metadata information, such as the patient's age, gender, body mass index, genetic information, the type of surgery the patient has undergone, etc. In one aspect, the metadata information used to identify the video record is automatically retrieved from the patient's existing medical record, and then used to identify the video record before the video record is archived in the database 830. Therefore, before performing a medical procedure on a patient, medical personnel can search the database 830 for video records of similar procedures performed on patients with certain common characteristics with the patient. For example, if medical personnel plan to perform a prostatectomy on a 65-year-old male patient with an elevated body mass index using the telesurgical system 50, the medical personnel can search the database 830 for surgical site video records of prostatectomies performed on other male patients of similar age and with a similar elevated body mass index using the telesurgical system 50.

[0076] In one aspect, the video recording of the surgical procedure is communicated to an optional personal computer 820 (shown in dashed lines) via database 830 and is available for viewing by the medical personnel planning to perform the surgical procedure. Additionally or alternatively, a video recording of an earlier surgical procedure can be communicated to the telesurgical system 50 via database 830 and is available for pre-operative or intra-operative viewing. In one aspect, the video recording is displayed on a display located on the surgeon's console 52 via the telesurgical system 50. In another aspect, the video recording of the first surgical procedure is displayed on a display located on the electronics cart 56.

[0077] Cloud-based video database

[0078] In one aspect, database 830 is implemented on a remote server using a cloud data storage server and is accessible to multiple healthcare providers. Figure 8 , as shown in dashed lines, surgical planning tool 800 optionally includes a telesurgical system 850 (shown in dashed lines) and a personal computer 840 (shown in dashed lines). In one aspect, telesurgical system 850 is similar to telesurgical system 50, and personal computer 840 is similar to personal computer 820, except that telesurgical system 50 and personal computer 820 are located at a first health care provider, while telesurgical system 850 and personal computer 840 are located at a second location or even at a second location with a second health care provider. In one aspect, a first patient requires surgical treatment for a medical condition, and a surgical procedure is performed at the first health care provider using telesurgical system 50. A video recording of the surgical procedure is archived on database 830. At a later time, a second patient requires surgical treatment for the same medical condition, and is scheduled to receive surgical treatment using telesurgical system 850 at the second health care provider. Prior to performing the surgical procedure on the second patient, medical personnel access database 830 through a secure network connection and search database 830 for surgical site video recordings of similar procedures. In one aspect, the medical personnel treating the second patient can retrieve the video recording of the surgical procedure of the first patient from the database 830 without knowing the identity of the first patient. In this way, the privacy of the first patient is maintained. In one aspect, the video recording of the surgical procedure of the first patient includes the markings and / or annotations made by the medical personnel treating the first patient.

[0079] Computer-based morphological matching and analysis

[0080] The surgical planning tool 800 can include a morphology matching and analysis algorithm implemented in the form of a computer executable code. On the one hand, the morphology matching and analysis algorithm is stored in a non-volatile memory device of the surgical planning tool 800 and is configured to analyze the video records archived in the database 830. As previously discussed, each video record archived in the database 830 can be identified and / or embedded with certain metadata information. This metadata information can include patient information, such as patient age, gender, and other information describing the patient's health or medical history. In addition, as previously discussed, the metadata information can include marks or annotations made by medical personnel. On the one hand, these marks and annotations are embedded with the video record and archived with the video in the database 830.

[0081] In one aspect, the morphology matching and analysis algorithm includes an image analysis component that identifies morphologies of shapes and colors that are shared among multiple video records stored on database 830. The morphology matching and analysis algorithm then examines the identified metadata associated with a subset of the video records to determine if any words and phrases are frequently associated with the videos within the subset. These analyses performed by the morphology matching and analysis algorithm can be used to assist medical personnel in making decisions regarding patient anatomy, preferred surgical approaches, disease states, potential complications, and the like.

[0082] Methods of using surgical planning tools

[0083] Fig. 9 A method 900 of using a surgical planning tool is shown. In one aspect, a surgical planning tool is used with Figure 8 At 910, facts or characteristics describing a medical patient (e.g., a medical condition suffered by the patient) are received by the medical device. The medical device can be located via a remote surgical system (e.g., Figure 1 A remotely operated surgical system 10 or Figure 4 The user interface of the telesurgery system 50 at the location may receive such fact or circumstance, or alternatively, through a user interface similar to Figure 8 At 920, the medical device retrieves at least one relevant video record of the surgical procedure from a medical device database using the fact or characteristic received at 910. At 930, the medical device determines surgical planning information using the video record. In one aspect, the surgical planning information includes the type of instruments used during the recorded procedure. At 940, the medical device displays the surgical planning information determined at 930 to a user.

[0084] Fig.10 1 is a schematic diagram of a storage atlas 1002 represented in a computer-readable storage device 1004 according to some embodiments. The storage atlas 1002 includes a first data information structure 1006 indicating instances of previously performed surgical procedures. Each different instance of the first data information structure 1006 corresponds to a different surgical procedure. A second data information structure 1008 associated with each surgical procedure event is stored at a corresponding time interval (t1, t2, ... t n ) makes the motion picture image segments (MPI1, MP12, ... MPI n ) and kinematic information fragments (KIN1, KIN2, ... KIN n ) and associated with the surgical system actuation state (Astate1, Astate2, ... Astate n) and the corresponding annotations (ANOT1, ANOT2, ... ANOT n ). The first control signal rule information structure 1010 generated based on multiple surgical events of a given type associates a kinematics / image signature / actuation state combination with a control signal, such as (SigK1, SigI1, Astate1, CNTL KIA1 )……(SigK n 、SigI n 、Astate n 、CNTL KIAn Each different instance of the first rule information structure 1010 corresponds to a different surgical procedure category. The third data information structure 1012 associated with each surgical procedure event is generated at a corresponding time interval (t1, t2, ... t n ) makes the motion picture image segments (MPI1, MP12, ... MPI n ) and tactile information fragments (HAP1, HAP2, ... HAP n ) is associated with the second diagnosis rule information structure 1014 generated based on multiple surgical operation events of a given type. DIAG1 、SigI DIAG2 , ...SigI DIAGm ) and diagnostic kinematic signature (SigK DIAG1 , SigK DIAG2 、……SigK DIAGm ) and associated with the diagnosis (DIAG1, DIAG2, ... DIAG m Each different instance of the second diagnosis rule information structure 1014 corresponds to a different surgical procedure category.

[0085] According to some embodiments, the image information can be in the form of video images spanning the entire visible spectrum, fluorescence, hyperspectral, CT / MRI. The image information or a combination of some of these image information can serve as a basis for assessing disease state / determining diagnosis.

[0086] Fig.111 is a schematic diagram of an exemplary instance of a first data information structure 1006 included in an atlas 1002 in a storage device 1004 according to some embodiments, which includes information about various surgical procedures. It should be understood that multiple surgical procedures are performed on multiple different patients by many different surgeons using many different instances of the robotic-assisted surgical system described herein. An instance of the first data information structure 1006 can be generated and stored in a computer-readable storage device for each surgical procedure performed. For example, the exemplary first data information structure 1006 includes a surgical procedure type field 1006-1 that indicates the type of surgical procedure, such as a prostatectomy, a hysterectomy, or a partial nephrectomy.

[0087] For example, the exemplary first data information structure 1006 includes a patient health record field 1006-2, which provides information about the patient undergoing surgery, such as age, weight, blood type, height, gender, and race. For example, the exemplary first data information structure 1006 includes a physician information field 1006-3, which provides information about the surgeon performing each operation, such as the general experience level and the experience level of operating the robot-assisted surgical system. For example, the exemplary first data information structure 1006 includes a surgical system field 1006-4, which provides information about the surgical system used to perform the operation, such as manufacturing, model, and serial number. For example, the exemplary first data information structure 1006 includes a surgical record field 1006-5, which provides information such as motion map images, instrument kinematics, and tactile feedback provided by the system during the surgical procedure. The exemplary first data information structure 1006 includes a surgical record field 1006-6, which provides annotation information, such as an identifier that contains descriptive information that has been associated with the image information, kinematic information, or tactile information within the exemplary first data information structure 1006.

[0088] Fig.121 is a schematic diagram of an exemplary instance of a second data information structure 1008 included in an atlas 1002 in a storage device 1004 according to some embodiments, which associates recorded motion map image segments, corresponding surgical instrument kinematic information segments, corresponding surgical system actuation states, and corresponding annotations from various surgical procedures. The exemplary second data information structure 1008 provides additional information details within the record field 1006-5 of the first data information structure 1006. On the one hand, motion map images of patient anatomical structures, corresponding surgical instrument kinematics, and corresponding surgical system actuation states are recorded and time stamped (t1, t2, ... tn) during the surgical procedure to produce a time-series record of surgical activities and corresponding surgical instrument kinematics during the surgical procedure. In some embodiments, the motion map images also include images of surgical instruments used to operate on the anatomical structure during the surgical procedure. The timestamp temporarily aligns the motion map images with the surgical instrument kinematics.

[0089] During a surgical procedure, a user (e.g., a surgeon or other member of a surgical team) may annotate recorded motion map information and associated recorded surgical instrument kinematic information with metadata that indicates corresponding surgical activities, such as, for example, dissection or suturing, anatomical features, particular surgical complexity, surgeon observations, or the stage of the procedure. For example, the annotations may include one or more of written notes identified to the recorded motion map information and / or recorded surgical instrument kinematic information, colors or markings (e.g., telestration) of images in the video recording, or a combination thereof. For example, the annotations may be later added to the timestamped recorded image information and / or recorded kinematic information.

[0090] It should be understood that during a telesurgery procedure, surgical activity can occur, resulting in changes in the actuation state of the surgical system. The surgeon can move his head into and out of the viewer 31, resulting in a change in the guidance state. For example, the surgeon can move his hands or feet into or out of contact with the controller input 36, resulting in a change in the following state. For example, the combination of instruments in use can be changed, resulting in a change in the instrument type state.

[0091] The operation of surgical instrument support surgical activities in one or more surgical states leads to the generation of kinematic information indicating instrument movement within the surgical system, which indicates the actual way in which the surgeon performs surgical activities. For example, the surgeon may have moved the instrument quickly or slowly. For example, the surgeon may have moved the surgical instrument along one or another path in one direction toward the anatomical structure or away from the anatomical structure in one direction. For example, the surgeon may have adjusted the position of different instruments before actuating a specific instrument. It should be understood that the combination of the recorded anatomical structure image information and the recorded instrument kinematic information and the recorded system actuation state provide a record of what images are presented to the surgeon during the surgical operation and what activities the surgeon is engaged in related to these images and what system actuation state the surgical system is at that time. The corresponding annotations can provide additional understanding of the associated images and kinematics recorded during the surgical operation. Note: Although kinematic information can be derived directly from the surgical system (e.g., via joint data or other mechanical tracking), in various other embodiments, kinematic information can be extracted from image processing or sensor data (e.g., tool / instrument tracking in endoscopic images).

[0092] Figures 13A-13C 1 is a schematic diagram showing an exemplary surgical instrument 1202 and an actuator assembly 1203 according to some embodiments, wherein the surgical instrument is shown in three different exemplary instrument actuation states. The exemplary instrument 1202 includes a jaw end effector 1204 that is capable of transitioning between an open state and a closed state, and transitioning between partially open / partially closed states continuously therebetween. The exemplary instrument 1202 also includes a two-degree-of-freedom (2-dof) wrist 1206 that is capable of moving between different two-dimensional (x, y) position states. The exemplary actuator assembly 1203 includes a first actuator 1208, which in some embodiments includes a jaw motor (JM) for actuating the jaw end effector 1204. The exemplary actuator assembly 1203 includes a second actuator 1210, which in some embodiments includes a wrist motor (WM) for actuating the wrist 1206. During a surgical procedure, surgical instrument 1202 may transition through various instrument actuation states corresponding to different activities during the surgical procedure. Each transition results in the generation of kinematic information that is captured and stored and indicates the instrument movement as the instrument transitions from its physical position and deployment in one state (e.g., open or closed) to its physical position and deployment in the next state. Fig.13AAs shown in , for example, a surgical procedure may involve a first surgical activity in which a first actuator 1208 (JM) arranges the jaw end effector 1204 into a fully open state and a second actuator 1210 (WM) arranges the wrist 1206 into a first position state (x1, y1). Fig. 13B As shown in , for example, the surgical procedure may involve a second surgical activity in which the first actuator 1208 transitions the jaw end effector 1204 to a fully closed state and the second actuator 1210 transitions the wrist 1206 to a second position state (x2, y2). Fig. 13C As represented in , for example, the surgical procedure may involve a third surgical activity in which the first actuator 1208 arranges the jaw end effector 1104 into a partially open / partially closed state and the second actuator 1210 transitions the wrist 1206 to a third position state (x3, y3).

[0093] Fig.14 1 is a schematic diagram showing an illustrative example of a first control signal rule information structure 1010 included in an atlas 1002 in a storage device 1004, including rules for associating kinematic signature information, image signature information, and system actuation state information with control signals in accordance with some embodiments. The rules are developed based on data from previous surgical procedures represented in the first and second data information structures 1006, 1008. The first rules associate morphology of anatomical images, instrument kinematics, and system state with control signals for controlling system operation or providing guidance to a surgeon during a surgical procedure.

[0094] According to some embodiments, the kinematic signature includes a multidimensional vector. In some embodiments, for example, the recorded kinematic motion of the instrument is decomposed into multiple vectors representing kinematic features, such as instantaneous velocity, instantaneous acceleration, instantaneous three-dimensional position, current motion path, and predicted motion path. Not only the motion and position of the instrument, but also the background of the instrument (such as the physical position of the anatomical structure relative to the instrument, the physical position of other instruments, the health of the patient, and the nature of the surgical operation) are relevant to the interpretation of the kinematic information. Moreover, previous instrument motion can be relevant to the assessment, such as the position to which the instrument moves next, which is at least partially based on the instrument kinematics. Therefore, in some embodiments, for example, the instrument kinematic vector also includes a vector indicating the position of the anatomical structure, the position of other instruments, the health of the patient, the type of surgical operation, and the previous motion of the instrument.

[0095] According to some embodiments, the image signature comprises a multidimensional vector. For example, in some embodiments, a recorded motion map image of an anatomical structure is decomposed into multiple vectors representing image features, such as color, texture, and geometry. Moreover, in a convolutional neural network, there are lower-level features (e.g., color edges, etc.); in subsequent layers, higher-level features are learned, ultimately leading to, for example, a classification of anatomical structures or tissue types. Not only the appearance of the anatomical structure, but also the context of the anatomical structure (such as the patient's health, the nature of the surgical procedure, the surgeon's skill level in a particular type of robotic-assisted surgery, and whether the system status indicates that the correct surgical instrument is currently installed in the system) is relevant to the interpretation of the anatomical image. Moreover, changes in the appearance of the anatomical structure during the surgical procedure can be relevant to the interpretation of the anatomical image. Therefore, in some embodiments, for example, the image vector also includes vectors indicating patient health record information, the type of surgical procedure, and a comparison of the appearance of the anatomical image at different stages of the surgical procedure.

[0096] According to some embodiments, different control signals are associated with different combinations of image signatures, kinematic signatures, and actuation system states. In some embodiments, some control signals control instrument actuation states. For example, certain combinations of image signatures, kinematic signatures, and actuation system states may correspond to unsafe surgical activities, and the corresponding control signals may operate to freeze the movement of the instrument by the surgical system, and / or generate notifications to the surgeon (e.g., visual, audible, and / or tactile warnings, or indications that relevant guidance / information is available). For example, the instrument moves outside the field of view of the camera. Alternatively, for example, certain combinations of image signatures, kinematic signatures, and actuation system states may correspond to particularly delicate surgical activities, and the corresponding control signals may operate to cause the surgical system to reduce the rate of instrument movement to a safe speed or may limit the range of motion of the instrument to avoid damage or may generate advisory notifications to the surgeon (e.g., an overlay of corrective / warning information displayed in the viewer, or a presentation of an entry to relevant guidance / information). For example, preserving nerves during prostatectomy, or suturing during mitral valve repair.

[0097] According to some embodiments, machine learning techniques can be used to generate image signatures and generate kinematic signatures. More specifically, for example, a classifier can be used together with expert knowledge to associate image signatures and kinematic signatures with control signals. Based on, for example, expert surgeon input, the surgical data within the first and second data information structures 1006, 1008 are evaluated to determine appropriate system operations in the context of different anatomical images, instrument kinematics, and surgical system states. The control signals that affect the determined system operations are associated with the anatomical image signatures, instrument kinematic signatures, and system states in the first rule information structure 1010. According to some embodiments, the image signature and kinematic signature can be combined with system state information to produce an overall signature corresponding to the control signal.

[0098] Fig.15 15 is a schematic flow diagram representing a process 1502 for generating a control signal based at least in part on anatomical image information, instrument kinematics, and system status according to some embodiments. According to some embodiments, the computer processor 58 is configured to perform the process 1502. During the performance of the surgical procedure, the rules block 1504 determines whether to cause the emission of a control signal based on the kinematic image during the surgical procedure, the instrument kinematics, and the surgical system status and the rules from the first rule information structure 1010.

[0099] More specifically, during the use of the system 10 to perform a surgical procedure, the rule block 1504 receives motion map image information generated during the surgical procedure, receives instrument kinematic information generated during the surgical procedure, and receives system state information generated during the surgical procedure. The camera 528 captures the motion map information during the surgical procedure. The computer processor 58 provides the corresponding image information to the rule block 1504 at block 1506. The instrument actuator (such as actuators 1208, 1210) receives an actuation command that causes the instrument to move during the surgical procedure. The computer processor 58 determines the instrument kinematic information based on the actuation command at block 1508 and provides the kinematic information to the rule block 1504. The system includes a sensor (not shown) that senses the system state, such as whether the operator has placed his head on the viewer, whether the operator has engaged the control input with the hand and / or foot 36, and which instruments are inserted for use. The computer processor 58 provides the system state information to the rule block 1504 at block 1510.

[0100] Likewise, during the surgical procedure, the control signal rule storage block 1512 provides the image signature information, kinematic signature information, and system actuation state information from the first rule block portion 1010A of the first control signal rule information structure 1010 to the rule block 1504. The rule block 1504 compares the image, kinematic, and system information provided by blocks 1506, 1508, 1510 with the associated image signature (SigI), kinematic signature (SigK), and system state (Astate) information from the first rule block portion 1010A, respectively. It should be understood that in some embodiments, the computer processor 58 is configured to convert the image, kinematic, and system information generated by the system 10 into a format suitable for comparison with the signature and state information from the first portion 1010A of the control signal rule information structure 1010. In particular, in some embodiments, the raw image / kinematic / system state is processed to derive a classification (signal / probability) that is then looked up in a table to determine what action to take based on the determined classification.

[0101] The decision module 1506 determines whether a match occurs between the provided image, kinematics, and system state information and the rule information from the first rule block portion 1010A. It should be appreciated that in machine learning embodiments, a match is determined based on a range of similarities between the image, kinematics, and system state information and the rule information. Thus, for example, a combination of generated image, generated kinematics, and generated system state information that is within a certain threshold limit of a certain rule is determined to match the rule.

[0102] In response to determining a match between the combination of image, kinematics, and system state information and the rule, block 1514 transmits a control signal from within the second rule portion 1010B of the control signal rule information structure 1010, which corresponds to the matching rule. For example, in response to determining that the image, kinematics, and system state information received during the surgical procedure matches SigI2, SigK2, and Astat2 of the first portion 1010A, block 1516 transmits a signal CNTL from within the second portion 101B of the control signal information structure 1010. IKA2. Thus, the control signal emitted depends on the actual system information. Moreover, the control signal emitted can be operated to control instrument actuation. For example, a control signal can be emitted to prevent a certain instrument from moving, such as to avoid a collision with another instrument. For example, a control signal can be emitted to control how a certain instrument movement occurs, such as by limiting the rate of speed. For example, a control signal can be emitted to control what the instrument does next, such as moving the instrument to a neutral position to prevent a collision with an instrument that the surgeon may use next. When no match is detected, the decision module feeds the control flow back to the rule block 1504. Similarly, after the emission of the control signal, the control flow returns to the rule block 1504 to continue comparing the image, kinematic and system information generated during the surgical procedure with the rules from the control signal information structure 1010.

[0103] Fig.16 is a schematic diagram representing an exemplary instance of a third data information structure 1012 included within an atlas 1002 of a storage device 1004 that associates recorded motion map image segments with tactile information segments in accordance with some embodiments. The exemplary third data information structure 1012 provides additional detail of the information within the record field 1006-5 of the first data information structure 1006. In one aspect, motion map images of the patient's anatomy and corresponding tactile feedback information imparted to the surgeon / operator during a surgical procedure are recorded and time-stamped (tl, t2 ... tn) to produce a chronological record of what the surgeon observes and what the surgeon senses through physical touch sensations at the control inputs during the surgical procedure. The timestamps temporarily register the motion map images with the surgical instrument kinematics.

[0104] Haptics generally describes touch feedback, which can include kinesthetic (force) and cutaneous (tactile) feedback, as well as vibration and movement. In teleoperated surgical systems, natural tactile feedback is largely eliminated because the surgeon does not directly manipulate the instruments. Figure 6 According to some embodiments, an artificial tactile sensor 660 mounted on the patient side of the surgical instrument 520 can acquire tactile information, and a tactile display on the surgeon side transmits the sensed information to the surgeon. Figure 5A A kinesthetic or force feedback system typically measures or estimates the forces applied by surgical instruments to the patient's anatomy within the surgical scene and provides resolved forces to the surgeon's hand via a force feedback device 39 mounted on a control input 36. Haptic feedback can be used to provide information such as local tissue deformation and pressure distributed on the tissue surface.

[0105] Reference again Fig.16, the third data information structure 1012 associates the motion map information segments with the immediate tactile feedback information delivered to the force feedback device 39. Thus, the data in the third data information structure can represent a record of the surgeon's visual and tactile input during the surgical procedure. This information can be used to help train the surgeon's mechanical skills and also to help train the surgeon in diagnosis.

[0106] Fig.17 1702 is a schematic flow chart showing a process 1702 of configuring a telerobotic-assisted surgical system to replay a surgical experience according to some embodiments. According to some embodiments, the computer processor 58 is configured to perform the process 1702. The processor 58 uses the recorded motion map information to configure the camera 528 to replay the recorded motion map at the viewer. The processor 58 uses the recorded tactile information to configure the force feedback device 39 to impart the recorded tactile feedback at the control input 36. In some embodiments, the force feedback device 39 includes a vibrotactile device.

[0107] It should be appreciated that this replay can help train a surgeon to more effectively use the surgical system based on the surgeon's own prior surgical experience or based on another surgeon's surgical experience. By viewing the viewer 31 and touch control input 36 while the images and corresponding tactile feedback forces are replayed, the surgeon can experience the recorded surgical procedure. For example, the surgeon can then try out alternate surgical approaches and compare the visual and tactile sensations during these alternate approaches with those of previous surgical procedures. Fig.18 1 is a diagram representing an illustrative example of a second diagnosis rule information structure 1014 included in the atlas 1002 in the storage device 1004, including a rule that associates image signature information and tactile feedback information with a diagnosis according to some embodiments. The rule is developed based on data from previous surgical procedures represented in the first and third data information structures 1006, 1012. The second rule associates the morphology of the anatomical image and tactile feedback with a diagnosis.

[0108] As explained above, the image signature comprises a multidimensional vector. In addition, according to some embodiments, the tactile feedback signature comprises a multidimensional vector. According to some embodiments, the recorded tactile feedback is decomposed into multiple vectors, which represent tactile feedback characteristics, such as the position, frequency, and intensity of the feedback force. Not only the position, frequency, and intensity of the feedback force, but also the context of the feedback force (such as the patient's health and the nature of the surgical procedure) is relevant to the interpretation of the feedback signal. Thus, for example, in some embodiments, the multidimensional tactile feedback vector also includes a vector indicating the patient's health and the type of surgical procedure.

[0109] According to some embodiments, machine learning techniques can be used to generate an image signature and to generate a tactile feedback signature. More specifically, for example, a classifier can be used together with expert knowledge to associate the image signature and the tactile signature with a diagnosis. Based on, for example, expert surgeon input, the surgical data within the first and third data information structures 1006, 1012 are evaluated to determine an appropriate diagnosis in the context of different anatomical images and tactile feedback forces. According to some embodiments, the image signature and the tactile signature can be combined to produce an overall signature corresponding to a diagnosis.

[0110] Fig.19 is a schematic flow diagram representing a process 1902 for generating a diagnosis based at least in part on anatomical image information and tactile feedback information according to some embodiments. According to some embodiments, the computer processor 58 is configured to perform the process 1902. During the performance of the surgical procedure, based on the kinematic images and tactile feedback forces during the surgical procedure and the rules from the second rules information structure 1014, the rules block 1904 determines whether to report a diagnosis.

[0111] More specifically, during the performance of a surgical procedure using the system 10, the rule block 1904 receives motion map image information generated during the surgical procedure and receives tactile feedback information generated during the surgical procedure. The camera 528 captures the motion map information during the surgical procedure. The computer processor 58 provides the corresponding image information to the rule block 1904 at block 1906. The tactile feedback force device 39 generates tactile feedback forces. The computer processor 58 determines the tactile feedback force information based on the actuation command at block 1908 and provides kinematic information to the rule block 1904.

[0112] Likewise, during the surgical procedure, the diagnostic rule storage block 1912 provides the image signature information and tactile feedback information from the first rule block portion 1014A of the diagnostic rule information structure 1014 to the rule block 1904. The rule block 1904 compares the image and tactile feedback information provided by blocks 1906, 1908 with the associated image signature (SigI) and tactile feedback signature (SigH), respectively, from the first rule block portion 1014A of the rule diagnostic structure 1014. It should be understood that in some embodiments, the computer processor 58 is configured to convert the image and tactile feedback information generated by the system 10 into a format suitable for comparison with the signature and status information from the first portion 1014A of the diagnostic rule information structure 1014.

[0113] The decision module 1914 determines whether a match occurs between the provided image and tactile feedback information and the rule information from the first rule portion 1014A of the diagnostic rule information structure 1014. It should be appreciated that in machine learning embodiments, a match is determined based on a range of similarities between the image and tactile information and the rule information. Thus, for example, a combination of generated image and generated tactile information that is within a certain threshold limit of a certain rule is determined to match the rule.

[0114] In response to determining a match between the combination of image, kinematics, and system state information and the rule, block 1914 transmits a control signal from within the second rule block 1010B portion of the first control signal rule information structure 1010, which is equivalent to matching the rule. For example, in response to determining that the image and tactile feedback information received during the surgical procedure matches SigI2 and SigH2 in the first portion 1014A of the diagnosis information structure 1014, block 1916 reports on the viewer 31 that the diagnosis DIAG1 forms the second portion of the diagnosis information structure 1014. Therefore, the diagnosis during the surgical procedure depends on the actual system information generated during the surgical procedure and the information recorded in many previous surgical procedures. During process 1902, when no match is detected, decision module 1914 feeds control flow back to rule block 1904. Similarly, after the diagnosis, flow control returns to rule block 1904 to continue comparing the image and tactile feedback generated during the surgical procedure with the rules from the diagnosis rule information structure 1014.

[0115] Although illustrative embodiments have been shown and described, extensive modifications, alterations, and substitutions are contemplated in the foregoing disclosure, and in some instances, some features of the embodiments may be employed without corresponding use of other features. One of ordinary skill in the art will recognize many variations, substitutions, and modifications. Therefore, the scope of the present disclosure should be limited only by the appended claims, and it is appropriate to construct the claims broadly and in a manner consistent with the scope of the embodiments disclosed herein.

Claims

1. A remote-controlled surgical system, comprising: Surgical instruments; an actuator coupled to impart motion to the surgical instrument in response to a control signal; a sensor coupled to track instrument kinematic information indicative of instrument motion, the instrument kinematic information indicative of surgical instrument motion; a control signal rules information structure that provides a classification for kinematic information and associates the classification with a control signal, wherein the control signal rules information structure provides the classification by matching an instrument kinematic signature of the instrument kinematic information with one or more rules in the control signal rules information structure, the instrument kinematic signature being determined based on the instrument kinematic information; a processor communicatively coupled to the actuator, the sensor, and the control signal rule information structure, the processor configured to perform operations comprising: receiving machine kinematic information from the sensor; identifying a classification of the received machine kinematic information based on the classification of kinematic information provided by the control signal rule information structure; and A control signal associated with the identified classification by the control signal rule information structure is transmitted.

2. The teleoperated surgical system according to claim 1, The processor is configured to perform operations including: The received instrument kinematics information is converted into instrument kinematics signature information.

3. The teleoperated surgical system according to claim 2, Wherein identifying the classification of the received machine kinematic information comprises comparing the converted machine kinematic information with kinematic signature information within one or more of the classifications provided by the control signal rule information structure.

4. The teleoperated surgical system according to claim 1, Wherein in response to the control signal, the teleoperated surgical system adjusts the speed of movement of the instrument.

5. The teleoperated surgical system according to claim 1, Wherein in response to the control signal, the teleoperated surgical system adjusts the range of motion of the instrument.

6. The teleoperated surgical system according to claim 1, Wherein in response to the control signal, the teleoperated surgical system adjusts the visual guidance provided to the operator.

7. The teleoperated surgical system according to claim 2, The kinematic signature of the instrument comprises a corresponding multi-dimensional vector.

8. The teleoperated surgical system according to claim 2, The kinematic signature of the instrument comprises a multi-dimensional vector representing one or more of instantaneous velocity, instantaneous acceleration, instantaneous three-dimensional position, current motion path, and predicted motion path.

9. The teleoperated surgical system according to claim 2, The kinematic signature of the instrument comprises a multi-dimensional vector representing one or more of the position of anatomical structures, the positions of other instruments, patient health, the type of surgical procedure, and previous motion of the instrument.

10. The teleoperated surgical system according to claim 1, The control signal comprises a surgical instrument end effector control signal.

11. The teleoperated surgical system of claim 1 , further comprising: One or more actuators configured to adjust the movement of the surgical instrument based on the transmitted control signal.

12. The teleoperated surgical system of claim 1, further comprising: a camera for capturing surgical image information within a surgical scene; wherein the control signal rule information structure provides classification for kinematic information and the surgical image information and associates the classification with control signals; The processor is communicatively coupled to the camera, and the processor is further configured to perform operations, the operations comprising: receiving surgical image information from the camera; and Wherein identifying a classification includes identifying a classification of a combination of the received instrument kinematic information and the received surgical image information based on a classification of the combined kinematic information and image signature information provided by the control signal rule information structure.

13. The teleoperated surgical system according to claim 12, The processor is configured to perform operations including: Converting the received instrument kinematics information into instrument kinematics signature information; and The received surgical operation image information is converted into surgical operation image signature information.

14. The teleoperated surgical system according to claim 13, The identification categories include: Identifying a classification of the combination of the received instrument kinematic information and the received surgical image information includes comparing the transformed instrument kinematic information with kinematic signature information within one or more of the classifications provided by the control signal rule information structure, and comparing the transformed received surgical image information with surgical image signature information within one or more of the classifications provided by the control signal rule information structure.

15. The teleoperated surgical system according to claim 13, wherein the kinematic signature of the apparatus comprises a corresponding multidimensional vector; and in, The motion map surgical image signature consists of a multi-dimensional vector.

16. The teleoperated surgical system according to claim 13, The surgical image signature of the instrument comprises a multi-dimensional vector representing one or more of color, texture, and geometry.

17. The teleoperated surgical system according to claim 13, in, The motion map surgical image signature includes a multi-dimensional vector representing one or more of patient health record information, surgical procedure type, and a comparison of anatomical image appearance at different surgical stages.

18. A teleoperated surgical system comprising: Surgical instruments; an actuator coupled to impart motion to the surgical instrument in response to a control signal; a camera for capturing surgical image information within a surgical scene, the surgical image information being generated with the assistance of the teleoperated surgical system during a surgical procedure; a control signal rules information structure that provides a classification for the image information and associates the classification with a control signal, wherein the control signal rules information structure provides the classification by matching a surgical image signature of the surgical instrument information to one or more rules in the control signal rules information structure, the surgical image signature being determined based on the surgical instrument information; a processor communicatively coupled to the actuator, the camera, and the control signal rule information structure, the processor configured to perform operations comprising: receiving image information from the camera; identifying a classification of the received image information based on the classification of the image information provided by the control signal rule information structure; and A control signal associated with the identified classification by the control signal rule information structure is transmitted.

19. The teleoperated surgical system according to claim 18, The processor is configured to perform operations including: The received image information is converted into image signature information.

20. The teleoperated surgical system according to claim 19, Wherein identifying the classification of the received image information comprises comparing the converted instrument image information with image signature information within one or more of the classifications provided by the control signal rule information structure.

21. The teleoperated surgical system of claim 18, Wherein in response to the control signal, the teleoperated surgical system adjusts the speed of movement of the instrument.

22. The teleoperated surgical system of claim 18, Wherein in response to the control signal, the teleoperated surgical system adjusts the range of motion of the instrument.

23. The teleoperated surgical system of claim 18, Wherein in response to the control signal, the teleoperated surgical system adjusts the visual guidance provided to the operator.

24. The teleoperated surgical system of claim 19, The image signature includes the corresponding multi-dimensional vector.

25. The teleoperated surgical system of claim 19, The image signature of the instrument comprises a multi-dimensional vector representing one or more of the image features.

26. The teleoperated surgical system of claim 18, The control signal comprises a surgical instrument end effector control signal.

27. The teleoperated surgical system of claim 1, further comprising: One or more actuators configured to adjust the movement of the surgical instrument based on the transmitted control signal.

Images