A remote operation surgery system involving scanning-based positioning

Abstract

To provide a method for use with a teleoperated surgical system.SOLUTION: The method comprises: determining patient position information during a setup for performing a surgical procedure within an instance of the surgical system; determining a match between the determined patient position and a patient position signature; and launching a support arm control signal within the surgical system that corresponds to the matched support arm signature.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] Claim of Priority This application claims the benefit of priority to U.S. Patent Application No. 62 / 421,089, filed on November 11, 2016, the entire disclosure of which is hereby incorporated by reference herein.

[0002] Aspects of the invention relate to medical devices used during surgery. More specifically, aspects relate to the positioning of surgical system components.

Background Art

[0003] Surgeons generally begin extensive deliberations before performing a surgical procedure. Conventionally, surgeons have been limited to examining general anatomical models such as photographs or diagrams. More recently, patient-specific anatomical information has become available through various preoperative diagnostic procedures (e.g., X-rays, CT, MRI, etc.).

[0004] In some cases, it may be desirable to make additional relevant anatomical and surgical procedure information available to the surgeon. In one aspect, it is desirable to provide a surgeon planning a surgery on a particular patient with a video recording of the surgical site of a previous surgical procedure performed on that particular patient. In another aspect, it is desirable to provide the surgeon with one or more surgical video recordings of surgical procedures on other patients that are similar to the surgical procedure planned for the particular patient. In one aspect, it is desirable to provide such information to the surgeon before the surgeon begins a particular surgical procedure. And in another aspect, it may be desirable to provide this information to the surgeon during the surgery.

[0005] In one aspect, it is desirable to construct a video database that includes video recordings of intraoperative surgical sites of various procedures received by various patients. In one aspect, it is desirable to configure a medical device capable of video recording such that it further includes an input that enables a surgeon using the medical device to highlight and annotate the video recording in real time during recording. In one aspect, it is desirable to configure a computer-based pattern matching algorithm to search individual records in the video database, identify relevant video recordings, and provide this relevant information for a particular surgical procedure to a surgeon.

SUMMARY OF THE INVENTION

[0006] The following summary introduces specific aspects of the subject matter of the present invention to provide a basic understanding. This summary is not an extensive overview of the subject matter of the present invention, is not intended to identify key or critical elements, or to delineate the scope of the subject matter of the present invention. This summary includes information related to various aspects and embodiments of the subject matter of the present invention, but its sole purpose is to present some aspects and embodiments in a general form as a prelude to the more detailed description that follows.

[0007] In one aspect, a method for use with a robotic surgery system is provided. Patient location information is determined for a surgical procedure during setup of the surgery system. A match is determined between the patient location information determined during setup and each patient's location signature. A support arm control signal corresponding to each matched patient location signature is activated during setup.

[0008] In another aspect, a method for use with a robotic surgery system is provided. The layout of the operating room is determined for a surgical procedure during setup of the surgery system. A match is determined between the operating room layout determined during setup and each operating room layout signature. An image representing the position of the surgery system module corresponding to each matched operating room layout signature is generated.

BRIEF DESCRIPTION OF THE DRAWINGS

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5A

Figure 5B

Figure 5C

Figure 5D

Figure 5E

Figure 5F

Figure 5G

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10A

Figure 10B

Figure 11A

Figure 11B

Figure 12A

Figure 12B

Figure 13A

Figure 13B

Figure 13C

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

[0010] This description and the accompanying drawings showing aspects, embodiments, implementations, or applications of the present invention should not be construed as limiting, and the claims define the invention to be protected. Various mechanical, compositional, structural, electrical, and operational changes can be made without departing from this description and the claims. In some instances, well-known circuits, structures, or techniques have not been shown in detail or described in order not to obscure the present invention. Like numbers in two or more figures represent the same or similar elements.

[0011] Elements described in detail with reference to one embodiment, implementation, or application may, whenever practical, be included in other embodiments, implementations, or applications in which those elements are not specifically shown or described. For example, if an element is described in detail with reference to one embodiment and not described with reference to a second embodiment, nevertheless, that element may be claimed to be included in the second embodiment. Thus, to avoid unnecessary repetition in the following description, one or more elements shown and described in connection with one embodiment, implementation, or application may be incorporated into other embodiments, implementations, or aspects, provided that one or more such elements do not render an embodiment or implementation non-functional or that two or more such elements do not provide conflicting functions, unless otherwise specified.

[0012] Aspects of the present invention are primarily described with respect to embodiments using the da Vinci® Surgical System (specifically, model IS4000, commercially available from Intuitive Surgical, Inc. of Sunnyvale, California, as the da Vinci® Xi™ HD™ Surgical System). However, one of ordinary skill in the art will understand that the aspects of the invention disclosed herein may be embodied and implemented in a variety of ways, including robotic and, where applicable, non-robotic embodiments and implementations. Embodiments with respect to the da Vinci® Surgical System (e.g., model IS4000 da Vinci® Xi™ Surgical System, model IS3000 da Vinci Si™ Surgical System) are merely illustrative and should not be considered as limiting the scope of the aspects of the invention disclosed herein.

[0013] In accordance with various aspects, the present disclosure describes a surgical planning tool that includes a medical device configured to video record the performance of a surgical procedure. The video recording can embed various metadata, such as highlights made by medical personnel. Additionally, the video recording can be tagged with various metadata, such as text annotations that describe a particular subject of the video, the identity of the patient corresponding to the video recording, historical or medical information about the patient, and the like. In one aspect, the tagged metadata is embedded in the video recording.

[0014] According to a further aspect, a robotic arm configured to attach a surgical instrument is positioned based at least in part on the location of at least one anatomical feature of a patient. According to yet another aspect, a cart-mounted robotic surgical system module can be positioned near a patient based at least in part on the configuration of the operating room in which the surgical procedure is performed. During a telesurgical procedure, the surgical instrument extends into the patient's body cavity through a surgical port at an incision in the patient's anatomical structure. The incision location and surgical port location are determined based at least in part on the type of surgical procedure, the dimensions of the patient's body, and the position of the patient on the operating table. In some embodiments, the surgical instrument is positioned during surgery to have a central axis of rotation at the port to minimize distortion to the anatomical tissue surrounding the port.

[0015] In some embodiments, a plurality of cart-mounted surgical system modules are arranged around a patient during surgery. One or more of the surgical system modules include robotic surgical arms configured to attach surgical instruments. In some embodiments, different surgical instruments are attached to each of the plurality of robotic surgical arms. During a surgical procedure, each instrument extends through a different surgical port to a different location within the patient's body cavity. Typically, a plurality of medical personnel move around the operating room during surgery. The medical personnel need to have immediate access to the patient, both for routine surgical activities and for responding to emergencies. A surgical procedure may include a series of surgical activities such as cutting, cauterizing, and suturing. Different surgical activities may require the use of different surgical instruments or different combinations of surgical instruments. During a surgical procedure, the medical personnel move between the patient and the cart-mounted surgical system modules within the operating room, or between the plurality of cart-mounted surgical system modules, to reconfigure the remote surgery system or perform other surgical activities, and need to exchange the surgical instruments extending through one or more surgical ports. For example, a surgical instrument used during a particular surgical activity can be removed when that surgical procedure is completed, and a different surgical instrument can be inserted in its place to perform a different surgical activity. The operating room in which the surgical procedure is performed has limited space to position the cart-mounted surgical system modules to facilitate access to the surgical ports while still ensuring sufficient spacing between the cart-mounted surgical system modules and the patient table, or between such modules, to allow the medical personnel to move freely and efficiently around the operating room during surgery.

[0016] An information structure that associates video recordings, the positioning of robotic surgical arms with the patient's position and / or physical characteristics such as height and weight, and the positioning of surgical system modules with the layout of the operating room can be stored in a locally implemented electronic medical record database or in a cloud data storage service. The video recordings can be made available to interested healthcare providers. The information structure is used with surgical system setup guidance to determine the positioning of the robotic surgical arms and surgical system modules.

[0017] A healthcare provider can use the metadata tags described above to search a medical device database based on one or more of the surgical procedures performed, the patient's physical characteristics, and the layout of the operating room, regarding the relationship between the video and the information structure of interest. Further, in one aspect, a surgical planning tool includes computer-based pattern matching and analysis algorithms. In one aspect, the pattern matching algorithm selects videos stored in an electronic medical record database and identifies correlations between visual features within the video recordings and associated metadata tags made by healthcare practitioners. The surgical planning tool can apply these correlations to newly encountered anatomical structures, thereby assisting healthcare providers performing the procedure in making decisions regarding the patient's anatomical structure, preferred surgical approach, medical condition, potential complications, etc.

[0018] In another aspect, the pattern matching algorithm selects videos and other records stored in an electronic medical record database to identify the most suitable positioning of the robotic surgical arm for different patient positions and anatomical structures. Further, the pattern matching algorithm selects videos and other records stored in the electronic medical record database to identify the most efficient use of the operating room space for different operating room layouts by determining the position of the surgical system modules. In some embodiments, a scanning device, such as a light irradiation scanning device (e.g., a laser scanning device, an IR scanning device, or an RF scanning device), can be used to record the position of the patient's body and the movements of medical personnel. Alternatively or additionally, the video can provide a record of the movements of medical personnel during a surgical procedure. Or, for example, tracking devices such as RFID tags can be worn by medical personnel to track their movements. Using the record of the position of the patient's body during a previous surgery, the most efficient settings of one or more robotic surgical arms can be determined to more quickly achieve the accurate positioning of instruments during the surgical setup. Using the record of the movements of medical personnel during a previous surgery, the most efficient settings of the surgical system modules can be determined to facilitate the ease and speed of movement of medical personnel during surgery.

[0019] Minimally Invasive Teleoperated Surgery System Referring now to the drawings, like reference numerals represent like parts throughout the several views. FIG. 1 is a plan view of a minimally invasive teleoperated surgery system 10, which is typically used to perform minimally invasive diagnostic or surgical procedures on a patient 12 lying on a movable operating table 14. The system includes a movable surgeon console 16 for use by a surgeon 18 during the procedure. One or more assistants 20 can also participate in the procedure. The minimally invasive teleoperated surgery system 10 further includes a movable patient-side cart 22 and a movable electronics cart 24. In some embodiments, the table 14, surgeon console 16, patient-side cart 22, and electronics cart 24 are mounted on wheels to provide mobility.

[0020] The patient-side cart 22 includes a plurality of segmented mechanical support arms 72. One end of each support arm is rotatably attached to a vertical support structure 74, and the other end attaches a surgical instrument 26 that is removably coupled. In some embodiments, each mechanical support arm 72 includes a first segment 72-1, a second segment 72-2, and a third segment 72-3. During the setup of a surgical procedure, the plurality of segments of at least one support arm 72 are moved to position a surgical instrument for insertion into a minimally invasive incision in the body of patient 12. During the surgical procedure, while the surgical instrument is inserted into the patient's body cavity, the surgeon 18 views the surgical site via the surgeon console 16. Images of the surgical site can be acquired by an endoscope 28 such as a stereoscopic endoscope, which can be manipulated by the patient-side cart 22 to face the endoscope 28. A computer processor disposed on the electronics cart 24 is used to process the images of the surgical site and can later be displayed to the surgeon 18 via the surgeon console 16. The number of surgical instruments 26 used at one time is generally influenced by diagnostic or surgical procedures, among other factors, and spatial constraints within the operating room. If it is necessary to replace one or more of the surgical instruments 26 being used during the procedure, the assistant 20 can remove the surgical instrument 26 from the patient-side cart 22 and replace the surgical instrument 26 with another surgical instrument 26 from a tray 30 within the operating room.

[0021] Figure 2 is a perspective view of the surgeon console 16. The surgeon console 16 includes a left-eye display 32 and a right-eye display 34 for presenting the surgeon 18 with an adjusted stereoscopic view of the surgical site that enables depth perception. The console 16 further includes one or more control inputs 36. One or more surgical instruments installed for use with the patient-side cart 22 (shown in FIG. 1) move in response to the operation of one or more control inputs 36 by the surgeon 18. The control inputs 36 provide the same mechanical degrees of freedom as their associated surgical instruments 26 (shown in FIG. 1), providing the surgeon 18 with telepresence, that is, the perception that the control inputs 36 are integral with the instruments 26, whereby the surgeon can have a strong sense of directly controlling the instruments 26. For this purpose, position, force, and tactile feedback sensors (not shown) can be used to transmit position, force, and tactile sensations from the surgical instrument 26 back to the surgeon's hand via the control inputs 36.

[0022] The surgeon console 16 is typically located in the same room as the patient, so the surgeon can directly monitor the procedure, physically attend as needed, and speak directly to the patient-side assistant without using a phone or other communication medium. However, the surgeon can be located in a different room, an entirely different building, or some other remote location away from the patient, enabling a remote surgical procedure.

[0023] FIG. 3 is a perspective view of the electronic device cart 24. The electronic device cart 24 can be coupled to the endoscope 28 and includes a computer processor that processes the captured images for later display to the surgeon on the surgeon console or to other suitable displays such as local and / or remote displays located elsewhere. For example, when a stereoscopic endoscope is used, the computer processor on the electronic device cart 24 can process the captured images to present the surgeon with an adjusted stereoscopic image of the surgical site. Such adjustments can include aligning the opposing images and can include adjusting the stereoscopic working distance of the stereoscopic endoscope. As another example, the image processing can include the use of previously determined camera calibration parameters to compensate for imaging errors of the image capture device such as optical aberrations. Optionally, the devices within the electronic device cart can be integrated into the surgeon console or the patient-side cart, or distributed at various other locations in the operating room.

[0024] FIG. 4 schematically shows a robotic surgery system 50 (such as the minimally invasive robotic surgery system 10 of FIG. 1). A surgeon can use a surgeon console 52 (such as the surgeon console 16 of FIG. 1) to control a patient-side cart 54 (such as the patient-side cart 22 of FIG. 1) during a minimally invasive procedure. The patient-side cart 54 can use an imaging device such as a stereoscopic endoscope to capture an image of the surgical site and output the captured image to a computer processor disposed in an electronic device cart 56 (such as the electronic device cart 24 of FIG. 1). The computer processor typically includes one or more data processing boards for the purpose of executing computer-readable code stored in a non-volatile memory device of the computer processor. In one aspect, the computer processor can process the captured images in various ways before subsequent display. For example, the computer processor can overlay the captured images with a virtual control interface before displaying the combined images to the surgeon via the surgeon console 52.

[0025] Additionally or alternatively, the captured image can be processed by a computer processor located external to the electronic device cart 56. In one aspect, the remote surgery system 50 includes an optional computer processor 58 (shown in dashed lines) similar to the computer processor disposed in the electronic device cart 56, and the patient-side cart 54 outputs the captured image for image processing to the computer processor 58 before displaying it on the surgeon console 52. In another aspect, the captured image first undergoes image processing by the computer processor on the electronic device cart 56 and then additional image processing by the computer processor 58 before being displayed on the surgeon console 52. The remote surgery system 50 can include an optional display 60, as shown in dashed lines. The display 60 is coupled to the computer processor disposed on the electronic device cart 56 and the computer processor 58, and the captured images processed by these computer processors can be displayed on the display 60 in addition to being displayed on the display of the surgeon console 52.

[0026] FIG. 5A is a perspective view of a patient-side cart 54 of a minimally invasive remote surgery system 10 according to an embodiment. The patient-side cart 54 includes four mechanical support arms 72. A surgical instrument manipulator 73 including a motor for controlling the movement of the instrument is attached to the end of each support arm assembly 72. Additionally, one or more sets of setup joints (e.g., non-powered and / or lockable) can be included for positioning the surgical instrument manipulator 73 attached to the patient for surgery. As shown, the patient-side cart 54 is placed on the floor. In other embodiments, the operative portion of the patient-side cart can be attached to a wall, to a ceiling, to an operating table 14 that also supports the patient's body 12, or to other operating room equipment. Further, although the patient-side cart 54 is shown to include four surgical instrument manipulators 73, more or fewer surgical instrument manipulators 73 may be used.

[0027] A functional remote operation surgical system will generally include a visual system portion that enables a user of the remote operation surgical system to view the surgical site from outside the patient's body 12. The visual system will typically include a camera device 26C for capturing video images and one or more video screens for displaying the captured video images. In some surgical system configurations, the camera device 26C includes an optical system that transfers the image from the tip of the camera device 26C to one or more imaging sensors (e.g., CCD or CMOS sensors) outside the patient's body 12. Alternatively, the imaging sensor(s) can be positioned at the tip of the camera device 26C, and the signal generated by the sensor(s) can be transmitted along a lead wire or wirelessly for processing and displayed on one or more video screens. An example of a video screen is the stereoscopic display on the surgeon console in a surgical system commercially available from Intuitive Surgical, Inc. of Sunnyvale, California.

[0028] Referring to FIG. 5A, a surgical instrument 26 operating at a surgical site within the patient's body 12 is attached to each surgical instrument manipulator 73. Each surgical instrument manipulator 73 can be provided in various forms such that the associated surgical instrument can move with one or more mechanical degrees of freedom (e.g., a total of six Cartesian degrees of freedom, less than five Cartesian degrees of freedom, etc.). Typically, the mechanical or control constraints limit each manipulator 73 from moving its associated surgical instrument around a center of motion on an instrument that remains stationary relative to the patient, and this center of motion is typically at the location where the instrument enters the body.

[0029] In one aspect, the surgical instrument 26 is controlled by computer-assisted remote operation. A functional minimally invasive remote operation surgical system includes control inputs that receive inputs from a user of the remote operation surgical system (e.g., a surgeon or other medical personnel). The control inputs communicate with one or more computer-controlled remote operation actuators, such as one or more motors to which the surgical instrument 26 is coupled. In this way, the surgical instrument 26 moves in response to the movement of the control inputs of the medical personnel. In one aspect, one or more control inputs are included in a surgeon console, such as the surgeon console 16 shown in FIG. 2. The surgeon can operate the control input 36 of the surgeon console 16 to operate the remote operation actuator of the patient-side cart 54. The force generated by the remote operation actuator is transmitted via a drive train mechanism that transmits the force from the remote operation actuator to the surgical instrument 26.

[0030] Referring to FIG. 5A, in one aspect, the surgical instrument 26 and the cannula 27 are removably coupled to the manipulator 73 with the surgical instrument 26 inserted through the cannula 27. One or more remote operation actuators of the manipulator 73 move the surgical instrument 26 as a whole. The manipulator 73 further includes an instrument carriage 75. The surgical instrument 26 is removably connected to the instrument carriage 75. In one aspect, the instrument carriage 75 houses internally one or more remote operation actuators that provide some controller operations, and the surgical instrument 26 converts its operations into various movements of an end effector on the surgical instrument 26. Thus, the remote operation actuators within the instrument carriage 75 move only one or more components of the surgical instrument 26, rather than the instrument as a whole. The inputs for controlling the instrument as a whole or the components of the instrument are such that the inputs provided to the control inputs by the surgeon or other medical personnel ("master" commands) are converted into corresponding operations by the surgical instrument ("slave" responses).

[0031] In an alternative embodiment, the instrument carriage 75 does not house a remote actuator. The remote actuators that enable the various movements of the end effector of the surgical instrument 26 are housed at a location remote from the instrument carriage 75, for example, at other locations on the patient side cart 54. Using a cable-based force transmission mechanism or the like, each movement of the remotely located remote actuator is transmitted to the corresponding instrument interface actuator output disposed on the instrument carriage 75. In some embodiments, the surgical instrument 26 is mechanically coupled to a first actuator that controls a first movement such as rotation of the surgical instrument in the longitudinal direction (z-axis). The surgical instrument 26 is mechanically coupled to a second actuator that controls a second movement such as two-dimensional (x, y) movement of the surgical instrument. The surgical instrument 26 is mechanically coupled to a third actuator that controls a third movement of the surgical instrument such as opening and closing or a jaw end effector.

[0032] FIG. 5B is an exemplary simplified block diagram showing an exemplary positioning of the robotic support arms 72A-72C of the robotic surgery system 10 during a surgical procedure, according to some embodiments. In some embodiments, the patient side system 54 includes at least three robotic support arms 72A-72C. In some embodiments, each of the robotic support arms 72A-72C includes first, second, and third segments 72-1, 72-2, and 72-3 that are rotatably attached. The centrally located robotic support arm 72 can support an endoscope camera 26C suitable for capturing an image within the field of view of the camera. The robotic support arms 72 to the left and right of the center can each support an instrument 26A and 26B, and the instruments 26A and 26B can manipulate anatomical tissue. During the setup of the surgical procedure, the support arm segments are pre-positioned to support the endoscope and instruments in the correct position and orientation for performing a medical procedure with robotic-assisted manipulation by the surgeon.

[0033] A user or operator O (generally a surgeon) performs a surgical procedure on a patient P by operating control input devices 36 such as a handle and a foot pedal at a master control console 16. The operator can view video frames of an image of a surgical site inside the patient's body through a stereoscopic display viewer 31. A computer processor 58 of the console 16 instructs the movement of endoscope surgical instruments 26A - 26C controlled remotely via a control line 159, and causes the movement of the instruments using a patient-side system 24 (also called a patient-side cart).

[0034] Figures 5C - 5E are explanatory diagrams showing a segmented mechanical support arm 72 in which first, second, and third segments 72-1, 72-2, and 72-3 are in three different horizontal positions, according to some embodiments. For the sake of brevity, only one mechanical support arm is shown, but the surgical system includes multiple support arms as shown in FIG. 1. The first segment 72-1 includes a first end 81-1 and a second end 81-2. The second segment 72-2 includes a first end 82-1 and a second end 82-2. The third segment 72-3 includes a first end 83-1 and a second end 8,3-2.

[0035] The first segment 72-1 is rotatably attached at its first end 81-1 to rotate horizontally about respective vertical axes 91 in a vertical support structure 74. The second segment 72-2 is rotatably attached at its first end 82-1 to rotate horizontally about respective vertical axes 92 at the second end 81-2 of the corresponding first segment 72-1. The third segment 72-3 is attached at its first end 83-1 to rotate horizontally about respective vertical axes 93 at the second end 82-2 of the corresponding second segment 72-2.

[0036] FIG. 5C is an explanatory diagram showing a state in which the three segments of the segmented mechanical support arm 72 extend completely to the right (from the perspective of the drawing). FIG. 5D is an explanatory diagram showing a state in which, with respect to their positions in FIG. 5C, the first segment 72-1 rotates clockwise around the vertical axis 91, the second segment 72-2 rotates counterclockwise around the vertical axis 92, and the rotational position of the third segment 72-3 does not change with respect to the second segment 72-2. FIG. 5E is an explanatory diagram showing a state in which, with respect to their positions in FIG. 5A, the first segment 72-1 rotates counterclockwise around the vertical axis 91, the second segment 72-2 rotates clockwise around the vertical axis 92, and the rotational position of the third segment 72-3 does not change with respect to the second segment 72-2.

[0037] FIGS. 5F-5G are explanatory diagrams showing specific details of the third segment 72-3 of FIGS. 5C-5E according to some embodiments, showing the third segment 72-3, also called an instrument manipulator, in two different vertical positions. The third mechanical support arm segment 72-3 includes first, second, third, and fourth sub-segments 101-104. The first sub-segment 101 includes a first end 101-1 and a second end 101-2. The second sub-segment 102 includes a first end 102-1 and a second end 102-2. The third sub-segment 103 includes a first end 103-1 and a second end 103-2. The fourth sub-segment 104 includes a first end 104-1 and a second end 104-2.

[0038] The first sub-segment 101 is rotatably attached at its first end 101-1 so as to rotate vertically about the respective horizontal axis 111 at the second end 82-2 of the second segment 72-2. The second sub-segment 102 is rotatably attached at its first end 102-1 so as to rotate vertically about the respective horizontal axis 112 at the second end 101-2 of the first sub-segment 101. The third sub-segment 103 is rotatably attached at its first end 103-1 so as to rotate vertically about the respective horizontal axis 113 at the second end 102-2 of the second sub-segment 102. The fourth sub-segment 104 is rotatably attached at its first end 104-1 so as to rotate vertically about the respective horizontal axis 113 at the second end 103-2 of the third sub-segment 103. According to some embodiments, the fourth sub-segment 104 includes an attachment structure 120 for firmly attaching a surgical instrument during a surgical procedure.

[0039] [[ID=,3]] FIG. 5F is an explanatory diagram showing a state in which the four sub-segments of the third mechanical support arm segment 72-3 are in a substantially folded or reduced configuration position according to some embodiments. FIG. 5G is an explanatory diagram showing a state in which the four sub-segments of the third mechanical support arm segment 72-3 are partially in the configuration position according to some embodiments. More specifically, in FIG. 5G, the rotational position about the horizontal axis 111 of the first sub-segment 101 has not changed relative to its rotational position in FIG. 5F. The second sub-segment 102 has rotated clockwise about the horizontal axis 112 relative to its rotational position in FIG. 5F. The third sub-segment 103 has rotated counterclockwise about the horizontal axis 113 relative to its rotational position in FIG. 5F. The fourth sub-segment 104 has rotated clockwise about the horizontal axis 114 relative to its rotational position in FIG. 5F.

[0040] FIG. 6 is a side view of a surgical instrument 26 including a distal end portion 650 and a proximal control mechanism 640 coupled by an elongate tube 610 having a central axis 611 of the elongate tube. The surgical instrument 26 is configured to be inserted into a patient's body and is used to perform surgical or diagnostic procedures. The distal end portion 650 of the surgical instrument 26 can provide any of a variety of end effectors 654 such as the forceps, needle driver, cautery device, cutting tool, imaging device (e.g., an endoscope or ultrasonic probe) shown. The surgical end effector 654 can include functional mechanical degrees of freedom such as jaws that open and close, or a knife that translates along a path. In the illustrated embodiment, the end effector 654 is coupled to the elongate tube 610 by a wrist 652 that allows the end effector to be oriented with respect to the central axis 611 of the elongate tube. The surgical instrument 26 can also include stored (e.g., stored in a semiconductor memory associated with the instrument) information that can be either permanent or updatable by a surgical system configured to operate the surgical instrument 26. Accordingly, the surgical system can provide one-way or two-way information communication between the surgical instrument 26 and one or more components of the surgical system.

[0041] FIG. 7 is a perspective view of a third arm segment 72-3, also referred to as a surgical instrument manipulator, according to some embodiments. The third arm segment 72-3 is shown in a state where no surgical instrument is installed. The third arm segment 72-3 includes first, second, third, and fourth sub-segments 101-104 rotatably attached about horizontal axes 111-114 as illustrated and described with reference to FIGS. 5F-5G. The third arm segment 72-3 further includes an instrument carriage 75 that can removably connect a surgical instrument (e.g., surgical instrument 26). The instrument carriage 75 houses a plurality of remote actuators. Each remote actuator includes an actuator output 705. When a surgical instrument is installed on the instrument manipulator 72-3, one or more instrument inputs (not shown) of a proximal control mechanism of the instrument (e.g., proximal control mechanism 640 of FIG. 6) are mechanically coupled to a corresponding actuator output 705. In one aspect, this mechanical coupling is direct, and the actuator output 705 directly contacts the corresponding instrument input. In another aspect, this mechanical coupling occurs via an intermediate interface such as a component of a drape configured to provide a sterile barrier between the third segment / instrument manipulator 72-3 and the associated surgical instrument.

[0042] In one aspect, the movement of one or more instrument inputs by a corresponding remotely operated actuator results in the movement of the mechanical degrees of freedom of the surgical instrument. For example, in one aspect, the surgical instrument installed on the instrument manipulator 72-3 is the surgical instrument 26 shown in FIG. 6. Referring to FIG. 6, in one aspect, the movement of one or more instrument inputs of the proximal control mechanism 640 by a corresponding remotely operated actuator causes the elongated tube 610 (and the attached wrist 652 and end effector 654) to rotate about the central axis 611 of the elongated tube with respect to the proximal control mechanism 640. In another aspect, the movement of one or more instrument inputs by a corresponding remotely operated actuator results in the movement of the wrist 652 and aligns the end effector 654 with respect to the central axis 611 of the elongated tube. In another aspect, the movement of one or more instrument inputs by a corresponding remotely operated actuator results in the movement of one or more movable elements (e.g., jaw members, knife members, etc.) of the end effector 654. Therefore, the various mechanical degrees of freedom of the surgical instrument installed on the instrument manipulator 26 can be moved by the operation of the remotely operated actuator of the instrument carriage 75.

[0043] Annotate the recorded video FIG. 8 shows a schematic diagram of an exemplary surgical planning tool 800. In one aspect, the surgical planning tool 800 includes a robotic surgery system 850 that communicates with an electronic medical device record database 830. The robotic surgery system 850 shown here is similar to the robotic surgery system shown in FIG. 4. In one aspect, the electronic medical record database 830 includes the medical records of patients who have been treated at a particular hospital. The database 830 can be implemented on a server located within the hospital premises. The medical record entries included in the database 830 can be accessed from the hospital's computers via an intranet network. Alternatively, the database 830 can be implemented on a remote server located at a location away from the hospital, for example, using one of a number of cloud data storage services. In this case, the medical record entries of the database 830 are stored on the cloud server and can be accessed by a computer having Internet access.

[0044] In one aspect, a surgical procedure is performed on a first patient using the robotic surgery system 850. The imaging device associated with the robotic surgery system 850 captures an image of the surgical site and displays the captured image as a video frame on the display of the surgeon console. In one aspect, a healthcare provider at the surgeon console 52 uses the input device of the surgeon console 52 to highlight or annotate the anatomical structures of a particular patient shown in the displayed video. An example of an input device and the like is the control input 36 shown in FIG. 2, which is coupled to a cursor that operates in association with a graphical user interface overlaid on the displayed video. The graphical user interface can include a QWERTY keyboard, a pointing device such as a mouse and an interactive screen display, a touch screen display, or other means for data or text input. Thus, the healthcare provider can highlight a particular tissue of interest within the displayed image or enter a text annotation.

[0045] In one aspect, the video of the surgical site is further displayed on a display disposed on the electronic device cart 56. In one aspect, the display of the electronic device cart is a touch screen user interface that can be used by medical personnel to highlight and annotate specific portions of the anatomical structure shown in the image displayed for viewing on the display of the electronic device cart. The user can highlight a portion of the displayed image by touching a portion of the patient's anatomical structure displayed on the touch screen user interface. Further, a graphic interface including a QWERTY keyboard can be overlaid on the displayed image. The user can use the QWERTY keyboard to enter text annotations.

[0046] In one aspect, the video of the surgical site captured by an imaging device associated with the teleoperated surgical system 850 is stored in the database 830 in addition to being recorded by the teleoperated surgical system 850 and displayed to the user in real time or near real time. Highlighting and / or annotations associated with the recorded video made by the user can also be stored in the database 830. In one aspect, the highlighting made by the user is embedded in the recorded video before being stored in the database 830. Later, the recorded video can be retrieved for viewing. In one aspect, a viewer of the recorded video can select whether to display the highlighting or suppress the display. Similarly, the annotations associated with the recorded video can also be stored in the database 830. In one aspect, the annotations made by the user are used to tag the recorded video and can be provided as a means to identify the subject matter included in the recorded video. For example, one annotation may describe the state of a particular medical condition. This annotation is used to tag the recorded video. Later, a person who wishes to view the recorded procedure related to this medical condition can find the video using a keyword search.

[0047] Searching for Saved Videos In some cases, it is desirable for healthcare providers to be able to view video recordings of past surgical procedures performed on a given patient. In one aspect, a patient who has previously undergone a first surgical procedure to treat a medical condition may subsequently require a second surgical procedure to treat a recurrence of the same medical condition or to treat an anatomical structure located near the surgical site of the first surgical procedure. In one aspect, the events at the surgical site of the first surgical procedure are captured in a video recording of the surgical site, and the video recording is stored in database 830 as part of the patient's electronic medical record. Before performing a second surgical procedure on the patient, a healthcare provider can search database 830 to find the video recording of the patient's previous surgical procedure.

[0048] In some cases, it is desirable for healthcare providers planning to perform a surgical procedure on a patient to be able to view video recordings of similar surgical procedures performed on persons with similar specific characteristics as the patient. In one aspect, the video recordings of the surgical sites of the surgical procedures can be tagged with metadata information such as the patient's age, gender, body mass index, genetic information, type of treatment the patient has received, etc. before each video recording is stored in database 830. In one aspect, the metadata information used to tag the video recordings is automatically retrieved from the patient's existing medical records at that time, and then the metadata information is used to tag the video recordings before storing the video recordings in database 830. Thus, before performing a medical treatment on a patient, a healthcare provider can search database 830 for video recordings of similar procedures performed on patients sharing specific characteristics in common with the patient. For example, if a healthcare provider is planning to perform a prostatectomy on a 65-year-old male patient with a high body mass index using the robotic surgical system 850, the healthcare provider can search database 830 for video recordings of the surgical sites of prostatectomies performed using the robotic surgical system 850 on other male patients of a similar age with a similarly high body mass index.

[0049] In one aspect, the video recording of the surgical procedure is communicated (as indicated by the dashed line) by the database 830 to the optional personal computer 820 and made viewable to the healthcare provider planning to perform the surgical procedure. Additionally or alternatively, the video recording of a previous surgical procedure is communicated by the database 830 to the teleoperated surgical system 850 and may be made viewable preoperatively or intraoperatively. In one aspect, the video recording is displayed on a display located at the surgeon console 52 by the teleoperated surgical system 850. In another aspect, the video recording of the first surgical procedure is displayed on a display located on the electronic device cart 56.

[0050] Cloud-based video database In one aspect, the database 830 is implemented on a remote server using a cloud data storage service and is accessible by a plurality of healthcare providers. Referring to FIG. 8, as shown by the dashed lines, the surgical planning tool 800 optionally includes a remote surgery system 850 (as shown by the dashed lines) and a personal computer 840 (as shown by the dashed lines). In one aspect, the remote surgery system 850 is similar to the remote surgery system 850 and the personal computer 840 is similar to the personal computer 820, except that the remote surgery system 850 and the personal computer 820 are located at a first healthcare provider and the remote surgery system 850 and the personal computer 840 are located at a second healthcare provider. In one aspect, a first patient requires surgical treatment for a medical condition and undergoes a surgical procedure using the remote surgery system 850 at the first healthcare provider. The video recording of the surgical procedure is stored in the database 830. Thereafter, a second patient plans to undergo surgical treatment for the same medical condition and undergoes a surgical treatment using the remote surgery system 850 at the second healthcare provider. Before performing the surgical procedure on the second patient, a healthcare professional accesses the database 830 via a secure Internet connection and searches the database 830 for video recordings of the surgical site of a similar procedure. In one aspect, the healthcare professional treating the second patient can search the database 830 for the video recording of the surgical procedure of the first patient without knowing the identity of the first patient. In this way, the privacy of the first patient is protected. In one aspect, the video recording of the surgical procedure of the first patient includes highlights and / or annotations made by the healthcare professional who treated the first patient.

[0051] Computer-based pattern matching and analysis The surgical planning tool 800 may include pattern matching and analysis algorithms implemented in the form of computer-executable code. In one aspect, the pattern matching and analysis algorithms are stored in the non-volatile memory device of the surgical planning tool 800 and configured to analyze video records stored in the database 830. As described above, each video record stored in the database 830 may be tagged and / or embedded with specific metadata information. This metadata information may include patient information such as the patient's age, gender, and other information describing the patient's health status or medical history. Further, as described above, the metadata information may include highlights or annotations made by medical personnel. In one aspect, these highlights and annotations are embedded in the video record and stored in the database 830 along with the video.

[0052] In one aspect, the pattern matching and analysis algorithms include an image analysis component that identifies patterns of shapes and colors shared among a plurality of video records stored in the database 830. Next, the pattern matching and analysis algorithms consider the tagged metadata associated with this subset of video records and determine whether words or phrases are frequently associated with the video within this subset. These analyses performed by the pattern matching and analysis algorithms can be used to assist medical personnel in making decisions about the patient's anatomical structure, preferred surgical approach, medical condition, potential complications, etc.

[0053] Method of Using the Surgical Planning Tool FIG. 9 shows a method 900 of using a surgical planning tool. In one aspect, the surgical planning tool is similar to the surgical planning tool 800 of FIG. 8. At 910, facts or characteristics describing a medical patient, such as the medical condition the patient is suffering from, are received by a medical device. The medical device can receive this fact or situation via a user interface disposed in a robotic surgery system (e.g., the robotic surgery system 10 of FIG. 1 or the robotic surgery system 50 of FIG. 4), or via a personal computer similar to the personal computer 820 of FIG. 2. At 920, the medical device uses the facts or features received at 910 to search a medical device database for at least one relevant video record of a surgical procedure. At 930, the medical device uses the video record to determine surgical planning information. In one aspect, the surgical planning information includes the types of instruments used in the recorded procedure. At 940, the medical device displays the surgical planning information determined at 930 to the user.

[0054] Positioning of the mechanical support arm and positioning of the surgical system module It will be appreciated that the positioning of the robotic surgical arm 72, including the positioning of its plurality of sub-segments 72-1, 72-2, 72-3 during a surgical procedure, may depend on various factors such as the type of surgical procedure, the patient's anatomical features, the position of the patient's body, the layout of the operating room, etc. Different surgeries may involve different surgical incision patterns. If the patient's anatomical structure is different, the spacing between the incisions used in the surgical procedure may be different. Different patients may require different body positions or orientations with respect to the operating room table during surgery. If the layout of the operating room is different, it may be necessary to reposition the robotic surgical arm to address the physical limitations imposed by the operating room layout.

[0055] Figures 10A - 10B are exemplary top views of two exemplary patients 1202, 1206 with different anatomical dimensions having similar incision patterns suitable for the same first surgical procedure, but with different intervals between the incisions due to different anatomical spacings. Figure 10A is an explanatory diagram representing a first patient 1202 having a first surgical incision 1204 of a first pattern with smaller anatomical dimensions and a first interval. Figure 10B is an explanatory diagram representing a second patient 1206 having a second incision 1208 of the first pattern with larger anatomical dimensions and a second interval. Since each patient undergoes the same surgery, it will be understood that the pattern of the first incision 1204 of the first patient 1202 and the second incision 1208 of the second patient 1206 is similar. However, the interval between the first incision 1204 of the first patient 1202 and the second incision 1208 of the second patient 1206 is different because the anatomical dimensions of the two patients are different. Each of the plurality of mechanical support arms 72 in Figures 5A - 5G, including each of its plurality of sub - segments 72 - 1, 72 - 2, 72 - 3, is arranged to insert the instrument 26 into the first incision 1204 of the first patient 1202 and is set to a different position to insert the instrument 26 into the second incision of the second patient. The setting information of the mechanical support arm is recorded in a computer - readable storage device using the processor 58, for example, as will be more fully described below.

[0056] Figures 11A-11B are exemplary top views of two different exemplary patients 1212, 1216 having different anatomical dimensions and having similar incision patterns 1214, 1218 suitable for the same second surgical procedure. The third and fourth patients 1212, 1216 have different anatomical intervals due to different body sizes, so the intervals between those incisions are different. It can be seen that the incision pattern of the second surgical procedure shown in FIG. 11A is different from the incision pattern of the first surgical procedure shown in FIGS. 10A-10B. It will be understood that the positions of the plurality of mechanical support arms 72 in FIGS. 5A-5G are different for the first surgical procedure and the second surgical procedure. The setting information of the mechanical support arm is recorded in a computer-readable storage device using the processor 58, for example, as will be more fully described below.

[0057] Figures 12A-12B are side views of exemplary patients 1302, 1306 disposed on an operating room table 14 at different tilt angle positions, having received the same surgical procedure and having a surgical instrument of the same type inserted into a surgical incision. FIG. 12A is an explanatory view showing an exemplary first patient 1302 lying on the operating table 14 and aligned parallel to the ground such that the head, abdomen, and legs are horizontal. Four surgical instruments are inserted into the surgical incision of the first patient's body for use in a surgical procedure. FIG. 12B is an explanatory view showing an exemplary second patient 1306 lying on the operating table 14 and tilted at an angle such that the head is lifted higher and the legs are lower. Four surgical instruments are inserted into the surgical incision of the second patient's body for use in a surgical procedure. It will be appreciated that each of the plurality of mechanical support arms 72 of FIGS. 5A-5G, each including its respective plurality of sub-segments 72-1, 72-2, 72-3, is arranged to insert the instrument 26 into the incision of the first patient 1202 and is set to a different position to insert the instrument 26 into the incision of the second patient 1204. Different ranges of angles can be used, for example, a range of 0° straight for the first patient 1302 and a range of 30 degrees downward for the second patient 1306. In such a case, the kinematics of the arm 72 can be used to readjust the arm configuration of the patient 1306 corresponding to the endoscope configuration. The setting information of the mechanical support arm is recorded in a computer-readable storage device using the processor 58, for example, as will be more fully described below.

[0058] It will be appreciated that the positioning of the surgical system modules, including the positioning of the master control console 16, the surgical instrument tray 30, the equipment cart 24, and the patient-side cart 54, may depend on various factors such as the type of surgical procedure, the layout of the operating room, and the behavior patterns of the operating room staff during the surgery. It may be necessary to position the system modules differently if the layout of the operating room is different because the fixtures in the operating room are different. For example, different behavior patterns of the operating room staff may require different positions of the system modules to facilitate the efficient and safe movement of the staff during the surgical procedure.

[0059] Figures 13A to 13C are explanatory diagrams showing the arrangements of three different surgical system modules. Figure 13A is an explanatory diagram including fixtures F1 to F3 arranged as shown with the operating room illustrated, and surgical system modules 16, 30, 24, and 54 positioned in a first arrangement as shown. The fixtures may include structures such as poles, cabinets, or walls, or obstacles such as electrical outlets. Figure 13B is an explanatory diagram including fixtures F1 to F3 arranged in the operating room as in Figure 13A, and also including fixture F4 arranged as shown, and surgical system modules 16, 30, 24, and 54 positioned in a second arrangement as shown. Figure 13C is an explanatory diagram including fixtures F5 to F7 arranged as shown with the operating room illustrated, and surgical system modules 16, 30, 24, and 54 positioned in a third arrangement as shown.

[0060] Referring again to Figures 5A to 5B, a position sensor 1000 for sensing the position of an object in an operating room according to some embodiments is shown. During surgery, the position sensor 1000 senses the position of the patient, the position of the fixtures in the operating room, the position of the system modules, and the position of the personnel in the operating room. Figures 5A to 5B show position sensors 1000 arranged at the four corners at a height such as to have a line of sight to the patient. In Figure 5B, only two of the four sensors 1000 are visible. The position sensor generates information indicating the position of anatomical features of the patient, such as the body surface, the posture of the patient's skeleton on the operating table, and the relative posture of the surgical system with respect to the patient's anatomical structure. In some embodiments, the position sensor includes a laser scanning sensor that scans the operating room and its contents to determine the position of items in the room. In some embodiments, the position sensor 1000 includes an infrared (IR) scanning sensor. The sensed patient position information, operating room layout, and information indicating the position of the system modules are recorded in a computer-readable storage device using a processor 58, for example as will be more fully described below. In some embodiments, surgical personnel wear RFID tags that can be tracked to identify personnel in the operating room.

[0061] FIG. 14 is an explanatory diagram showing a storage atlas 1002 in a computer-readable storage device 1004 according to some embodiments. The storage atlas 1002 includes a first information structure 1006 showing examples of previous surgical procedures. A second information structure 1008 associates a surgical procedure with the anatomical features of the patient, the placement of the patient on the operating table, and the position of the mechanical support arm during the surgical procedure. A third information structure 1010 associates a surgical procedure with the layout features of the operating room, the behavior patterns of the operating room staff during the surgical procedure, and the positions of the surgical system modules during the surgical procedure. A first rule information structure 1012 associates the anatomical feature information of the patient and the position information of the patient with a mechanical setting control signal. A second rule information structure 1014 associates the layout information of the operating room and the staff information of the operating room with the position information of the system modules. FIG. 15 is an explanatory diagram showing an exemplary example of the second information structure 1008 included in the atlas 1002 according to some embodiments. FIG. 16 is an explanatory diagram showing an exemplary example of the third information structure 1010 included in the atlas 1002 according to some embodiments.

[0062] FIG. 17 is an explanatory diagram showing an exemplary first rule information structure 1012 for associating the anatomical feature information of the patient and the position information of the patient with a position control signal of a mechanical support arm according to some embodiments. The first rule is developed based on data from previous surgeries represented in the first and second data information structures 1006, 1008. The first rule correlates the pattern of the patient's anatomical structure and the patient's position with a control signal used to control the setting of the position of the mechanical support arm 76 for use during a first type of surgical procedure.

[0063] More specifically, for an exemplary first type of surgical procedure, the first rule information structure 1012 includes a surgical procedure signature (SigST1 ··· SigST l ), the anatomical structure signature of the patient (SigAn1 ··· SigAn n ), and the position signature of the patient (SigPos1 ··· SigPosm ) to a mechanical arm position control signal (CNTL ARM1 ···CNTL ARMP ). In some embodiments, the patient's anatomical structure signature includes a multi-dimensional vector. In some embodiments, the patient's anatomical structure signature indicates patient health record information such as weight, height, age, demographic information, and preoperative three-dimensional images of the patient such as computed tomography (CT), magnetic resonance imaging (MRI), etc. Further, the patient's anatomical structure signature can include surgical planning information such as structured light imaging systems, or body surface scans of the patient using other depth sensing techniques such as time-of-flight and radio frequency imaging, and geometric or physical-based models used to estimate deformations of the patient's body in a given configuration generated from the above information. In some embodiments, the patient's position signature includes a multi-dimensional vector. In some embodiments, the scanned patient position information is used to determine patient position information such as three-dimensional models of the patient's body on the operating table obtained by intraoperative imaging techniques including injection pressure, measurements from pressure mats on the operating table, cone beam CT, intraoperative CT or MRI, surface scanning techniques, or multi-viewpoint surface reconstruction from multiple cameras. According to some embodiments, different mechanical arm position control signals are associated with different combinations of the patient's anatomical structure signature and the patient's position signature. The patient position information functions as rough information indicating where the patient's body is located within the operating room. The patient anatomical information functions as detailed information indicating the location of specific anatomical features of the patient's body.

[0064] According to some embodiments, machine learning techniques can be used to generate a first rule corresponding to a patient's anatomical feature signature and a patient's position signature. More specifically, for example, a classifier can be used together with expert knowledge to correlate a patient's anatomical feature signature and a patient's position signature with a robotic arm position control signal. The surgical data within the first and second data information structures 1006, 1008 is evaluated, for example, based on the input of an expert surgeon, to determine the appropriate positions of the robotic arm 72 including the positions of each of those segments 72-1, 72-2, and 72-3 and the positions of each of those sub-segments 101-104. A control signal for achieving the determined set position of the robotic arm is associated in the first rule information structure 1012 with the patient's anatomical feature signature and the patient's position signature. According to some embodiments, the patient's anatomical feature signature and the patient's position signature can be combined together to generate a composite signature corresponding to the control signal.

[0065] FIG. 18 is an exemplary flow diagram depicting a process 1802 for generating a control signal based at least in part on a patient's anatomical information, a patient's position information, and a type of surgical procedure, according to some embodiments. A computer processor 58 is configured to execute process 1802 in accordance with some embodiments. During the setup of a surgical procedure, a rule block 1804 and a match block 1814 determine whether to activate a control signal based on the patient's anatomical information, the patient's position information, the type of surgical procedure, and the rules from the first rule information structure 1012.

[0066] More specifically, during the setup of the mechanical support arm 72 for a surgical procedure using the system 10, the rule block 1804 receives patient anatomical information 1806 that can be input to the processor 58. The rule block 1804 receives patient position information input to the processor 58 obtained using the position sensor 1000. The rule block 1804 receives surgical type information that may include, for example, incision pattern information input to the processor 58.

[0067] Also, during the surgical procedure, the control signal rule memory block 1812 provides the rule block 1804 with patient anatomical feature signature information, patient position signature information, and surgical procedure type information from within the first rule block portion 1012A of the first control signal rule information structure 1012. The rule block 1804 compares the patient anatomical features, patient position, and surgical procedure type information respectively provided by blocks 1806, 1808, 1810 with the associated patient anatomical feature signature, patient position signature, and surgical system type information from the first rule block portion 1012A. In some embodiments, the computer processor 58 is configured to convert the patient anatomical feature information, patient position information, and surgical procedure type information obtained by the system 10 into a format suitable for comparison with the signature and status information from the first portion 1012A of the control signal rule information structure 1012. Specifically, in some embodiments, the raw patient anatomical features / patient position / surgical procedure type are processed to derive a classification (signal / probability), which is then looked up in a table to determine the position of the mechanical support arm based on the determined classification.

[0068] The decision module 1806 determines whether there is a match between the anatomical features of the patient, the patient's position, and the type of surgical procedure information provided on the one hand and the rule information from the first rule block portion 1012A on the other hand. In machine learning embodiments, it will be understood that the match is determined based on the range of similarity between the anatomical features of the patient, the patient's position, and the type of surgical procedure and the rule information. Thus, for example, a combination of the anatomical features of the patient, the patient's position, and the type of surgical procedure within a certain threshold limit of a particular rule is determined to match that rule.

[0069] In response to a determination that there is a match between the combination of the anatomical features of the patient, the patient's position, and the type of surgical procedure information and the rule, block 1814 activates a control signal from within the second rule portion 1012B of the control signal rule information structure 1012 corresponding to the matching rule. For example, in response to a determination that the anatomical features of the patient, the patient's position, and the type of surgical procedure information received during the surgical procedure match SigAn2, SigPos1, and SigSur1 of the first portion 1012A, block 1816 activates signal CNTL ARMS1 from within the second portion 1012B of the control signal information structure 1012. The activated control signal has a value corresponding to the combination of the anatomical position, the patient's position, and the surgical procedure signature with which it is associated as a signature within the information structure 1012. The control signal moves the mechanical support arm 72 to the position corresponding to these associated signatures. More specifically, according to some embodiments, the activated control signal causes a controlled movement of the rotational position around each of the axes 91 - 93 of the first to third segments 72-1 to 72-3 and has a value that causes a controlled rotational movement of the rotational position around each of the axes 111 - 114 of the first to fourth sub-segments 101 - 104.

[0070] FIG. 19 is an explanatory diagram showing an exemplary second rule information structure 1014 for associating operating room layout information and operating room personnel information with system module information according to some embodiments. The second rule is created based on data from previous surgeries represented in the first and third data information structures 1006, 1010. The second rule correlates the pattern of the operating room layout and the activities of the operating personnel with the position information of the system modules that can be presented on the display 60, and provides guidance regarding the placement of the surgical system modules during the setup of the surgical procedure.

[0071] For an exemplary first type of surgical procedure, the second rule information structure 1014 associates a surgical type signature (SigST1 ··· SigST l ), an operating room layout signature (SigLT1 ··· SigLT n ), and a personnel activity signature (SigPA1 ··· SigPA m ) with a display image identifier (ImageID1 ··· ImageID P ) of the system module layout information for display on the display screen 60. In some embodiments, the operating room layout signature includes a multi-dimensional vector. In some embodiments, the operating room layout signature indicates the position of fixtures, the dimensions of the room, attachment points, and models of articulated devices such as boom-mounted lights. In some embodiments, the personnel activity signature includes a multi-dimensional vector. In some embodiments, the recorded personnel activity information includes the role of the personnel, spatio-temporal movement traces, and behavioral history. According to some embodiments, different system module positions are associated with different combinations of the operating room layout signature and the operating room personnel activity signature. According to some embodiments, machine learning techniques can be used to generate the second rule. More specifically, for example, a classifier can be used together with expert knowledge to correlate the operating room layout signature and the operating room personnel activity signature with the system module positions.

[0072] FIG. 20 is an exemplary flowchart representing a process 2002 for generating computer display image information representing a recommended system module position based at least in part on operating room layout information, operating staff activity information, and surgical procedure type information, according to some embodiments. A computer processor 58 is configured to execute process 2002 according to some embodiments. During the setup of a surgical procedure, rule block 2004 and matching block 2014 determine whether to display an image showing a proposed positioning of the surgical system module based on patient anatomical information, patient location information, and the type of surgical procedure, and rules from a second rule information structure 1014.

[0073] More specifically, during the setup of surgical system modules 16, 24, 30, 54 for a surgical procedure, rule block 2004 receives operating room layout information 2006 that may be input to the processor 58. Rule block 2004 receives operating staff activity information input to the processor 58. Rule block 2004 receives surgical type information that may include incision pattern information, for example, input to the processor 58.

[0074] Also, during a surgical procedure, the image rule memory block 2012 provides the layout signature information of the patient's operating room, the activity signature information of the personnel, and the type information of the surgical procedure from within the second rule block portion 1014A of the second rule information structure 1014 to the rule block 2004. The rule block 2004 compares the operating room layout, the activity information of the surgical personnel, and the type information of the surgical procedure provided by blocks 2006, 2008, and 2010 respectively, with the relevant operating room layout signature, personnel activity signature, and surgical system type information from the second rule block portion 1014A. In some embodiments, the computer processor 58 is configured to convert the operating room layout information, the activity information of the surgical personnel, and the type information of the surgical procedure obtained by the system 10 into a format suitable for comparing with the signature and status information from the first portion 1014A of the control signal rule information structure 1014.

[0075] The decision module 2006 determines whether there is a match between the operating room layout, personnel activity, and type information of the surgical procedure provided on the one hand and the rule information from the second rule block portion 1014A on the other hand. In machine learning embodiments, it will be understood that the match is determined based on the range of similarity between the operating room layout, personnel activity, and type of surgical procedure and the rule information. Thus, for example, a combination of operating room layout, personnel activity, and type of surgical procedure within a certain rule - based threshold limit is determined to match that rule.

[0076] Block 2014 activates an image signal from within the second rule portion 1014B of the second rule information structure 1014 corresponding to the matching rule in response to a determination that there is a match between the operating room layout, personnel activity, and type information of the surgical procedure and the rule. For example, in response to a determination that the operating room layout, personnel activity, and type information of the surgical procedure received during the surgical procedure match SigOR2, SigAct1, and SigSur1 of the first portion 1014A, block 2016 activates signal IMG from within the second portion 1014B of the information structure 1014A1 Activate it. According to some embodiments, for example, an image such as an image representing the layout of one of the operating rooms and the positions of the system modules in FIGS. 13A - 13C is generated on the display screen 60.

[0077] Exemplary embodiments have been shown and described. In the foregoing disclosure, and in some cases, extensive modifications, changes, and substitutions are contemplated, and some features of the embodiments can be employed without the corresponding use of other features. For example, in some embodiments, the processor 58 is coupled to a memory device such as the storage device 1004 that includes a set of instructions executable on the processor 58 to cause the processor 58 to execute operations. In some embodiments, the operations include determining the position information of the patient for a surgical procedure during the setup of the surgical system. The operations further include determining a match between the position information of the patient determined during setup and each patient's position signature. The operations further include activating, during setup, a support arm control signal within the surgical system corresponding to each matched patient position signature.

[0078] Furthermore, in some embodiments, the processor 58 is coupled to a memory device such as the storage device 1004 that includes a set of instructions executable on the processor 58, and causes the processor 58 to execute operations including determining the layout information of the operating room for a surgical procedure during the setup of the surgical system. The operations further include determining a match between the layout information of the operating room determined during setup and each operating room's layout signature. The operations further include generating an image representing the positions of the surgical system modules corresponding to each matched operating room layout signature.

[0079] Those skilled in the art will recognize many variations, alternatives, and modifications. Thus, the scope of the present disclosure should be limited only by the appended claims, and it is appropriate to interpret the claims broadly and in a manner consistent with the scope of the embodiments disclosed herein.

[0080] The content of the claims as originally filed is described below as an example. [Example 1] A method for use with a remote surgery system, the method comprising: determining patient position information for a surgical procedure during setup of the surgery system; determining a match between the determined patient position information and respective patient position signatures during the setup; activating, within the surgery system, a support arm control signal corresponding to each matched patient position signature during the setup. Method. [Example 2] determining position information of anatomical features of a patient for the surgical procedure during setup of the surgery system; further comprising determining a match between the determined position information of the patient's anatomical features and respective anatomical feature position signatures; The activating step includes activating, within the surgery system, a support arm control signal corresponding to a combination of each matched patient position signature and each matched anatomical feature position signature, the method according to Example 1. [Example 3] The method according to Example 1, further comprising moving a mechanical support arm to a position corresponding to the activated control signal value. [Example 4] The method according to Example 1, further comprising moving a mechanical support arm segment to a rotational position about a horizontal axis corresponding to the activated control signal value. [Example 5] The method according to Example 1, further comprising moving a mechanical support arm segment to a rotational position about a vertical axis corresponding to the activated control signal value. [Example 6] moving a mechanical support arm segment to a rotational position about a vertical axis corresponding to the activated control signal value; The method according to Example 1, further comprising moving the mechanical support arm segment to a rotational position around the vertical axis corresponding to the activated control signal value. [Example 7] The method according to Example 1, wherein the step of determining the patient's position information includes scanning with a laser position sensor. [Example 8] The method according to Example 1, wherein the step of determining the patient's position information includes scanning with an infrared position sensor. [Example 9] For each of a variety of occurrences of the setup of one or more examples of the surgical system for performing a surgical procedure, recording the patient's position information during the setup and recording the position information of the support arm during the setup; The method according to Example 1, further comprising determining, at least partially based on the recorded patient position information and the recorded support arm position information, a respective patient position signature associated with each support arm position. [Example 10] For each of a variety of occurrences of the setup of one or more examples of the surgical system for performing a surgical procedure, recording the patient's anatomical feature information during the setup, recording the patient's position information during the setup, and recording the position information of the support arm during the setup; The method according to Example 2, further comprising determining, at least partially based on the recorded patient anatomical feature information, the recorded patient position information, and the recorded support arm position information, a respective combination of each patient's anatomical feature signature and each patient's position signature for association with each support arm position. [Example 11] A method for use with a teleoperated surgical system, the method comprising: Determining layout information of an operating room for a surgical procedure during the setup of the surgical system; A step of determining a match between the determined operating room layout information and the layout signature of each operating room during the setup; A step of generating an image representing the position of the surgical system module corresponding to the layout signature of the matching operating room, including: Method. [Example 12] A step of determining the activity information of the personnel for the surgical procedure during the setup of the surgical system; A step of determining a match between the determined activity information of the personnel and the activity signature of each personnel, further including: The step of generating an image includes generating an image representing the position of the surgical system module corresponding to the combination of the matching layout signature of each operating room and the matching activity signature of each personnel, according to the method described in Example 11. [Example 13] The step of determining the layout information of the operating room includes the step of scanning with a laser position sensor, according to the method described in Example 11. [Example 14] The step of determining the layout information of the operating room includes the step of scanning with an infrared position sensor, according to the method described in Example 11. [Example 15] For each of various occurrences of the setup of one or more examples of the surgical system for performing a surgical procedure, a step of recording the layout information of the operating room and recording the position of the system module during the setup; A step of determining the layout signature of each operating room associated with the layout of each operating room, at least partially based on the recorded layout information of the operating room and the recorded position information of the system module, further including: [Example 16] For each of the various occurrences of the setup of one or more examples of the surgical system for performing a surgical procedure, recording the layout information of the operating room during the setup, recording the activity information of the personnel during the setup, and recording the position information of the system modules during the setup; determining, at least partially based on the recorded layout information of the operating room, the recorded activity information of the personnel, and the recorded position information of the system modules, each combination of a respective layout signature of each operating room and a respective activity signature of each personnel for association with each system module; the method according to Example 12, further comprising. [Example 17] A surgical system including a surgical instrument and a surgical instrument actuator, the surgical system comprising: a processor; a memory device holding a set of instructions executable on the processor; and having causing the surgical system by the set of instructions to determine position information of a patient for a surgical procedure during setup of the surgical system; determine a match between the determined patient position information and a respective patient position signature during the setup; activate a support arm control signal corresponding to the matched patient position signature within the surgical system during the setup; and execute operations including A surgical system. [Example 18] The operations further include determining position information of anatomical features of the patient for the surgical procedure during the setup of the surgical system; determining a match between the determined position information of the anatomical features of the patient and a respective anatomical feature position signature; The starting up includes activating a support arm control signal corresponding to a combination of the respective patient position signatures that match and the respective anatomical feature position signatures that match within the surgical system, for the surgical system described in Example 17. [Example 19] The operation further includes moving a mechanical support arm to a position corresponding to the activated control signal value, for the surgical system described in Example 17. [Example 20] The operation further includes moving a mechanical support arm segment to a rotational position around a horizontal axis corresponding to the activated control signal value, for the surgical system described in Example 17. [Example 21] The operation further includes moving a mechanical support arm segment to a rotational position around a vertical axis corresponding to the activated control signal value, for the surgical system described in Example 17. [Example 22] The operation includes moving a mechanical support arm segment to a rotational position around a vertical axis corresponding to the activated control signal value, and moving a mechanical support arm segment to a rotational position around a vertical axis corresponding to the activated control signal value, for the surgical system described in Example 17. [Example 23] Determining the patient's position information includes scanning with a laser position sensor, for the surgical system described in Example 17. [Example 24] Determining the patient's position information includes scanning with an infrared position sensor, for the surgical system described in Example 17. [Example 25] The operation includes for each of a variety of occurrences of a setup of one or more examples of the surgical system for performing a surgical procedure, recording the patient's position information during the setup and recording the position information of the support arm during the setup, and The surgical system according to Example 17, further comprising determining a respective patient position signature associated with each support arm position based at least in part on the recorded patient position information and the recorded support arm position information. [Example 26] For each of a variety of occurrences of a setup of one or more examples of the surgical system for performing a surgical procedure, recording anatomical feature information of a patient during the setup, recording position information of the patient during the setup, and recording position information of a support arm during the setup. The surgical system according to Example 18, further comprising determining a respective combination of a respective patient anatomical feature signature and a respective patient position signature for association with each support arm position based at least in part on the recorded patient anatomical feature information, the recorded patient position information, and the recorded support arm position information. [Example 27] A surgical system including a surgical instrument and a surgical instrument actuator, the surgical system comprising: a processor; a memory device holding a set of instructions executable on the processor, and having: causing the surgical system by the set of instructions to determine layout information of an operating room for a surgical procedure during a setup of the surgical system; determine a match between the determined layout information of the operating room and a respective operating room layout signature during the setup; generate an image representing a surgical system module position corresponding to the matched operating room layout signature. Surgical system. [Example 28] Determining staff activity information for the surgical procedure during the setup of the surgical system. further comprising determining a match between the activity information of the determined personnel and the activity signature of each respective personnel, Generating an image includes generating an image representing the surgical system module positions corresponding to a combination of the layout signature of each respective operating room that matches and the activity signature of each respective personnel, the surgical system according to Example 27. [Example 29] Determining the layout information of the operating room includes scanning with a laser position sensor, the surgical system according to Example 27. [Example 30] The step of determining the layout information of the operating room includes the step of scanning with an infrared position sensor, the surgical system according to Example 27. [Example 31] For each of a variety of occurrences of the setup of one or more examples of the surgical system for performing a surgical procedure, recording the layout information of the operating room and recording the system module positions during the setup, further comprising determining, at least in part based on the recorded layout information of the operating room and the recorded position information of the system module, the layout signature of each respective operating room associated with the layout of each respective operating room, the surgical system according to Example 27. [Example 32] For each of a variety of occurrences of the setup of one or more examples of the surgical system for performing a surgical procedure, recording the layout information of the operating room, recording the activity information of the personnel during the setup, and recording the position information of the system module during the setup, further comprising determining, at least in part based on the recorded layout information of the operating room, the recorded activity information of the personnel, and the recorded position information of the system module, each combination of the layout signature of each respective operating room and the activity signature of each respective personnel for associating with each respective system module, the surgical system according to Example 28.

Claims

1. A surgical system, the surgical system comprising: one or more processors; and a memory storing instructions, wherein when the instructions are executed by the one or more processors, the surgical system is caused to perform operations including: receiving surgical procedure information; determining a match between the surgical procedure information and a surgical procedure signature; and generating an image representing the positioning of the surgical system based at least in part on the matched surgical procedure signature, wherein the operations further include: receiving operating room layout information; and determining a match between the operating room layout information and an operating room layout signature, wherein generating the image includes generating the image representing the positioning of the surgical system based at least in part on a combination of the match between the surgical procedure information and the surgical procedure signature and the match between the operating room layout information and the operating room layout signature, A surgical system.

2. The operations further include: receiving personnel activity information; and determining a match between the personnel activity information and a personnel activity signature, wherein generating the image includes generating the image representing the positioning of the surgical system based at least in part on a combination of the match between the surgical procedure information and the surgical procedure signature and the match between the personnel activity information and the personnel activity signature, the surgical system according to claim 1.

3. A surgical system, the surgical system comprising: one or more processors; and a memory storing instructions, wherein when the instructions are executed by the one or more processors, the surgical system is caused to perform operations including: receiving surgical procedure information; determining a match between the surgical procedure information and a surgical procedure signature; and generating an image representing the positioning of the surgical system based at least in part on the matched surgical procedure signature, wherein the operations further include: receiving personnel activity information; determining a match between the personnel activity information and a personnel activity signature; receiving operating room layout information; and determining a match between the operating room layout information and an operating room layout signature, Generating the image includes generating the image representing the positioning of the surgical system based at least in part on a combination of the match between the surgical procedure information and the surgical procedure signature, the match between the staff activity information and the staff activity signature, and the match between the operating room layout information and the operating room layout signature. Surgical system. **Claim 4** The surgical system according to any one of claims 1 to 3, wherein the image representing the positioning of the surgical system includes information representing the position of a recommended surgical system module. **Claim 5** The surgical system according to any one of claims 1 to 3, wherein generating the image includes generating the image based at least in part on patient anatomical information. **Claim 6** The surgical system according to claim 1 or 3, wherein the operating room layout signature includes a multi-dimensional vector. **Claim 7** The surgical system according to claim 1 or 3, wherein the operating room layout signature indicates one or more of the position of a fixture, the dimensions of the room, or the mounting points. **Claim 8** A method of operating a surgical system, the method of operation comprising: a step in which a processor of the surgical system receives surgical procedure information; a step in which the processor determines a match between the surgical procedure information and a surgical procedure signature; a step in which the processor generates an image representing the positioning of the surgical system based at least in part on the matched surgical procedure signature; a step in which the processor receives operating room layout information; a step in which the processor determines a match between the operating room layout information and an operating room layout signature, and the step of generating the image includes the step in which the processor generates an image representing the positioning of the surgical system based at least in part on a combination of the match between the surgical procedure information and the surgical procedure signature and the match between the operating room layout information and the operating room layout signature. Method of operation. **Claim 9** The method further includes a step in which the processor receives staff activity information, and a step in which the processor determines a match between the staff activity information and a staff activity signature. The step of generating the image includes the step of the processor generating the image representing the positioning of the surgical system based at least in part on a combination of the match between the surgical procedure information and the surgical procedure signature and the match between the staff activity information and the staff activity signature. The method of operation according to claim 8.

10. A method of operating a surgical system, the method of operation comprising: a step of a processor of the surgical system receiving surgical procedure information; a step of the processor determining a match between the surgical procedure information and a surgical procedure signature; a step of the processor generating an image representing the positioning of the surgical system based at least in part on the matched surgical procedure signature; a step of the processor receiving staff activity information; a step of the processor determining a match between the staff activity information and a staff activity signature; a step of the processor receiving operating room layout information; a step of the processor determining a match between the operating room layout information and an operating room layout signature, wherein the step of generating the image includes the step of the processor generating an image representing the positioning of the surgical system based at least in part on a combination of the match between the surgical procedure information and the surgical procedure signature, the match between the staff activity information and the staff activity signature, and the match between the operating room layout information and the operating room layout signature. Method of operation.

11. The image representing the positioning of the surgical system includes information representing the position of a recommended surgical system module. The method of operation according to any one of claims 8 to 10.

12. The step of generating the image includes the step of the processor generating the image based at least in part on patient anatomical information. The method of operation according to any one of claims 8 to 10.

13. The operating room layout signature includes a multi-dimensional vector. The method of operation according to claim 8 or 10.

14. The operating room layout signature indicates one or more of the position of a fixture, the dimensions of the room, or a mounting point. The method of operation according to claim 8 or 10.