Surgical support system and surgical support method

Abstract

To facilitate surgery using a surgical support system including multiple robot arms.SOLUTION: A surgical support system according to one embodiment comprises: an endoscopic camera; a surgical instrument; a plurality of robot arms having a camera arm which has multiple degrees of freedom and in which the endoscopic camera is attached to its distal end and an instrument arm which has multiple degrees of freedom and in which the surgical instrument is attached to its distal end; a display which displays an endoscopic image captured by the endoscopic camera; an operation input device which receives a user's operation; and a processing circuit. The processing circuit is configured to identify a surgical target that is a target of surgery using the surgical instrument in a patient's body, and operate the plurality of robot arms in response to the user's operation on the operation input device.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present disclosure relates to a surgery assistance system and a surgery assistance method. [Background technology]

[0002] Patent Document 1 discloses a physical medical element fixation system. The system includes a memory that stores instructions and a processor configured to execute the instructions. The processor performs the steps of accessing image data that depicts an image of a patient's interior space, the image depicting a physical medical element positioned on an anatomical surface within the interior space, and identifying an area within the image in which to depict a fiducial marker on the physical medical element. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] U.S. Patent Application Publication No. 2021 / 0153959 Summary of the Invention [Problem to be solved by the invention]

[0004] There is a need for an improved system and method for assisting a surgeon in surgery.

[0005] Therefore, an object of the present disclosure is to provide a surgery assistance system and a surgery assistance method that can make it easier to perform surgery using a surgery assistance system that includes multiple robotic arms. [Means for solving the problem]

[0006] A surgical assistance system according to one aspect of the present disclosure includes an endoscopic camera, a surgical instrument, a plurality of robotic arms including a camera arm having multiple degrees of freedom and having the endoscopic camera attached to its tip, and an instrument arm having multiple degrees of freedom and having the surgical instrument attached to its tip, a display for displaying endoscopic images captured by the endoscopic camera, an operation input device for accepting user operations, and a processing circuit, wherein the processing circuit is configured to identify a surgical target on a patient's body that is to be operated on using the surgical instrument, and to operate the plurality of robotic arms in response to the user's operation on the operation input device.

[0007] A surgical assistance method according to one aspect of the present disclosure is a surgical assistance method for assisting surgery using a surgical assistance system including a robotic arm having multiple degrees of freedom and a surgical instrument attached to a tip thereof, and a processing circuit, the method including acquiring, by the processing circuit, image information indicating an endoscopic image captured inside the patient's body, and identifying, by the processing circuit, a surgical target in the patient's body in the endoscopic image that is to be operated on using the surgical instrument, based on the image information.

[0008] A surgical assistance system according to another aspect of the present disclosure is a surgical assistance system comprising: a plurality of robotic arms each having a plurality of degrees of freedom; a plurality of medical instruments attached to the tip ends of the plurality of robotic arms, an operation input device that accepts user operations; and a processing circuit that operates the plurality of robotic arms in accordance with the user's operation on the operation input device, wherein the processing circuit is configured to: acquire surgical situation information indicating the situation of a surgery using the surgical instruments; and acquire recommended operation information indicating the content of the recommended operation from a learned model that has undergone machine learning to estimate a recommended operation that is recommended as an operation to be accepted by the operation input device in the surgical situation indicated by the surgical situation information, by inputting input information including the surgical situation information into the learned model. [Effects of the Invention]

[0009] According to the present disclosure, it is possible to provide a surgery assistance system and a surgery assistance method that can facilitate surgery using a surgery assistance system including multiple robotic arms. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram for explaining the configuration of a surgery support system according to a first embodiment. [Figure 2] FIG. 1 is a diagram illustrating an example of the configuration of a surgical support robot. [Figure 3] 3 is an enlarged perspective view of an arm base, an arm, and a medical instrument in the surgery support robot of FIG. 2. FIG. [Figure 4] FIG. 1 is an enlarged perspective view of an example tool of a surgical instrument. [Figure 5] FIG. 1 is a perspective view of an endoscopic camera. [Figure 6] FIG. [Figure 7] 1 is a block diagram showing an example of the configuration of a control system of a surgery support system according to a first embodiment. FIG. [Figure 8] 10 is a flowchart showing the flow of a first example of a surgical procedure. [Figure 9] FIG. 1 is a diagram illustrating a trocar. [Figure 10] FIG. 10 is a diagram illustrating roll-in. [Figure 11] FIG. 10 is a diagram illustrating teaching of a pivot position. [Figure 12] 10A and 10B are diagrams illustrating the movement of a medical instrument inserted into a trocar. [Figure 13] 10 is an example of a screen displayed on a display of a remote control device. [Figure 14] 10 is an example of a screen displayed on a display of a remote control device. [Figure 15] 10 is an example of a screen displayed on a display of a remote control device. [Figure 16] 10 is an example of a screen displayed on a display of a remote control device. [Figure 17] 10 is an example of a screen displayed on a display of a remote control device. [Figure 18] FIG. 10 is a diagram for explaining the configuration of a surgery support system according to a second embodiment. [Figure 19] 1 is a diagram illustrating an operating table device and an environmental information acquisition device, showing an operating room from the side. FIG. [Figure 20] FIG. 10 is a block diagram showing an example of the configuration of a control system of a surgery assistance system according to a second embodiment. [Figure 21] 10 is a flowchart showing the flow of a second example of a surgical procedure. [Figure 22] FIG. 1 is a diagram illustrating a three-dimensional model of a patient. [Figure 23] FIG. 10 is a diagram illustrating the identification of position coordinates of a surgical target. [Figure 24] 10 is an example of a screen displayed on a display of a remote control device. [Figure 25] 10 is an example of a screen displayed on a display of a remote control device. [Figure 26] 10 is an example of a screen displayed on a display of a remote control device. [Figure 27] 10 is an example of a screen displayed on a display of a remote control device. [Figure 28] 10 is an example of a screen displayed on a display of a remote control device. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment will be described with reference to the drawings.

[0012] First Embodiment [Configuration of surgical support system] FIG. 1 is a diagram for explaining the configuration of a surgery support system 1A according to the first embodiment, and is a diagram showing various elements included in the surgery support system 1A arranged in an operating room R, as viewed from above. As shown in FIG. 1, the surgery support system 1A includes a surgery support robot 10, a remote control device 20, and an operating table 31. In this embodiment, the surgery support system 1A is a system in which a surgeon U1 operates the remote control device 20 to perform a surgical operation, such as an endoscopic surgery, on a patient P using the surgery support robot 10, as in robot-assisted surgery and robotic telesurgery.

[0013] FIG. 1 also shows a display device 61, an instrument table 62, and the like that may be included in the surgery assistance system 1A. FIG. 1 also shows a surgeon U1 and a surgical assistant U2, who may be users of the surgery assistance system 1A. The display device 61 is placed in a position that is visible to the surgical assistant U2. Images from an endoscopic camera, etc. are displayed on the display device 61. In the following description, any user among the members of the medical team, including the surgeon U1 and the surgical assistant U2, may also be referred to as a user U.

[0014] In this embodiment, the surgery support system 1A is a system that uses a master-slave surgery support robot 10. The remote control device 20 constitutes the master device, and the surgery support robot 10 constitutes the slave device. The remote control device 20 is placed away from the surgery support robot 10, and the surgery support robot 10 is remotely controlled by the remote control device 20 during surgery. In the surgery support system 1A, a surgeon U1 operates the remote control device 20 to input commands, and the surgery support robot 10 performs an action corresponding to the command.

[0015] 1, during surgery, the surgery support robot 10 is placed beside an operating table 31 on which a patient P lies in an operating room R. In this embodiment, each component included in the surgery support system 1A is placed in the operating room R, but some of the components of the surgery support system 1A may be placed outside the operating room R or in a remote location. For example, the remote control device 20 may be placed outside the operating room R during surgery.

[0016] [Configuration of surgical assist robot] FIG. 2 illustrates an example of the configuration of a surgical support robot 10. The surgical support robot 10 constitutes an interface between the surgical support system 1A and the patient P. The surgical support robot 10 includes a support body 11, a positioner 12, an arm base 13, a plurality of arms 14, a plurality of medical instruments 15, and a first controller 16. In this embodiment, the surgical support robot 10 includes four arms 14, and a medical instrument 15 is attached to each arm 14.

[0017] The support 11, positioner 12, arm base 13, arm 14, and medical instrument 15 are connected in this order in a continuous manner from the support 11 to the medical instrument 15. In this specification and claims, the end facing the support 11 will be referred to as the "base end," and the end facing the opposite direction will be referred to as the "distal end."

[0018] (Support) The support body 11 supports the positioner 12. The support body 11 is configured to be freely movable. In this embodiment, the support body 11 is configured as a cart that can move within the operating room R. Specifically, the support body 11 includes a main body 111, a plurality of wheels 112 that support the main body 111, and a handle 113 that is connected to the main body 111. Note that, in FIG. 2, for simplicity of the drawing, only one wheel 112 is shown. The user U can grip the handle 113 to steer the support body 11 and move the surgical support robot 10 to any position. Note that, during surgery, the position of the support body 11 in the operating room R is fixed.

[0019] An interface device 114 is disposed near the handle 113 on the support 11. The interface device 114 is configured to receive operations from the user U, mainly for preparing for surgery before surgery. Specifically, the interface device 114 is configured to receive operations for moving or changing the posture of the positioner 12, the arm base 13, the multiple arms 14, or any combination thereof. The interface device 114 may include, as an input interface, a lever, a button, a touch panel, a joystick, or any combination thereof. The interface device 114 may include, as an output interface, a display.

[0020] For example, an image captured by a camera 139 included in the arm base 13 is displayed on a display included in the interface device 114. The camera 139 faces downward and captures an image of at least one of the operating table 31 and the patient P placed on the operating table 31. For example, the input interface included in the interface device 114 accepts an operation for changing the magnification of the image captured by the camera 139 and displayed on the display. The user U can adjust the position of the surgical support robot 10 relative to the operating table 31 or the patient P while viewing the image displayed on the display included in the interface device 114.

[0021] (positioner) As shown in FIG. 2 , the positioner 12 extends from the support 11 and connects the support 11 to the arm base 13. The positioner 12 has a link structure. The positioner 12 is configured as a multi-axis joint robot arm having multiple degrees of freedom. In this embodiment, the positioner 12 is configured as a vertical multi-joint robot arm having seven degrees of freedom. The positioner 12 can freely change the position and orientation of the arm base 13 relative to the support 11 in three-dimensional space. The configuration of the positioner 12 is not particularly limited as long as it is configured to support the arm base 13, and may be, for example, a linear motion device, an elevating device, a fixing device, or the like. The fixing device may be a bracket or the like that fixes the arm base 13 to a ceiling, a wall, or the like.

[0022] The positioner 12 includes a base 120, a plurality of positioner links, and a mechanical interface 127. The base 120 is fixed to an upper portion of the main body 111. The plurality of positioner links are sequentially connected and extend from the base 120. The plurality of positioner links include a first link 121 rotatably connected to the base 120 via a first joint JP1, a second link 122 swingably connected to the first link 121 via a second joint JP2, a third link 123 swingably connected to the second link 122 via a third joint JP3, a fourth link 124 swingably connected to the third link 123 via a fourth joint JP4, a fifth link 125 swingably connected to the fourth link 124 via a fifth joint JP5, and a sixth link 126 swingably connected to the fifth link 125 via a sixth joint JP6. The mechanical interface 127 is rotatably connected to the sixth link 126 via a seventh joint JP7. The mechanical interface 127 is connected to the arm base 13.

[0023] The positioner 12 is controlled by the first controller 16 and includes a positioner driving device 129 that drives the joints JP1, . . . , JP7 (see FIG. 7).

[0024] (Arm base) Fig. 3 is an enlarged perspective view of the arm base 13, the arm 14, and the medical instrument 15 of the surgical support robot 10 of Fig. 2. In this embodiment, each of the four arms 14 has substantially the same structure. For this reason, Fig. 3 shows only one arm 14 of the four arms 14 connected to the arm base 13, and the other three arms 14 are omitted.

[0025] The arm base 13 functions as a "hub" that serves as a base for the multiple arms 14. The positioner 12 and the arm base 13 form a manipulator support that movably supports the multiple arms 14.

[0026] The arm base 13 includes a main body 131, a positioner coupling part 132 arranged on the upper part of the main body 131, and a plurality of manipulator coupling parts 133 arranged on the lower part of the main body 131. The positioner coupling part 132 is coupled to the mechanical interface 128 of the positioner 12. The main body 131 has a certain longitudinal direction and is arch-shaped with the longitudinal direction being the extension direction of the chord. The plurality of manipulator coupling parts 133 are distributed and arranged in the longitudinal direction of the main body 131. In this embodiment, four manipulator coupling parts 133 are arranged. The base end part of the arm 14 is detachably coupled to each manipulator coupling part 133.

[0027] (arm) The arm 14 may also be referred to as a patient-side manipulator 14. The multiple arms 14 are detachably attached to and supported by the arm base 13. In this embodiment, as shown in FIG. 3, four arms 14A, ..., 14D are arranged side by side in the longitudinal direction of the arm base 13. Hereinafter, when the four arms 14A, ..., 14D are referred to without distinction, they may be referred to as "arms 14."

[0028] Each arm 14 has a link structure. Each of the multiple arms 14 is configured as a robot arm. In this embodiment, each of the multiple arms 14 is configured as a vertical articulated robot arm. The position and orientation of the tip of each of the multiple arms 14 can be freely moved in three-dimensional space relative to the arm base 13.

[0029] The arm 14 includes a base 140 and a plurality of arm links connected in sequence and extending from the base 140. The base 140 has an interface 140a configured to be attachable to the manipulator connection portion 133. The plurality of arm links include a first link 141 rotatably connected to the base 140 via a first joint JA1, a second link 142 swingably connected to the first link 141 via a second joint JA2, a third link 143 swingably connected to the second link 142 via a third joint JA3, a fourth link 144 swingably connected to the third link 143 via a fourth joint JA4, a fifth link 145 swingably connected to the fourth link 144 via a fifth joint JA5, a sixth link 146 swingably connected to the fifth link 145 via a sixth joint JA6, and a seventh link 147 swingably connected to the sixth link 146 via a seventh joint JA7.

[0030] The seventh link 147 is connected to the tip link 148 via an eighth joint JA8. The eighth joint JA8 is a linear joint that linearly movesably connects the tip link 148 to the seventh link 147. Specifically, the seventh link 147 has a certain longitudinal direction, and the tip link 148 is connected to the seventh link 147 so as to be movable in the longitudinal direction of the seventh link 147 relative to the seventh link 147.

[0031] The distal link 148 has an interface 148a configured to allow attachment of the medical instrument 15. By attaching the medical instrument 15 to the interface 148a, the medical instrument 15 is held at the distal end of the arm 14.

[0032] In the arm 14 described above, there are seven joints between the base 140 and the seventh link 147. That is, the arm 14 has redundant degrees of freedom for setting the seventh link 147 to a desired position and posture in three-dimensional space.

[0033] Each arm 14 includes an arm driving device 149 that is controlled by the first controller 16 and drives the joints JA1, . . . , JA8 of the arm 14 (see FIG. 7).

[0034] (Medical equipment) The multiple medical instruments 15 are used in surgery on patient P. A medical instrument 15 is held at the tip of each of the multiple arms 14. The multiple medical instruments 15 include an endoscopic camera 151 and multiple surgical instruments 152. Specifically, the endoscopic camera 151 is held at the tip of one of the multiple arms 14, and a surgical instrument 152 is held at the tip of each of the other three arms 14. Note that, hereinafter, the arm 14 of the four arms 14 to which the endoscopic camera 151 is attached may be referred to as the "camera arm 14," and the other three arms 14 to which the surgical instruments 152 are attached may be referred to as the "instrument arms 14."

[0035] 3 shows a surgical tool 152 at the tip of the arm 14. The surgical tool 152 may also be called an instrument. The surgical tool 152 is a tool used to perform a surgical procedure on a surgical target inside the body of a patient P.

[0036] In this specification and the claims, the term "surgical target" may refer to an affected area within the body of patient P, such as a lesion, or to a certain area within the body of patient P near the affected area that includes the affected area, or to a specific location within the body of patient P at or near the affected area. For example, the surgical target is a location that is the target of a surgical operation by the surgical instrument 152. For example, the surgical target may refer to an organ itself that includes the affected area, such as the small intestine or the stomach. For example, the surgical target may refer to a portion of the body of patient P on which a predetermined type of surgical operation is performed. The surgical target may also refer to only the portion of the body of patient P that comes into contact with the surgical instrument 152 (more specifically, the tool 152c described below) or a member grasped by the surgical instrument 152 (e.g., a suture needle) in order to perform a surgical operation. For example, in the case of performing surgery to remove a portion of the body of patient P, the surgical target may refer to the portion to be removed, or to the portion to be incised with a scalpel or the like. The surgical target may refer to a portion of bodily tissue on which a predetermined type of surgical operation is performed, or to the space within the body near the bodily tissue.

[0037] Each surgical instrument 152 has a base 152a configured to be removably attached to the interface 148a of the arm 14, a shaft 152b extending from the base 152a, and a tool 152c located at the tip of the shaft 152b.

[0038] There are multiple types of surgical instruments 152, each differing from the others in at least one of shape, size, and function. For example, the tool 152c of the surgical instrument 152 performs a surgical task according to its type. Examples of the types of surgical tasks include mechanical tasks such as grasping, blunt dissection, sharp dissection, ligation, suturing, and incision of the body tissue of the patient P, including blood vessels, as well as incision or coagulation of the body tissue using high-frequency current. Each surgical instrument 152 can function as forceps, a grasper, scissors, a stapler, a needle holder, or an electric scalpel.

[0039] The tool 152c operates in response to the operation of the surgeon U1. Specifically, the surgery support robot 10 is equipped with an instrument driving device 153 (see FIG. 7). The instrument driving device 153 is controlled by the first controller 16, which receives a command from the remote control device 20 in response to the operation of the surgeon U1, and operates the tool 152c.

[0040] FIG. 4 is an enlarged perspective view of an example of a tool 152c of the surgical instrument 152. The tool 152c shown in FIG. 4 functions as a forceps and includes a first link 152c1 rotatably connected to the shaft 152b via a first joint JT1, a second link 152c2 swingably connected to the first link 152c1 via a second joint JT2, and a pair of jaws 152c3 and 152c4 connected to the second link 152c2. The pair of jaws 152c3 and 152c4 are arranged to be able to open and close. That is, the base ends of the jaws 152c3 and 152c4 are connected to the second link 152c2 via a third joint JT3 and a fourth joint JT4 so as to be rotatable about the same line or lines parallel to each other.

[0041] The opening and closing movements and changes in orientation of the pair of jaws 152c3, 152c4 are achieved by driving them with the instrument driving device 153. Note that the tool 152c shown in Fig. 4 is an example, and the number of joints, shape, and the like are not limited to this. For example, depending on the type of surgical instrument 152, the tool 152c may not have any joints.

[0042] 5 is a perspective view of the endoscopic camera 151. For example, the endoscopic camera 151 includes an objective lens and a light guide. An endoscopic image captured by the endoscopic camera 151 is sent to the first controller 16 and then to the second controller 23 described below. However, the endoscopic image captured by the endoscopic camera 151 may also be sent to the second controller 23 without going through the first controller 16.

[0043] (Arm side controller) Returning to FIG. 3 , an arm-side controller 17 is disposed on each arm 14. In this embodiment, the arm-side controller 17 is fixed to the tip link 148 of each arm 14. The arm-side controller 17 includes a plurality of operation buttons 171 and a joystick 172. The plurality of operation buttons 171 include a manual operation button (which may also be referred to as an enable button) that allows the arm 14 to be manually moved while being pressed, and a button (which may also be referred to as a pivot button) that allows a pivot position, which will be described later, to be set when pressed.

[0044] [Configuration of remote control device] The configuration of the remote control device 20 will be described. Fig. 6 is a perspective view of the remote control device 20. The remote control device 20 constitutes an interface between the surgery support system 1A and the surgeon U1, and is used to receive input from the surgeon U1 and remotely operate the surgery support robot 10.

[0045] The remote control device 20 includes an operation input device 21 for receiving input of commands from the surgeon U1, a display device 22 for displaying endoscopic images captured by the endoscopic camera 151, a second controller 23, and a speaker 24. The operation input device 21 includes a pair of left and right operation manipulators 211L and 211R, a plurality of operation pedals 212, and a touch panel 213. The operation manipulators 211L and 211R are devices used to manually operate the surgery support robot 10, particularly the arms 14 and medical instruments 15. The operation input device 21 may also be referred to as an operation input tool.

[0046] The operating manipulators 211L and 211R are operated by the left and right hands, respectively, of the surgeon U1. The operating manipulators 211L and 211R have a link structure with multiple joints. The operating manipulators 211L and 211R receive operations to change the positions and postures of the endoscopic camera 151 and the surgical instrument 152. For example, the operating manipulators 211L and 211R receive operations to open or close the surgical instrument 152 of the corresponding arm 14.

[0047] The multiple operation pedals 212 include, for example, a pedal that accepts operation of the endoscopic camera 151 and a pedal that accepts operation of the medical instrument 15, such as incision with the surgical instrument 152. For example, the multiple operation pedals 212 include a pedal that accepts a switching command for the arm 14 operated by each of the operation manipulators 211L and 211R. That is, the arm 14 of the surgery support robot 10 is associated with the operation manipulators 211L and 211R by operation of the operation pedals 212, for example, and the associated arm 14 moves in accordance with the operation of the operation manipulators 211L and 211R.

[0048] The second controller 23 is communicably connected to the first controller 16. The second controller 23 receives information on the endoscopic image captured by the endoscopic camera 151 and displays it on the display device 22. The second controller 23 acquires operation inputs accepted by the operation input device 21 and transmits them to the first controller 16. That is, the surgeon U1 commands the movement and operation of the medical instrument 15 by directly moving the operation manipulators 211L and 211R while checking the affected area on the endoscopic image displayed on the display device 22.

[0049] In addition, a setting screen for making various settings of the remote control device 20 is displayed on the touch panel 213. The surgeon U1 can make various settings by operating the touch panel 213. For example, the setting screen displayed on the touch panel 213 allows the positions of the operation input devices 21, such as the operation manipulators 211L and 211R, and the position of the display device 22, to be adjusted before surgery. That is, the second controller 23 displays the setting screen on the touch panel 213 and, based on input information received from the touch panel 213, operates the operation input device 21 corresponding to the input information, such as a servo motor MM, which will be described later. In addition, in this embodiment, as will be described later, the second controller 23 can display an endoscopic image captured by the endoscopic camera 151 on the touch panel 213.

[0050] The second controller 23 outputs audio information from the speaker 24. The audio information may be various types of audio information stored in advance in the memory 232, such as warning sound information indicating an alarm sound. The audio information may also be audio information indicating a voice picked up by a microphone held by a user U other than the surgeon U1.

[0051] [Control system configuration of surgical support system] Fig. 7 is a block diagram showing an example of the configuration of a control system of the surgery support system 1A according to the first embodiment. In the surgery support robot 10, the first controller 16 is electrically connected to a positioner drive device 129, multiple arm drive devices 149, an endoscopic camera 151, and multiple instrument drive devices 153. The first controller 16 is also electrically connected to multiple arm-side controllers 17 and an interface device 114. Note that, for simplicity of illustration, Fig. 7 shows only one block of each of the arm drive device 149, instrument drive device 153, and arm-side controller 17.

[0052] The positioner driving device 129 includes servo motors MP1, ..., MP7 and rotation sensors EP1, ..., EP7. The servo motors MP1, ..., MP7 are arranged at the joints JP1, ..., JP7, respectively. The servo motors MP1, ..., MP7 are motors that rotate the joints JP1, ..., JP7, respectively. The rotation sensors EP1, ..., EP7 are sensors that detect the amount of rotation of the servo motors MP1, ..., MP7, respectively. The rotation sensors EP1, ..., EP7 are, for example, encoders.

[0053] The arm driving device 149 includes servo motors MA1, ..., MA8 and rotation sensors EA1, ..., EA8. The servo motors MA1, ..., MA8 are arranged at the joints JA1, ..., JA8, respectively. The servo motors MA1, ..., MA8 are motors that rotate the joints JA1, ..., JA8, respectively. The rotation sensors EA1, ..., EA8 are sensors that detect the amount of rotation of the servo motors MA1, ..., MA8, respectively. The rotation sensors EA1, ..., EA8 are, for example, encoders.

[0054] The instrument driving device 153 is a device for driving joints JT1, ..., JTn (n is a natural number) included in the surgical instrument 152. The instrument driving device 153 includes servo motors MT1, ..., MTn (n is a natural number) and rotation sensors ET1, ..., ETn (n is a natural number). The servo motors MT1, ..., MTn are motors that rotate and drive the joints JT1, ..., JTn, respectively. The rotation sensors ET1, ..., ETn are sensors that detect the amount of rotation of the servo motors MT1, ..., MTn, respectively. The rotation sensors ET1, ..., ETn are, for example, encoders.

[0055] In this embodiment, the tool driving device 153 is disposed within the tip link 148. That is, the servo motors MT1, ..., MTn are disposed within the tip link 148, and the driving forces of the servo motors MT1, ..., MTn are transmitted to the joints JT1, ..., JTn via power transmission mechanisms disposed within the base 152a and the shaft 152b.

[0056] Each of the positioner drive unit 129, arm drive unit 149, and tool drive unit 153 includes a reducer and a power transmission mechanism corresponding to each servo motor of each drive unit. The reducer reduces the output of the corresponding servo motor to amplify the torque. The power transmission mechanism transmits the output of the corresponding servo motor to a corresponding link or the like. The power transmission mechanism may be composed of multiple gears, a transmission wire, a transmission belt, or any combination thereof.

[0057] (Configuration and functions of the first controller) The first controller 16 includes at least one processor 161 such as a CPU, and a memory 162. The memory 162 is configured from a semiconductor memory such as a volatile memory or a nonvolatile memory, a hard disk, an SSD (Solid State Drive), or other storage device. The at least one processor 161 executes a program stored in the memory 162, causing the first controller 16 to perform various processes.

[0058] The first controller 16 includes servo amplifiers corresponding to the plurality of servo motors included in the positioner driving device 129, the arm driving device 149, and the tool driving device 153. Each servo amplifier is electrically connected to the corresponding servo motor. The servo amplifier calculates a drive current corresponding to a command value calculated by the processor 161 and supplies the drive current to the corresponding servo motor.

[0059] The first controller 16 controls the overall operation of the surgery assist robot 10. For example, the first controller 16 controls the operation of the positioner 12, the operation of each arm 14, and the operation of each medical instrument 15.

[0060] Various types of information required for robot-assisted surgery and its preparations are stored in the memory 162 of the first controller 16. The memory 162 stores information related to the content of the surgery, such as information indicating the surgical procedure, information indicating the surgical target, and information related to the surgical procedure.

[0061] Furthermore, for example, the memory 162 stores information regarding the configuration of elements included in the surgery support system 1A used in surgery. For example, the memory 162 may store information regarding the surgical instruments 152 used in surgery, information regarding the surgery support robot 10, information regarding the remote control device 20, etc. The information may be information regarding the type and dimensions of each element, such as the medical instruments 15, the surgery support robot 10, and the remote control device 20.

[0062] The memory 162 may store information about the surgical instrument 152 held at the tip of each arm 14. The information about the surgical instrument 152 may include information about the type, shape, dimensions, direction of movement, and range of movement of the surgical instrument 152.

[0063] The first controller 16 acquires image information indicating an endoscopic image captured by the endoscopic camera 151 and transmits it to the second controller 23.

[0064] (Configuration and function of the second controller) The first controller 16 of the surgery support robot 10 is connected to the second controller 23 of the remote control device 20 so as to be able to communicate with each other.

[0065] The second controller 23 includes at least one processor 231 such as a CPU, and a memory 232. The memory 232 is configured from a semiconductor memory such as a volatile memory or a nonvolatile memory, a hard disk, an SSD (Solid State Drive), or other storage device. The at least one processor 231 executes a program stored in the memory 232, causing the second controller 23 to perform various processes.

[0066] The second controller 23 controls the overall operation of the remote control device 20. Based on information, data, commands, etc. received from the first controller 16, the second controller 23 controls the operations to be performed by the operation manipulators 211L and 211R, etc., and the image display operation of the display device 22, etc.

[0067] For example, the second controller 23 receives image information indicating an endoscopic image captured by the endoscopic camera 151 from the first controller 16, and displays the endoscopic image on the display device 22. The second controller 23 may perform conversion processing or the like on the image information and display it on the display device 22.

[0068] The second controller 23 transmits information corresponding to the operation input received by the operation input device 21 as an operation command to the first controller 16. For example, the operation command may be a command related to the position, posture, or movement of the surgical instrument 152, or any combination thereof. For example, each of the operation manipulators 211L and 211R includes servo motors MM1, ..., MMn (n is a natural number representing the number of active joints) and rotation sensors EM1, ..., EMn (n is a natural number representing the number of active joints) that are respectively arranged at the multiple joints included in each manipulator. The rotation sensors EM1, ..., EMn are, for example, encoders. Note that in the figure, for simplicity, blocks of the servo motors MM1, ..., MMn and rotation sensors EM1, ..., EMn are shown only for the operation manipulator 211L. The detection values ​​of the rotation sensors EM of the operating manipulators 211L and 211R detect the amount of rotation of each joint, and the processor 231 of the second controller 23 generates an operation command based on the amount of rotation of each joint.

[0069] When the first controller 16 receives the operation command, it controls the operations of the arm driving device 149 and the tool driving device 153 based on the operation command.

[0070] The second controller 23 also receives a position command from the first controller 16. The position command includes position and posture commands corresponding to the positions and postures of the positioner 12 and arm 14 of the surgical support robot 10. For example, the position command includes commands corresponding to the rotation angles of the joints of the positioner 12 and arm 14. Based on the received position command, the second controller 23 controls the servo motors MM of the operating manipulators 211L and 211R so that the positions and postures of the operating manipulators 211L and 211R move in accordance with the position and posture of the tip link 148 of the arm 14.

[0071] Each of the first controller 16 and the second controller 23 may execute each process under centralized control by a single processor, or may execute each process under distributed control through cooperation of multiple processors. Each controller may be configured with at least one or a combination of two or more of a computer, a personal computer, a microcontroller, a microprocessor, a programmable logic device (PLD) such as a field-programmable gate array (FPGA), a programmable logic controller (PLC), and a logic circuit, for example.

[0072] The first controller 16 and the second controller 23 may be configured integrally. In other words, the processing of one of the first controller 16 and the second controller 23 may be performed by the other of the first controller 16 and the second controller 23. For example, the surgery assistance system 1A does not need to include the second controller 23, and an operation command may be sent directly from the operation input device 21 to the first controller 16, or image information may be sent directly from the first controller 16 to the display 22. One or more processors and one or more memories included in the first controller 16, the second controller 23, or any combination thereof is an example of a processing circuit.

[0073] The first controller and the second controller cooperate to execute various processes in the surgeon support stage described below, such as an identification process for identifying the surgical target area, camera adjustment control, instrument adjustment control, and an estimation process for estimating recommended operations. Details of these processes will be described later.

[0074] [First example of surgical procedure] Fig. 8 is a flow chart showing the flow of a first example of the procedure for performing surgery. The first example of the procedure for performing surgery shown in Fig. 8 will be described with reference to Figs. 9 to 17 as appropriate.

[0075] The steps in the first example of the surgical procedure are classified into a preparation stage and a surgeon assistance stage. The preparation stage is a stage in which various preparatory tasks are performed in the operating room R so that the surgical assistance system 1A is ready to perform surgical assistance. The surgeon assistance stage is a stage in which the surgery of surgeon U1 is supported after the preparation stage.

[0076] (Preparation stage) In the preparation stage, the user U determines the body position of the patient P and the posture of the operating table 31 during the surgery (step S1).

[0077] Depending on the content of the surgery, the user U considers and decides the body position of the patient P and the posture of the operating table 31. The user U adjusts the operating table 31 so that the patient P is in the decided posture, and has the patient P lie down on the operating table 31 so that the patient P is in the decided body position.

[0078] The user U determines the positions of multiple ports on the body surface of the patient P and inserts a trocar T into the determined positions (step S2). A port refers to an insertion position of the trocar T, and the trocar T is a tubular member that is inserted into the body wall of the patient P to guide the medical instrument 15 into the body cavity. The multiple ports correspond to the multiple arms 14, respectively.

[0079] Specifically, the user U determines the position of the port, that is, the insertion position of the trocar T on the body surface of the patient P, depending on the content of the surgery. As shown in Fig. 9, the user U inserts the trocar T into the determined position into the body wall of the patient P. Four medical instruments 15 are inserted into the four trocars T, respectively.

[0080] Next, the user U moves the surgery support robot 10 to the side of the operating table 31 so that it can support the surgery, that is, performs a so-called roll-in (step S3).

[0081] In step S4, the surgical support robot 10 is put into a roll-in position. More specifically, information about the roll-in positions of the positioner 12 and each arm 14 is stored in the memory 162, and the processor 161 controls the positioner driving device 129, the arm driving device 149, etc., based on the information about the roll-in position, so that the positioner 12 and each arm 14 take the roll-in position.

[0082] The roll-in position is the position when the surgical support robot 10 is moved close to the patient P. The roll-in position of the positioner 12 is a position in which the support 11 is positioned close to the patient P, thereby allowing the arm base 13 to be positioned above the patient P. The roll-in position of the arms 14 is a position in which the medical instruments 15 are removed and the arms 14 are folded so that they do not interfere with the patient P when the arm base 13 is positioned above the patient P.

[0083] The roll-in posture may vary depending on the type of surgery, the patient's position, etc. That is, the memory 162 may store information regarding the roll-in posture according to the type of surgery, the patient's position, etc. For example, when the user U inputs to the interface device 114 the area to be operated on, such as the abdomen, and the position at which the surgery support robot 10 is to be placed relative to the patient P, such as the right side of the patient, the first controller 16 controls the positioner 12, etc., so that the surgery support robot 10 assumes the roll-in posture according to the input information.

[0084] After the posture of the surgical support robot 10 is shifted to the roll-in posture, as shown in FIG. 10 , the user U operates the handle 113 to move the surgical support robot 10 so that the arm base 13 is positioned above the operating table 31. In a first example of the surgical procedure, an image of the downward direction is captured by a camera 139 disposed on the arm base 13, and the captured image of the camera 139 is displayed on the touch panel display of the interface device 114. The user U moves the surgical support robot 10 so that a predetermined position of the captured image displayed on the interface device 114 is aligned with a predetermined trocar T.

[0085] After the roll-in is completed, the position of the surgical support robot 10 is fixed relative to the floor surface. That is, the support body 11 is fixed to the floor surface. Therefore, the roll-in determines the position of the base end of the positioner 12 relative to the operating table 31. After the roll-in is completed, the user U may fine-tune the position and orientation of the arm base 13.

[0086] After the roll-in is completed, the processor 161 of the first controller 16 shifts the posture of the multiple arms 14 to a ready posture (step S4). That is, the multiple arms 14 are unfolded from the roll-in posture in which the multiple arms 14 are folded to a predetermined ready posture so that the next task can be easily performed.

[0087] In this example, information indicating, for example, the preparatory posture is stored in advance in the memory 162. Specifically, after completing the roll-in, the user U inputs, for example, information about the surgical site such as the abdomen, and information about the position of the surgical support robot 10 relative to the patient P, such as the right side of the patient, via the interface device 114. Based on the information input by the user U, the processor 161 selects one piece of posture information from the plurality of pieces of posture information stored in advance in the memory 162, and determines the posture indicated by the selected posture information as the preparatory posture of the plurality of arms 14. Note that the method of determining the preparatory posture is not particularly limited.

[0088] Once the ready posture is determined, the processor 161 controls the operation of the multiple arms 14 so that the posture of the multiple arms 14 transitions to the determined ready posture. For example, only while the user U is performing a movement permission operation on an input device such as the interface device 114 or the arm-side controller 17, the processor 161 operates the arm 14 so that it transitions from the current posture, the roll-in posture, to the ready posture. For example, only while the user U is pressing a predetermined manual operation button included in the arm-side controller 17, the processor 161 operates the arm 14, and stops the arm 14 when the user U releases the manual operation button.

[0089] After the arm 14 reaches the ready position, the pivot position is taught and the medical instrument is attached (step S5).

[0090] Fig. 11 is a diagram for explaining teaching of the pivot position PP. Fig. 12 is a diagram for explaining the movement of the medical instrument 15 when inserted into the trocar T.

[0091] The pivot position PP is a position that serves as a fulcrum when moving the medical instrument 15 during surgery. As shown in Fig. 12, by teaching the pivot position PP, the processor 161 controls the operation of the arm 14 so that the medical instrument 15 rotates and moves around the pivot position PP as a fulcrum. In other words, by teaching the pivot position PP, it is possible to limit the effect of the movement of the arm 14 on the body wall of the patient P.

[0092] In step S5, a pivot position teacher 200 that teaches the pivot position PP is attached to the tip of the arm 14. The pivot position teacher 200 has a shaft 201. While pressing a predetermined manual operation button included in the arm-side controller 17, the user U manually moves the arm 14 until the tip 202 of the shaft 201 is positioned at the port position, that is, the insertion position of the trocar T inserted on the body surface of the patient P, as shown in FIG.

[0093] With the tip 202 of the pivot position teacher 200 positioned at the insertion position of the trocar T on the body surface of the patient P, the user U presses a predetermined teaching button included in the arm-side controller 17, whereby the pivot position PP is taught and stored in the memory 162. After teaching the pivot position PP, the user U detaches the pivot position teacher 200 from the arm 14. The user U inserts the medical instrument 15 into the trocar T, and then attaches the medical instrument 15, which is inserted into the trocar T, to the arm 14. This teaching operation and attachment of the medical instrument 15 are performed for each arm 14. In other words, the pivot position PP is set individually for each arm 14.

[0094] Note that some of the medical instruments 15 may have a function as the pivot position teacher 200. That is, instead of positioning the tip 202 of the pivot position teacher 200 at the insertion position of the trocar T on the body surface of the patient P, a predetermined part of the medical instrument 15, such as the tip of the endoscopic camera 151, may be positioned at the insertion position of the trocar T on the body surface of the patient P.

[0095] When the attachment of the arm 14 to the medical instrument 15 inserted in the trocar T is completed in this way, the preparation stage ends and the procedure moves to the operator assistance stage.

[0096] (Surgeon assistance stage) In the surgeon assistance stage, the processor 231 of the second controller 23 executes an identification process for identifying the surgical target in the acquired endoscopic image, and stores the identification information, which is information related to the identification of the surgical target, in the memory 232 (step S6). In the identification process of this embodiment, the position of the surgical target in the endoscopic image is identified. That is, the processor 231 of the second controller 23 identifies which part of the image captured as the endoscopic image is the surgical target. The identification information may include information related to the position of the surgical target in the endoscopic image, as well as the condition of the surgical target, the shape of the surgical target, the size of the surgical target, or a combination thereof.

[0097] Specifically, image information showing an endoscopic image captured by the endoscopic camera 151 inside the body cavity of the patient P is sent to the second controller 23 of the remote control device 20 via the first controller 16. The processor 231 of the second controller 23 identifies the surgical target in the endoscopic image based on the received image information. An example of a method for identifying the surgical target in the endoscopic image is described below.

[0098] (Method for identifying surgical subjects in images 1) Identification method 1 is a method of identifying a surgical target in an endoscopic image by image recognition processing. Processor 231 identifies a surgical target in an endoscopic image by image recognition processing based on image information and surgical content information, and stores identification information including position information of the surgical target on the endoscopic image in memory 232.

[0099] The surgical content information relates to the content of a surgery performed using the surgical instrument 152. For example, the surgical content information includes information indicating the name of a disease, information indicating a surgical procedure, or a combination thereof. The processor 231 searches the entire endoscopic image for a location that matches a feature that has been registered in advance in association with the surgical content information. If the processor 231 detects a matching location, it identifies the matching location as the surgical target in the endoscopic image.

[0100] The processor 231 may perform image recognition processing using AI (Artificial Intelligence). The processor 231 may input an endoscopic image into a trained model generated using a known machine learning algorithm such as a neural network, and output a result of identifying the surgical target in the endoscopic image. For example, a trained model may be prepared for each surgical procedure, for example, for each disease name, and the processor 231 may use a trained model corresponding to the surgical procedure (for example, the disease name). The trained model corresponding to the disease name is generated, for example, by machine learning the relationship between the endoscopic image of the surgery corresponding to the disease name and the range of the lesion in the endoscopic image marked by the endoscopist, as training data.

[0101] (Method for identifying surgical subjects in images 2) Identification method 2 is a method for identifying the surgical target in the endoscopic image based on the judgment and operation of the surgeon U1 viewing the endoscopic image. A mode selection button for accepting selection of an identification mode for identifying the surgical target is displayed on the setting screen displayed on the touch panel 213. When the surgeon U1 operates the touch panel 213 to select the identification mode, the processor 231 causes the endoscopic image received from the endoscopic camera 151 to be displayed on the touch panel 213.

[0102] When the surgeon U1 determines that there is a surgical target in the endoscopic image, the surgeon U1 specifies the area determined to be the surgical target on the endoscopic image by operating the touch panel 213. For example, the surgeon U1 may specify the position of the surgical target on the endoscopic image by using a finger or a stylus pen to surround or mark the area determined to be the surgical target on the endoscopic image displayed on the touch panel 213. In this way, the processor 231 may identify the area specified by the user U on the endoscopic image as the surgical target, and store identification information including position information of the identified surgical target on the endoscopic image in the memory 232.

[0103] Furthermore, the area of ​​the surgical target does not have to be designated on the touch panel 213. The surgeon U1 may designate the surgical target on the endoscopic image by operating the operation input device 21 while viewing the endoscopic image on the display device 22.

[0104] The above-described identification methods 1 and 2 are merely examples, and other methods may be used to identify the surgical target. Furthermore, both identification methods 1 and 2 may be used. That is, if the processor 231 is unable to identify the surgical target in the endoscopic image using the image recognition processing of identification method 1, the processor 231 may present information indicating this to the surgeon U1 on the display 22 or touch panel 213. The surgeon U1 may manipulate the orientation of the endoscopic camera 151 to search for an orientation in which the surgical target can be identified using the image recognition processing. As a result, if the surgeon U1 is able to determine the position of the surgical target in the endoscopic image but the surgical target is not identified using the image recognition processing, the surgeon U1 may perform an operation to identify the position of the surgical target himself, as in identification method 2.

[0105] 13 shows a screen D1 displayed on the display 22 of the remote control device 20 after the identification process. After identifying the surgical target, the processor 231 causes the display 22 to display a surgical target image K1 indicating the identified surgical target superimposed on the endoscopic image.

[0106] 13 is an image having a predetermined transparency so that the surgical object can be visually recognized even when the surgical object is superimposed on the endoscopic image. The surgical object image K1 has the same contour as the identified surgical object.

[0107] The display mode of the surgical target image K1 shown in the figure is merely an example. That is, the surgical target image K1 may be displayed in a manner that allows the user U to easily grasp the position of the surgical target in the endoscopic image. The display mode may vary depending on the type of surgical target. The outline of the surgical target image K1 may be a square, circle, ellipse, or other shape that encompasses most or all of the identified surgical target. The surgical target image K1 may also be a frame. This is to make it easier to visually identify the surgical target enclosed by the frame in the surgical target image K1. In this case, the frame may be the same shape as the identified surgical target, or may be a square, circle, ellipse, or other shape that encompasses most or all of the identified surgical target. The surgical target image does not have to be an image that represents an area. If the surgical target represents a line to be incised with a scalpel or a point through which a needle such as a suture needle will be inserted, the surgical target image may be a line image or a point image.

[0108] (Camera adjustment control) After the surgical target is identified, the processor 231 determines whether or not the camera adjustment control is permitted (step S7). If it is determined that the camera adjustment control is permitted (step S7: Yes), the camera adjustment control is executed (step S8). If it is determined that the camera adjustment control is not permitted (step S7: No), step S8 is skipped.

[0109] The camera adjustment control is a control for adjusting the endoscopic camera 151 based on the identified surgical subject. The adjustment of the endoscopic camera 151 includes adjustment of the imaging range, which is the field of view of the endoscopic camera 151, the viewpoint position, the line of sight direction, or a combination thereof. For example, in the camera adjustment control, the operation of the camera arm 14 is controlled based on the identified surgical subject, i.e., based on the identification information, so that the surgical subject is positioned at a predetermined position in the endoscopic image.

[0110] In this embodiment, the determination of whether or not the camera adjustment control is permitted in step S7 is made based on whether or not the surgeon U1 has input a permission command. Fig. 14 shows an example of a confirmation screen D2 for confirming with the surgeon U1 whether or not to permit the camera adjustment control. After the surgical target is identified, the processor 231 outputs a confirmation notice, such as that shown in a message window W1 in Fig. 14, to the display device 22 or touch panel 213 of the remote operation device 20, for example, to confirm with the surgeon U1 whether or not to permit the camera adjustment control.

[0111] When the surgeon U1 inputs a permission command to the operation input device 21 to permit camera adjustment control, the processor 231 of the second controller 23 sends a permission command including identification information to the first controller 16. The processor 161 of the first controller 16 controls the operation of the camera arm 14 based on the received permission command, i.e., based on the identified surgical subject, so that the identified surgical subject is located at a predetermined position in the endoscopic image.

[0112] 15 shows the display screen D3 of the remote control device 20 after the camera adjustment control has been executed. The processor 161 controls the operation of the camera arm 14 so that the surgical target is located in the center of the endoscopic image, as shown in FIG.

[0113] When the surgeon U1 inputs a disallowance command to the operation input device 21 to not allow camera adjustment control, the processor 231 erases the confirmation notice from the display screen. Thereafter, the surgeon U1 may operate the operation input device 21 while looking at the screen of the display device 22 to move the camera arm 14 so that the specified surgical target is located in the center of the endoscopic image.

[0114] The processor 231 may execute the camera adjustment control not only immediately before the start of surgery but also as appropriate during surgery. The processor 231 may also execute the camera adjustment control constantly during a predetermined operation step during surgery.

[0115] The camera adjustment control is not limited to controlling the operation of the camera arm 14 so that the surgical target is positioned at a predetermined position in the endoscopic image. The camera adjustment control may be any control that adjusts the imaging range, viewpoint position, line of sight direction, or a combination thereof of the endoscopic camera 151 in accordance with the surgical target identified by the identification process.

[0116] For example, the camera adjustment control may include, instead of or in addition to controlling the operation of the camera arm 14 described above, control for zooming so that the size of the identified surgical target is within a predetermined range relative to the entire endoscopic image, or may include focus control for adjusting the focal position to the identified surgical target.

[0117] For example, instead of or in addition to controlling the operation of the camera arm 14 described above, the camera adjustment control may control the operation of the camera arm 14 so that the surgeon U1 can easily view the identified surgical target based on the positional relationship between the surgical target and the surgical instrument 152. For example, the camera adjustment control may include control to operate the camera arm 14 so as to reduce the area where the tool 152c overlaps with the surgical target. In this case, the processor 231 may perform image recognition processing, determine the positional relationship between the surgical target and the surgical instrument 152, and control the operation of the camera arm 14 based on the determination result. The relative position between the surgical support robot 10 and the surgical target, which will be described in a second embodiment later, can also be used to determine the positional relationship between the surgical target and the surgical instrument 152.

[0118] (Instrument adjustment control) Next, the processor 231 determines whether or not the appliance adjustment control is permitted (step S9). If it is determined that the appliance adjustment control is permitted (step S9: Yes), the appliance adjustment control is executed (step S10). If it is determined that the appliance adjustment control is not permitted (step S9: No), step S10 is skipped.

[0119] The instrument adjustment control is a control for adjusting the position, posture, movement, or a combination thereof, of the surgical instrument 152 based on the identified surgical object. In the instrument adjustment control, the movement of the instrument arm 14 is controlled based on the identified surgical object, i.e., based on the identification information, so that at least a portion of the surgical instrument 152 is included in the endoscopic image.

[0120] In this embodiment, whether or not the instrument adjustment control in step S9 is permitted is determined based on whether or not a permission command has been input by the surgeon U1. For example, after the camera adjustment control or after a command not to permit the camera adjustment control has been input, the processor 231 outputs a confirmation notice to the remote operation device 20, for example, on the display device 22 or the touch panel 213, to ask the surgeon U1 whether or not to permit the instrument adjustment control.

[0121] When the surgeon U1 inputs a permission command to the operation input device 21 to permit instrument adjustment control, the processor 231 of the second controller 23 sends the permission command including identification information to the first controller 16. The processor 161 of the first controller 16 controls the operation of the instrument arm 14 based on the received permission command, i.e., based on the identified surgical target, so that at least a portion of the surgical instrument 152 is included in the endoscopic image.

[0122] Figure 16 shows the display screen D4 of the remote control device 20 after the instrument adjustment control has been executed. For example, the processor 161 controls the operation of the instrument arm 14 so that the tool 152c of the surgical instrument 152 is displayed at the edge of the screen, as shown in Figure 16. For example, the processor 161 may recognize the surgical instrument 152 in the image by image recognition processing, and control the operation of the instrument arm 14 so that the surgical instrument 152 does not overlap with the surgical target in the endoscopic image.

[0123] It should be noted that the processor 161 does not have to control the movement of all of the multiple (three in this example) instrument arms 14 based on the identified surgical target, but may control the movement of all of one or two of the three instrument arms 14. The instrument arms 14 controlled by the instrument adjustment control may be the two instrument arms 14 corresponding to the operation manipulators 211L and 211R, respectively.

[0124] When the surgeon U1 inputs a disallowance command to the operation input device 21 to not allow the instrument adjustment control, the processor 231 erases the confirmation notice from the display screen. Thereafter, the surgeon U1 may operate the operation input device 21 while looking at the screen of the display device 22 to move the instrument arm 14 so that the instrument arm 14 is included in the endoscopic image.

[0125] (Recommended action estimation process) After the adjustment of the positions of the endoscopic camera 151 and the surgical instrument 152 is completed, the surgical procedure is carried out in accordance with the operation of the surgeon U1 on the remote control device 20. While the surgical procedure is being carried out, the processor 231 executes a process of estimating a recommended procedure (step S11).

[0126] The process of estimating a recommended operation is a process of guiding a user U, particularly a surgeon U1, on an operation for surgery. The process of estimating a recommended operation is a process of estimating a recommended operation that is recommended as an operation to be received by the operation input device 21. In other words, the process of estimating a recommended operation is a process of estimating a recommended action, which is an action of the arm 14 or the surgical instrument 152 corresponding to the recommended operation. Note that estimating a recommended operation is substantially the same as estimating a recommended action, and therefore, in the following description, estimating a recommended operation may also mean estimating a recommended action. Recommended operation information indicating the content of a recommended operation and recommended action information indicating a recommended action corresponding to a recommended operation may be the same information.

[0127] In the process of estimating a recommended operation of this embodiment, the processor 231 estimates a recommended operation based on the surgical target identified in the identification process, i.e., based on the identification information, and displays recommended operation information indicating the content of the estimated recommended operation on the display 22. The memory 232 stores an estimation program for estimating a recommended operation or a recommended action, and the processor 161 estimates the recommended operation by executing the estimation program.

[0128] The estimation program is a program for estimating a recommended operation or recommended action based on the identification information. The estimation program can be created by referring to the operations performed on the identified surgical target, for example, using endoscopic images and operation history from past surgeries. For example, by executing the estimation program, recommended operation information indicating the direction in which the surgical instrument 152 should be moved can be generated from the position of the surgical target included in the identification information.

[0129] For example, the estimation program may be a trained model for receiving input information including identifying information and outputting recommended operation information. The trained model is generated using a known machine learning algorithm such as a neural network. For example, a large number of data sets may be prepared, each containing identifying information about a surgical target identified in a past surgery and operation information indicating the operation performed when the surgical target was identified, and these data sets may be used as training data (i.e., training data). The input information may include endoscopic image information in addition to the identifying information. In this case, the identifying information may include position information indicating the position of the surgical target on the endoscopic image, as well as status information indicating the status of the surgical target.

[0130] 17 shows a display screen D5 for explaining an example of a method for presenting recommended operation information. For example, assume that the current situation is one in which a certain surgical instrument 152 needs to be brought close to and brought into contact with a surgical target or a portion thereof to perform a certain operation. In this case, the processor 231 executes the estimation program to estimate the surgical instrument 152 that should be brought close to the surgical target, the direction in which the surgical instrument 152 should be moved, and the target position of the surgical instrument 152. Based on these estimation results, the processor 231 causes the display 22 to display, as recommended operation information, an image G1 in the shape of an arrow pointing from the surgical instrument 152 to be moved toward the surgical target, superimposed on the endoscopic image, as shown in FIG. 17.

[0131] After the recommended operation information is presented on the display 22, the surgeon U1 operates the operation input device 21, causing the arm 14 or the surgical instrument 152 to perform the recommended operation corresponding to the recommended operation. Thereafter, the processor 231 determines that the presented recommended operation has been completed, estimates the next recommended operation, and presents it on the display 22. In this way, the processor 231 sequentially estimates the recommended operation each time a recommended operation and its corresponding recommended operation are completed, and displays the recommended operation information on the display 22.

[0132] (Automatic execution of recommended actions) After the recommended operation information is presented to the surgeon U1, the surgeon U1 does not have to perform the recommended operation on the operation input device 21. For example, the processor 231 may determine whether or not to permit the arm 14 or the medical instrument 15 to automatically execute the recommended operation, in accordance with a user's input to the operation input device 21. In this case, the processor 231 may display, on the display 22 or the touch panel 213 of the remote operation device 20, a confirmation screen for confirming with the surgeon U1 whether or not the arm 14 or the medical instrument 15 will automatically execute the recommended operation.

[0133] When the surgeon U1 inputs a permission command to the operation input device 21 to permit automatic execution of the recommended action, the processor 231 of the second controller 23 sends the permission command including the recommended action information to the first controller 16. The processor 161 of the first controller 16 may control the arm 14 or the medical instrument 15 to execute the recommended action based on the received permission command, i.e., based on the recommended action information.

[0134] (Alert function) Furthermore, the result of the process of estimating the recommended operation does not have to be presented to the user U. The recommended operation information may be used in the computer to confirm whether the operation of the surgeon U1 is proceeding according to the recommended procedure. For example, after the process of estimating the recommended operation, the processor 231 may determine whether or not operation information indicating the content of the operation received by the operation input device 21 (hereinafter, determination target information) matches the recommended operation information acquired in the estimation process. When the processor 231 determines that the determination target information does not match the recommended operation information, it may output warning information.

[0135] The warning information may be output in any manner as long as it indicates that the operation of the surgeon U1 does not match the recommended operation. The warning information may be image information, audio information, vibration information, force information, or the like.

[0136] For example, if processor 231 determines that the information to be judged does not match the recommended operation information, it may display warning information on display 22, such as text information such as "This is not a recommended operation" or "Moving there is not recommended," or a warning image.

[0137] For example, when the processor 231 determines that the determination target information does not match the recommended operation information, the processor 231 may output a warning sound such as a buzzer or a warning message voice from the speaker 24 as warning information.

[0138] For example, when the processor 231 determines that the judgment target information does not match the recommended operation information, it may control the servo motors MM1, ..., MMn included in the operating manipulator 211L or 211R to notify the operator U1 that the operation of the operator U1 does not match the recommended operation via the operating manipulator 211L or 211R. In this case, the processor 231 may apply vibration to the operating manipulator 211L or 211R, or may generate torque to make it difficult for the operating manipulator 211L or 211R to perform an operation that does not match the recommended operation. Furthermore, when the processor 231 determines that the judgment target information does not match the recommended operation information, it may vibrate equipment that is in contact with the operator U1 other than the operating manipulator 211L or 211R.

[0139] The alert function regarding the output of the warning information described above can be employed in both cases where the recommended operation information is not presented to the surgeon U1 and where the recommended operation information is presented to the surgeon U1.

[0140] (Action and effect) As described above, in this embodiment, the processor 231 of the second controller 23 is configured to identify a surgical target that is a target of surgery on the body of the patient P using the surgical instrument 152. Therefore, by identifying the surgical target, it is possible to make the surgery easier for the surgeon U1 during the surgery.

[0141] In this embodiment, the processor 231 superimposes a surgical target image K1 indicating the surgical target identified by the identification process on the endoscopic image and displays it on the display 22. This makes it easier for the surgeon U1 to grasp the position of the surgical target on the endoscopic image during surgery.

[0142] In this embodiment, camera adjustment control is executed to adjust the endoscopic camera 151 based on the surgical target identified in the identification process, so that the effort required to adjust the endoscopic camera 151 can be reduced.

[0143] In this embodiment, instrument adjustment control is performed to adjust the position, posture, movement, or a combination thereof, of the surgical instrument 152 based on the surgical target identified in the identification process, thereby reducing the effort required to adjust the position of the surgical instrument 152, etc.

[0144] In this embodiment, a recommended operation is estimated as an operation to be accepted by the operation input device 21. Therefore, various processes according to the recommended operation make it easier for the surgeon U1 to proceed with the surgery.

[0145] For example, when the recommended operation information is presented to the surgeon U1, the surgeon U1 can proceed with the surgery while checking that the operation to be performed is correct.

[0146] For example, if the operator U1 allows the arm 14 or the medical instrument 15 to automatically perform the recommended action by allowing the operator U1 to automatically perform the recommended action, the operational burden on the operator U1 can be reduced.

[0147] For example, if the operation received by the operation input device 21 does not match the recommended operation, warning information is output, thereby preventing the operator U1 from performing an erroneous operation.

[0148] Second Embodiment Next, a surgery support system 1B according to a second embodiment will be described. In this embodiment, elements that are the same as or similar to those in the first embodiment are given the same reference numerals in the drawings, and detailed descriptions thereof will be omitted.

[0149] 18 is a diagram illustrating the configuration of a surgery support system 1B according to the second embodiment. In addition to the elements included in the surgery support system 1A according to the first embodiment, the surgery support system 1B further includes an operating table device 30 including an operating table 31, and an environmental information acquisition device 40.

[0150] [Operating table equipment configuration] Fig. 19 is a diagram for explaining the operating table apparatus 30 and the environmental information acquisition device 40, and is a side view of the inside of the operating room R. Note that Fig. 19 shows only the operating table apparatus 30 and the environmental information acquisition device 40, and omits other elements of the surgery support system 1B.

[0151] The operating table apparatus 30 includes an operating table 31 on which the patient P is placed, and a support structure 32 that supports the operating table 31. The operating table 31 may also be referred to as a table body 31. The support structure 32 includes a movement structure that moves the operating table 31 relative to the floor surface. More specifically, the support structure 32 is configured to be able to change the position, orientation, posture, or any combination thereof of the operating table 31. The position, orientation, posture, or any combination thereof of the operating table 31 may also be referred to as a table state.

[0152] Specifically, the support structure 32 includes a fixed element 32a that is fixed to the floor surface and at least one movable element 32b that is connected to the fixed element 32a and is displaceable relative to the fixed element 32a. The operating table 31 is fixed to an upper portion of the movable element 32b. At least one joint is interposed between the fixed element 32a and the movable element 32b. The joint is a prismatic joint or a rotary joint.

[0153] The table state may include the tilt of at least a portion of the operating table 31. For example, the support structure 32 is configured to be able to change the tilt of the operating table 31 relative to the horizontal plane, in other words, the tilt of the support surface of the operating table 31 on which the patient P rests. The support structure 32 may be configured to be able to tilt the support surface only in one predetermined direction, or may be configured to be able to tilt the support surface in multiple directions. For example, the support structure 32 may be configured to be able to move the movable element 32b so that the orientation of the top of the head of the patient P resting on the operating table 31 relative to the horizontal plane and / or the left-right orientation of the body of the patient P resting on the operating table 31 can be changed.

[0154] The operating table 31 may also have multiple bases, each having a support surface. The multiple bases may be connected to each other in a displaceable manner. For example, as illustrated in FIG. 19, the multiple bases 31a, 31b may be capable of changing their orientation relative to each other. For example, the base 31a on which the back of the patient P is placed and at least one second base 31b, separate from the first base 31a, on which the feet of the patient P are placed may be capable of changing their orientation relative to each other. A movable element 32b may be fixed to each of the multiple bases, or some of the multiple bases may be fixed to a fixed element 32a.

[0155] The operating table device 30 includes a drive device 33 (see FIG. 20), an operation input device 34 for receiving input of commands from a user U, and a third controller 35.

[0156] The drive device 33 drives the support structure 32 to displace the operating table 31. The drive device 33 includes at least one actuator. The drive device 33 includes at least one actuator that displaces at least one movable element 32b. The type of actuator is not particularly limited. For example, the actuator may be an electric motor, a hydraulic actuator, or a pneumatic actuator.

[0157] In this embodiment, the operation input device 34 is a touch panel, but may be realized by a known operation tool such as a lever, a button, a joystick, a pedal, or a motion capture device.

[0158] The operating table apparatus 30 includes a detector that detects table status information indicating the table status. In this embodiment, the detector includes at least one position sensor 36 that detects a physical quantity corresponding to the position of at least one movable element 32b. The third controller 35 is communicatively connected to the drive device 33, the operation input device 34, and the position sensor 36 via wired or wireless communication. For example, the position sensor 36 may be a sensor that detects the amount of linear movement of a linear joint, or a rotation angle sensor of a rotary joint, such as an encoder. Examples of sensors that can be used to detect the table status information include a gyro sensor, a tilt sensor, a geomagnetic sensor, an encoder, a potentiometer, a limit switch, a stroke sensor, a vision sensor, a 3D scanner, a laser tracker, a laser scanner, radar, a LiDAR (Light Detection and Ranging), a stereo camera, or any combination thereof.

[0159] [Configuration of environmental information acquisition device] The environmental information acquisition device 40 is a device for acquiring information about the environment within the operating room R. The environmental information acquisition device 40 is equipped with a plurality of object detection sensors 41. In this embodiment, the object detection sensors 41 detect three-dimensional information of objects within the operating room R. The object detection sensors 41 detect, for example, the three-dimensional position and three-dimensional shape of objects within the operating room R. The environmental information acquisition device 40 can also be referred to as a three-dimensional information acquisition device that acquires three-dimensional information within the operating room R.

[0160] In this embodiment, for example, the object detection sensor 41 is used to detect three-dimensional information of a patient P in the operating room R, as will be described later. Also, for example, the object detection sensor 41 is used to detect three-dimensional information of objects (which may include a patient) around the surgical support robot 10 in the operating room R, as will be described later.

[0161] The three-dimensional information of an object may be information indicating the shape and position of the object in the operating room R. For example, the three-dimensional information of a patient P may be information indicating the body shape of the patient P, the position and posture of the patient P in the operating room R.

[0162] The object detection sensor 41 is, for example, a vision sensor. The object detection sensor 41 may also be a distance measuring sensor or measuring device capable of acquiring three-dimensional information. The object detection sensor 41 may also be a three-dimensional scanner, a laser tracker, a laser scanner, a radar, a LiDAR (Light Detection And Ranging), or a stereo camera. The multiple object detection sensors 41 may be of different types.

[0163] The multiple object detection sensors 41 are arranged at intervals within the operating room R. The multiple object detection sensors 41 may be stationary sensors fixed within the operating room, or may be portable sensors that can be carried around within the operating room. For example, a given object detection sensor 41 may be supported on a side wall or ceiling within the operating room R. A given object detection sensor 41 may also be supported by a support structure such as a tripod at a predetermined position on the floor of the operating room R. A given object detection sensor 41 may also be fixed to the surgical support robot 10. Specifically, a given object detection sensor 41 may be fixed to any one of the support body 11, the positioner 12, the arm base 13, and the arm 14.

[0164] The environmental information acquisition device 40 may include a moving device 42. The moving device 42 is configured to change at least one of the position of the object detection sensor 41 within the operating room R and the orientation of the object detection sensor 41 within the operating room R. The moving device 42 enables a single moved object detection sensor 41 to detect objects from multiple positions or multiple directions in the operating room R.

[0165] The moving device 42 includes a moving structure 42a. The moving structure 42a is configured to support the object detection sensor 41 and to move the object detection sensor 41 by being driven by a driving device 42b (see FIG. 20) described below. For example, FIG. 19 shows, as an example of the moving structure 42a, a rotation mechanism that supports the object detection sensor 41 by suspending it from the ceiling of the operating room R. The rotation mechanism is configured to rotate and move the object detection sensor 41 about a certain vertical line C that passes through the ceiling of the operating room R. In the example of FIG. 19, the object detection sensor 41 is disposed at a position on the moving structure 42a that is a predetermined distance away from the vertical line C, but it may also be disposed on the vertical line C.

[0166] However, the configuration of the moving structure 42a is not particularly limited. The moving structure 42a may be a link mechanism having multiple joints, with its base end fixed to the operating room R and the object detection sensor 41 fixed to its tip. That is, the moving device 42 may be a multi-joint robot arm. In this case, the driving device 42b may be at least one servo motor that drives at least one joint of the multi-joint robot arm. That is, the object detection sensor 41 and the moving device 42 that supports it may form an arm-type three-dimensional measuring machine. Furthermore, the moving structure 42a may include wheels that rotate on the floor surface and a support that rotatably supports the wheels. In this case, the driving device 42b may be a motor that drives the wheels to rotate.

[0167] The environmental information acquisition device 40 includes a fourth controller 43 that controls the operation of the driving device 42b (see FIG. 20).

[0168] [Control system configuration of surgical support system] 20 is a block diagram showing an example of the configuration of a control system of a surgery support system 1B according to the second embodiment. Of the configuration of the control system of the surgery support system 1B, the explanation of the surgery support robot 10 and the remote control device 20 overlaps with the first embodiment and will be omitted.

[0169] (Configuration and functions of the third controller) The first controller 16 of the surgery support robot 10 is connected to the third controller 35 of the operating table device 30 so as to be able to communicate with each other.

[0170] The third controller 35 includes at least one processor 351 such as a CPU, and a memory 352. The memory 352 is configured with a storage device such as a semiconductor memory such as a volatile memory or a nonvolatile memory, a hard disk, or an SSD (Solid State Drive). The at least one processor 351 executes a program stored in the memory 352, causing the third controller 35 to perform various processes.

[0171] The third controller 35 controls the overall operation of the operating table apparatus 30. For example, the third controller 35 controls the drive device 33 to change the attitude of the operating table 31. For example, the processor 351 of the third controller 35 controls the drive device 33 based on a command input to the operation input device 34 so that the position and attitude of the operating table 31 correspond to the command. Alternatively, for example, the third controller 35 controls the drive device 33 based on a command received from the first controller 16 so that the position and attitude of the operating table 31 correspond to the command.

[0172] The third controller 35 sends table status information indicating the position, orientation, or posture of the operating table 31 to the first controller 16. Based on the received table status information, the first controller 16 can control the operation of the positioner 12, the operation of the arm 14, and the operation of the medical instrument 15. For example, the first controller 16 can determine the preparatory posture based on the table status information, and control the operation of the positioner 12, the operation of the arm 14, and the operation of the medical instrument 15 so as to assume the determined preparatory posture.

[0173] The position of the operating table 31 indicated by the table status information includes information regarding the posture of the operating table 31. The table status information may include, for example, information indicating the inclination angle of the support surface on which the operating table 31 is placed relative to the horizontal plane. The table status information may also include information indicating the orientation of the operating table 31, such as the orientation of the operating table 31 in the operating room R. In this embodiment, the table status information indicating the position of the operating table 31 is a detection value by the position sensor 36, but the table status information may also be a command value for the drive device 33.

[0174] (Configuration and functions of the 4th controller) The first controller 16 of the surgery support robot 10 is connected to the fourth controller 43 of the environmental information acquisition device 40 so as to be able to communicate with each other.

[0175] The fourth controller 43 includes at least one processor 431 such as a CPU, and a memory 432. The memory 432 is configured with a storage device such as a semiconductor memory such as a volatile memory or a nonvolatile memory, a hard disk, or an SSD (Solid State Drive). The at least one processor 431 executes a program stored in the memory 432, causing the fourth controller 43 to perform various processes.

[0176] The fourth controller 43 sends information detected by the object detection sensors 41 to the first controller 16. The fourth controller 43 may process the information detected by the object detection sensors 41 and send it to the first controller 16. For example, the fourth controller 43 may combine the detection information from the multiple object detection sensors 41 to generate surrounding object information that is information about surrounding objects around the multiple arms 14, and send the surrounding object information to the first controller 16. Alternatively, the surrounding object information may be generated by the first controller 16 that receives the information detected by the object detection sensors 41.

[0177] The first controller 16 can control the operation of the positioner 12, the operation of the arm 14, and the operation of the medical instrument 15 based on the surrounding object information received from the environmental information acquisition device 40. For example, the first controller 16 can determine the ready posture based on the surrounding object information, and control the operation of the positioner 12, the operation of the arm 14, and the operation of the medical instrument 15 so as to assume the determined ready posture.

[0178] The peripheral object information is information relating to the positions and postures of objects around the surgical support robot 10. The fourth controller 43 can generate peripheral object information relating to the positions and postures of objects around the surgical support robot 10 based on the detection information of the object detection sensor 41 and position information indicating the position of the object detection sensor 41.

[0179] When the object detection sensor 41 is located at a fixed position within the operating room R, the position information indicating the position of the object detection sensor 41 is stored in advance in the memory 432 as known information.

[0180] If the position of the object detection sensor 41 can be displaced within the operating room R by the moving device 42, position information indicating the position of the object detection sensor 41 is calculated based on information related to the position and attitude of the moving device 42. Specifically, the moving device 42 includes a driving device 42b and a position sensor 42c that detects information corresponding to the position of the object detection sensor 41. If the driving device 42b is configured to include at least one servo motor that drives at least one joint of the articulated robot arm, the position sensor 42c may be a rotation sensor or the like that is disposed in each joint and detects the rotation angle of each joint.

[0181] Furthermore, if the object detection sensor 41 is movable within the operating room R, for example by being fixed to the arm 14 of the surgical support robot 10, position information indicating the position of the object detection sensor 41 can be obtained by an encoder included in the arm driving device, an encoder included in the positioner driving device, the object detection sensor 41 of the environmental information acquisition device 40, or a combination thereof.

[0182] Each of the first controller 16, the second controller 23, the third controller 35, and the fourth controller 43 may execute each process under centralized control by a single processor, or may execute each process under distributed control through cooperation of multiple processors. Each controller may be configured with at least one or a combination of two or more of a computer, a personal computer, a microcontroller, a microprocessor, a programmable logic device (PLD) such as a field-programmable gate array (FPGA), a programmable logic controller (PLC), and a logic circuit, for example.

[0183] Any combination of two or more of the first controller 16, the second controller 23, the third controller 35, and the fourth controller 43 may be integrated. In other words, the processing of one of the first controller 16, the second controller 23, the third controller 35, and the fourth controller 43 may be performed by another of the first controller 16, the second controller 23, the third controller 35, and the fourth controller 43. For example, the first controller 16 may control the entire operating table apparatus 30 instead of the third controller 35, in which case the third controller 35 may be omitted. For example, the first controller 16 may control the entire environmental information acquisition device 40 instead of the fourth controller 43, in which case the fourth controller 43 may be omitted. One or more processors and memories included in the first controller 16, the second controller 23, the third controller 35, the fourth controller 43, or any combination thereof are examples of processing circuits.

[0184] [Second example of surgical procedure] Fig. 21 is a flow chart showing the flow of a second example of the procedure for performing surgery. The second example of the procedure for preparing for surgery shown in Fig. 21 will be described with reference to Figs. 22 to 27 as appropriate.

[0185] In the second example of the surgical procedure, the position of the surgical target is identified by two methods. One is the identification method described in the first example of the surgical procedure, which identifies the position of the surgical target on an endoscopic image. The other is a method of identifying the position of the surgical target relative to the medical instrument 15 from the relative positions of the surgical support robot 10 and the patient P.

[0186] (Preparation stage) In the preparation stage, three-dimensional model information indicating a three-dimensional model of the patient P is stored in the memory 162 of the first controller 16 (step S21).

[0187] 22 shows a patient model MP, which is a three-dimensional model of patient P. For example, the three-dimensional model information representing the patient model MP may be data obtained by 3D scanning of the patient P. For example, the three-dimensional model information may be data obtained by a computed tomography (CT) examination, in other words, a CT scan, or data obtained by processing the data.

[0188] 22 shows a portion MV (hereinafter, also referred to as a "target portion MV") corresponding to at least a part of a surgical target V inside the body of a patient P. The three-dimensional model information includes position information of the target portion MV in the patient model MP.

[0189] For example, the position of the target location MV in the patient model MP can be specified by a user U via an information processing terminal such as a personal computer. The user U looks at the CT scan data displayed on the display of the information processing terminal and specifies the position of the surgical target V, whereby information indicating the position of the surgical target V in the patient P is stored in the memory of the information processing terminal. The information indicating the position of the surgical target V in the patient P corresponds to the position information of the target location MV in the patient model MP.

[0190] The first controller 16 may receive the position information of the target location MV in the patient model MP from an external device such as a CT device via a communication network, or may receive the information from a portable storage medium by the user U inserting the storage medium into the first controller 16.

[0191] For example, the position of the target location MV in a patient model MP lying on its back can be used to estimate the position of the target location MV in a patient model MP lying in a different position. The position information of the target location MV in the patient model MP acquired in step S21 is used in step S25, which will be described later.

[0192] Next, the user U determines the position of the patient P and the posture of the operating table 31 during the surgery (step S22).

[0193] The user U considers and determines the body position of the patient P and the posture of the operating table 31 according to the content of the surgery. The user U lies the patient P on the operating table 31 so that the body position is as determined. The user U also operates the operation input device 34 of the operating table device 30 so that the body position of the patient P and the posture of the operating table 31 are as determined. The third controller 35 controls the drive device 33 based on the input received by the operation input device 34, and changes the posture of the operating table 31.

[0194] The user U determines the positions of a plurality of ports on the body surface of the patient P, and inserts the trocar T into the determined positions (step S23). Step S23 is the same as step S2, and therefore a description thereof will be omitted.

[0195] Next, the user U moves the surgery support robot 10 to the side of the operating table 31 so that it can support the surgery, that is, performs a so-called roll-in (step S24). Step S24 is the same as step S3, and therefore a description thereof will be omitted.

[0196] After the roll-in is completed, the processor 161 executes a process of identifying the relative position between the surgical support robot 10 and the surgical subject (step S25). Specifically, the processor 161 acquires relative position information indicating the relative position between the patient P and the surgical support robot 10, and identifies the position of the surgical subject relative to the surgical support robot 10 based on the relative position information. A method of identifying the relative position between the surgical support robot 10 and the surgical subject will be described.

[0197] 23 is a diagram illustrating the identification of the position coordinates of the surgical subject. Specifically, after the roll-in is completed, the processor 431 of the fourth controller 43 sends three-dimensional information of objects in the operating room R from the environment information acquisition device 40 to the first controller 16. The processor 161 of the first controller 16 calculates the positional relationship between the surgery support robot 10 and the patient P based on the acquired three-dimensional information. For example, the processor 161 calculates the position and posture of the patient P relative to the surgery support robot 10.

[0198] More specifically, the processor 161 sets a three-dimensional coordinate system defined for the surgical support robot 10. Although Fig. 23 shows an example in which a coordinate system defined for the arm base 13 is defined, the coordinate system is not limited to this. For example, the coordinate system may be set based on the support 11 instead of the arm base 13. The coordinate system may be set for each arm 14. The coordinate system may be set based on the operating room R.

[0199] The positional relationship between the surgical support robot 10 and the patient P does not have to be based on the three-dimensional information sent from the fourth controller 43. For example, when rolling in, the positional relationship between the surgical support robot 10 and the patient P may be set to a predetermined positional relationship by adjusting a certain element of the surgical support robot 10 to be in a predetermined position and orientation relative to the operating table 31.

[0200] The first controller 16 may also use table status information received from the third controller 35 of the operating table device 30 to calculate the position and posture of the patient P relative to the surgical support robot 10. The table status information may be a detection value by the position sensor 36 or a command value for the drive device 33.

[0201] The processor 161 calculates the position coordinates of the surgical target V in the set coordinate system based on the three-dimensional information. More specifically, the processor 161 identifies the position coordinates of the surgical target V using the three-dimensional information acquired by the environment information acquisition device 40 after roll-in and the three-dimensional information of the patient P acquired in advance in step S21.

[0202] That is, since the correspondence between the patient model MP and the target location MV can be known from the three-dimensional model information acquired in advance in step S21, it is also possible to estimate the position of the target location MV relative to the patient model MP when the patient model MP is made to assume the same posture as that of the patient P after roll-in. Therefore, the processor 161 calculates the position coordinates of the affected area V in the set coordinate system from the posture of the patient P acquired by the environment information acquisition device 40 after roll-in and the position of the target location MV relative to the patient model MP when the patient P assumes that posture.

[0203] After obtaining the position coordinates of the surgical target, the processor 161 of the first controller 16 shifts the posture of the multiple arms 14 to a preparatory posture (step S26). After the arms 14 reach the preparatory posture, the pivot position is taught and the medical instrument is attached (step S27). Step S26 is the same as step S4, and step S27 is the same as step S5, so their explanations are omitted.

[0204] (Surgeon assistance stage) In the surgeon support stage, the processor 231 of the second controller 23 executes an identification process to identify the surgical target in the acquired endoscopic image (step S28). The identification process in step S28 is the same as that in step S6, and therefore a description thereof will be omitted.

[0205] (Camera adjustment control) After the surgical target is identified, the processor 231 determines whether or not the camera adjustment control is permitted (step S29). If it is determined that the camera adjustment control is permitted (step S29: Yes), the camera adjustment control is executed (step S30). If it is determined that the camera adjustment control is not permitted (step S29: No), step S30 is skipped.

[0206] Steps S29 and S30 are basically the same as steps S7 and S8, but the camera adjustment control in step S30 differs from the camera adjustment control in step S8 in that it uses the information acquired in step S28 and the information acquired in step S25. In other words, the camera adjustment control executed in step S30 adjusts the endoscopic camera 151 using not only the result of identifying the position of the surgical subject on the endoscopic image, but also the result of identifying the position of the surgical subject on the coordinate system.

[0207] For example, Fig. 24 shows an example of a display screen D6 of the remote control device that is displayed before the camera adjustment control is executed. As shown in Fig. 24, the line of sight of the endoscopic camera 151 may be slightly deviated from the surgical subject, so that the surgical subject is not within the field of view. Even in such a case, the processor 161 can calculate the relative position between the endoscopic camera 151 and the surgical subject from the relative position information between the surgical support robot 10 and the surgical subject acquired in step S25. In other words, the processor 161 can calculate the position of the surgical subject relative to the location imaged by the endoscopic camera 151.

[0208] The processor 161 may send the calculation result of the position of the surgical target relative to the location imaged by the endoscopic camera 151 to the second controller 23. Then, as shown in Fig. 24, the processor 231 of the second controller 23 causes the display 22 to display target direction information W2, which indicates the direction of the surgical target that is outside the field of view relative to the location imaged by the endoscopic camera 151, superimposed on the endoscopic image.

[0209] After the target direction information W2 is displayed on the display 22, the surgeon U1 can operate the operation input device 21 to move the field of view of the endoscopic camera 151 in the direction indicated by the target direction information W2, thereby adjusting the orientation of the endoscopic camera 151 so that the surgical target is included in the endoscopic image.

[0210] Alternatively, after the target direction information W2 is displayed on the display 22, the surgeon U1 may input a permission command to the operation input device 21 to permit control to move the field of view of the endoscopic camera 151 in the direction indicated by the target direction information W2. The processor 231 of the second controller 23 sends the permission command including the target direction information W2 to the first controller 16. Based on the received permission command, the processor 161 of the first controller 16 controls the operation of the camera arm 14 so as to move the field of view of the endoscopic camera 151 in the direction indicated by the target direction information W2.

[0211] In this way, even if the surgical target is not included in the endoscopic image, adjustment of the position and posture of the endoscopic camera 151 is assisted when the surgical target is included in the endoscopic image.

[0212] In step S28, when the endoscopic image includes the surgical target, the processor 231 may use the relative position information between the surgical support robot 10 and the surgical target acquired in step S25 to identify the surgical target on the endoscopic image. This makes it possible to improve the accuracy of identifying the surgical target on the endoscopic image.

[0213] (Instrument adjustment control) Next, the processor 231 determines whether or not the appliance adjustment control is permitted (step S31). If it is determined that the appliance adjustment control is permitted (step S31: Yes), the appliance adjustment control is executed (step S32). If it is determined that the appliance adjustment control is not permitted (step S31: No), step S32 is skipped.

[0214] Steps S31 and S32 are basically the same as steps S9 and S10, but the camera adjustment control in step S32 differs from the instrument adjustment control in step S10 in that it uses the information acquired in step S28 and the information acquired in step S25. In other words, unlike step S10 in the first example, the instrument adjustment control executed in step S32 uses not only the result of identifying the position of the surgical subject on the endoscopic image but also the result of identifying the position of the surgical subject on the coordinate system to adjust the position, posture, movement, or a combination thereof of the surgical instrument 152.

[0215] For example, the processor 161 of the first controller 16 controls the operation of the instrument arm 14 based on the result of identifying the position of the surgical subject on the endoscopic image and the result of identifying the position of the surgical subject on the coordinate system so that at least a part of the surgical instrument 152 is included in the endoscopic image. This allows the instrument arm 14 to be operated while grasping the relationship between the position of the tool 152c in the body cavity and the position of the surgical subject in the three-dimensional coordinate system, thereby preventing the tool 152c from coming into contact with body tissue that it was not intended to come into contact with during the operation of the instrument arm 14.

[0216] After the adjustment of the positions of the endoscopic camera 151 and the surgical instrument 152 is completed, the surgical procedure is carried out in accordance with the operation of the surgeon U1 on the remote control device 20. In this embodiment, too, the process of estimating the recommended operation and related processes described in step S11 of the first embodiment may be executed while the surgical procedure is being carried out.

[0217] In this embodiment, the same effects as in the first embodiment can be obtained.

[0218] In this embodiment, both an identification process for identifying the surgical target in the endoscopic image and an identification process for identifying the relative position between the surgical support robot 10 and the surgical target are executed. In this way, the position of the surgical target is identified from two perspectives, thereby improving the accuracy of identifying the position of the surgical target.

[0219] In this embodiment, a process for identifying the relative position between the surgical support robot 10 and the surgical subject makes it possible to determine the position of the surgical subject relative to the field of view of the endoscopic camera even if the surgical subject is not included in the endoscopic image.

[0220] (Variation) When the surgical instrument is not visible in the endoscopic image, presenting to the user U the direction outside the field of view of the endoscopic camera 151 in which the surgical instrument 152 is located will assist the operation of the user U. The processor 231 may execute an instrument position presenting process that presents to the user U the direction outside the field of view of the endoscopic camera 151 in which the surgical instrument 152 is located.

[0221] For example, the processor 231 may perform image recognition processing on the endoscopic image to determine whether or not the endoscopic image includes the surgical instrument 152. The processor 231 may execute instrument position presentation processing when it is determined that the endoscopic image does not include the surgical instrument 152.

[0222] In the instrument position presentation process, the processor 231 may receive relative position information between the endoscopic camera 151 and the surgical instrument 152 from the first controller 16. The positional relationship between the endoscopic camera 151 and the surgical instrument 152 can be geometrically grasped from the amount of movement, such as the rotation angle at the joint of each arm 14, and dimensional information, such as the length of the link of the arm 14. The processor 231 may calculate the position of the surgical instrument 152 relative to the line of sight of the endoscopic camera 151 based on the relative position information, and may display the calculated instrument position information, which indicates the direction of the surgical instrument 152, on the display 22 by superimposing it on the endoscopic image.

[0223] Fig. 25 is a display screen D7 for explaining an example of a method for presenting tool position information. As shown in Fig. 25, the processor 231 may display, on the display 22, images W3, W4, and W5 indicating the direction of the tool 152c relative to the field of view of the endoscopic camera 151 at the screen edge of the endoscopic image as tool position information.

[0224] Figure 26 shows a display screen D8 illustrating another example of a method for presenting instrument position information. As shown in Figure 26, a screen D7 displayed on the display 22 includes an inner region R1 and an outer region R2 surrounding the inner region R1. The processor 231 displays an endoscopic image in the inner region R1. The processor 231 calculates the position of each surgical instrument 152 relative to the field of view of the endoscopic camera 151 based on relative position information between the endoscopic camera 151 and the surgical instruments 152. The processor 231 displays instrument images B1, B2, and B3 representing each surgical instrument 152 at positions in the inner region R1 or the outer region R2 that correspond to the positions of each surgical instrument 152 relative to the field of view of the endoscopic camera 151.

[0225] The processor 231 may identify multiple surgical targets and display multiple surgical target images corresponding to the multiple surgical targets on the display. In this case, the types of the multiple surgical targets may be different from one another. FIG. 27 shows an example of a display screen D9 displaying multiple surgical target images K1 and K2. The surgical target image K1 is an image showing the extent of a certain lesion as the surgical target, and the surgical target image K2 is an image showing the extent of a resection target including the lesion as the surgical target. The processor 231 may also display text information K3 and K4 on the display explaining the types of surgical targets represented by the multiple surgical target images K1 and K2. The surgical target images superimposed on the endoscopic image may be any image that indicates the surgical target, such as a text image, a leader line image, or an arrow image.

[0226] <Other embodiments> Although examples of embodiments of the present disclosure have been described above, the present disclosure is not limited to the above-described embodiments. That is, various modifications and improvements are possible within the scope of the present disclosure. For example, various modifications made to the embodiments and forms constructed by combining components of different embodiments are also included within the scope of the present disclosure.

[0227] The above embodiments have been described as examples of the technology disclosed in this application. However, the technology of the present disclosure is not limited to these and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made as appropriate. Furthermore, the components described in the above embodiments can be combined to create new embodiments. For example, some configurations or methods in one embodiment may be applied to another embodiment, and some configurations in an embodiment can be separated and arbitrarily extracted from other configurations in that embodiment. Furthermore, the components described in the accompanying drawings and detailed description include not only components essential for solving the problem, but also components that are not essential for solving the problem, and are used to illustrate the technology. Two blocks shown in order in a flowchart may be executed simultaneously or in reverse order, depending on the circumstances.

[0228] The functions of the elements disclosed herein can be performed using circuits or processing circuits, including general-purpose processors, special-purpose processors, integrated circuits, ASICs (Application Specific Integrated Circuits), conventional circuits, or any combination thereof, configured or programmed to perform the disclosed functions. A processor is considered a processing circuit or circuit because it includes transistors and other circuitry. In this disclosure, a circuit, unit, or means is hardware that performs the recited functions or hardware that is programmed to perform the recited functions. The hardware may be hardware disclosed herein or other known hardware that is programmed or configured to perform the recited functions. Where the hardware is a processor, which is considered a type of circuit, the circuit, means, or unit is a combination of hardware and software, and the software is used to configure the hardware or processor.

[0229] The configuration and shape of each element of the surgery assistance system are not limited to those described in the first and second embodiments. For example, the number of arms, the configuration of the base, and the number and arrangement of object detection sensors are not limited to those described in the above embodiments.

[0230] In the first and second embodiments, the surgical assistance system for identifying a surgical target is described, but the surgical assistance system is not limited thereto. The processing circuit may not be configured to identify a surgical target on a patient's body that is a target for surgery using the surgical instrument.

[0231] In the second embodiment, the position of the surgical target is identified by two methods: identifying the position of the surgical target on an endoscopic image and identifying the position of the surgical target relative to the medical instrument 15 in a three-dimensional coordinate system. However, it is also possible to identify only one of these methods. For example, in the second example of the surgical procedure, step S28 may be omitted.

[0232] The locations where each process, such as the identification process, camera adjustment control, instrument adjustment control, and recommended operation estimation process, is executed are not particularly limited, and may be executed by any circuit included in the surgery support system. For example, the identification of the position of the surgical target in the endoscopic image may be executed by the processor 161 on the above-mentioned surgery support robot 10, instead of by the processor 231 on the remote control device 20. In this case, identification information indicating the identified position of the surgical target on the endoscopic image may be sent to the remote control device 20 together with the endoscopic image.

[0233] The surgery assistance system 1A of the first embodiment may include one or both of an operating table device and an environmental information acquisition device. The surgery assistance system 1B of the second embodiment may not include one or both of an operating table device and an environmental information acquisition device.

[0234] The flowcharts shown in the above embodiments are examples of surgical procedures and do not limit the present disclosure. For example, after roll-in and before attaching the medical instrument to the arm, there may be no need to teach each arm. That is, in the above first and second embodiments, the movement of the arm 14 is restricted by teaching the pivot position PP, but the movement of the arm 14 may be restricted by other means, such as mechanically connecting the arm or the medical instrument 15 to a trocar.

[0235] In the first and second embodiments, multiple processes have been described as processes using the results of the identification of the surgical target by the identification process, such as superimposing the surgical target on the endoscopic image, camera adjustment control, instrument adjustment control, estimation of recommended operations, and presentation of the position of the surgical target relative to the endoscopic image. However, the processing circuitry may not have the function to execute some of these processes. For example, the image of the surgical target may not be displayed on the endoscopic image.

[0236] The arm, the positioner, or a combination thereof may be an example of a robotic arm.

[0237] The rotation sensor is not limited to an encoder, and may be any sensor that can detect the amount of rotation of a servo motor or the amount of rotation of a joint.

[0238] Although the support of the surgical support robot is configured as a cart that can move within the operating room, the support is not limited to this. The support of the surgical support robot may be configured to move within the operating room by being controlled by a first controller rather than being steered by the user U. For example, when the position of the surgical support robot 10 is predetermined in the simulation stage as in the second embodiment, the position information may be stored in memory, and movement control of the support of the surgical support robot may be performed based on the position information. Furthermore, the support of the surgical support robot may be fixed in the operating room so as to be immovable.

[0239] Although the operating table apparatus 30 includes a drive device that displaces the movable element 32b relative to the fixed element 32a, the operating table apparatus need not include a drive device. For example, the operating table apparatus may be configured so that the orientation of the operating table can be changed by manually displacing the movable element 32b relative to the fixed element 32a. The orientation of the operating table may not change in the orientation of the support surface relative to the horizontal plane. In other words, the operating table apparatus 30 may be configured without the movable element 32b.

[0240] Examples of table status information indicating the position, orientation, or posture of the operating table 31 include detection values ​​by the position sensor 36 and command values ​​to the drive device 33, but the table status information indicating the position of the operating table 31 may also be information detected by the object detection sensor 41.

[0241] The camera adjustment control does not have to be executed based on a permission command input by the surgeon U1, and may be executed automatically regardless of permission from the surgeon U1. For example, the processor 231 may determine whether the identified surgical target is at a predetermined position in the endoscopic image, and if it is determined that the surgical target is not at the predetermined position (for example, if it is determined that the surgical target is not located at the center of the image), it may permit and execute the camera adjustment control.

[0242] Alternatively, camera adjustment control may not be performed. For example, the surgeon U1 may operate the operation input device 21 to adjust the position of the camera arm 14 so that the surgical subject is positioned at a predetermined position in the endoscopic image. In this case, the processor 231 may output information indicating the operation of the camera arm 14 to position the surgical subject at a predetermined position in the endoscopic image as recommended operation information indicating the content of the recommended operation estimated in the recommended operation estimation process. For example, after identifying the surgical subject in step S6, a message recommending camera adjustment, such as "Adjust so that the surgical subject is positioned at the center of the camera field of view," may be output from the display 22, the speaker 24, or the like as recommended operation information. Based on the identified surgical subject, the processor 231 may determine whether the identified surgical subject is located at a predetermined position in the endoscopic image. If it is determined that the surgical subject is not located at the predetermined position, the processor 231 may output a message recommending camera adjustment as recommended operation information from the display 22, the speaker 24, or the like.

[0243] 17 shows an image of an arrow pointing from the surgical instrument 152 to be moved toward the surgical target as the recommended operation information, but the recommended operation information is not limited to this. For example, the recommended operation information may be text information indicating the content of the recommended operation, such as "Pass the needle here" or "Pass the needle here," as shown in FIG. 28. The recommended operation information does not have to be image information, but may also be audio information, and the processor 231 may output the recommended operation information from the speaker 24 included in the remote operation device 20.

[0244] The recommended operation information may be recommended instrument information for recommending the next surgical instrument 152 to be moved, recommended direction information for recommending the direction in which to move the surgical instrument 152, recommended instrument position information for recommending the target position or target posture of the surgical instrument 152, recommended contact position information for recommending the location at which to contact the tool 152c on the surgical target area or its surrounding area, or a combination thereof. For example, the location indicated by the recommended contact position information may be a location recommended for an incision, a location recommended for a suture, etc.

[0245] In the process of estimating a recommended operation described in the above embodiment, the recommended operation is estimated based on the surgical target identified in the identification process, but the recommended operation does not have to be estimated based on the surgical target. For example, the process of estimating a recommended operation may be one that estimates, from the operation received by the operation input device 21 and the details of the surgery, a recommended operation that is to be recommended as an operation to be received by the operation input device 21 after receiving the operation.

[0246] The process of estimating the recommended operation may be performed using a trained model generated using a known machine learning algorithm such as a neural network. For example, the memory 232 may store a trained model that has undergone machine learning to estimate a recommended operation that is recommended as an operation to be accepted by the operation input device 21 in a certain surgical situation. The processing circuit may acquire surgical situation information indicating the situation of the surgery, and input input information including the surgical situation information to the trained model, thereby acquiring recommended operation information indicating the content of the recommended operation from the trained model.

[0247] The situation indicated by the surgical situation information may be a situation that changes from moment to moment during surgery. For example, the surgical situation information may be information indicating the progress of the surgery. For example, the surgical situation information may include operation information indicating the content of the operation accepted by the operation input device 21. That is, the situation of the surgery indicated by the surgical situation information may be a situation that has arisen due to the operation accepted by the operation input device 21. That is, the operation accepted by the operation input device 21 includes the operation immediately before accepted by the operation input device 21. Furthermore, the operation accepted by the operation input device 21 may include not only the operation immediately before accepted by the operation input device 21, but also all or part of the operation history accepted by the operation input device 21 from the start of the surgery to the present.

[0248] That is, the memory 232 may store a learned model that has been subjected to machine learning to estimate a recommended operation that is recommended as an operation to be accepted by the operation input device 21 after the operation input device 21 accepts an operation. The processing circuit may acquire operation information indicating the content of the operation accepted by the operation input device 21 of the remote operation device 20, and input input information including the operation information to the learning model, thereby acquiring recommended operation information indicating the content of the recommended operation from the learning model.

[0249] For example, a trained model may be prepared for each surgical procedure, for example, for each disease. In this case, processor 231 may use a trained model corresponding to the surgical procedure (for example, the disease). A trained model corresponding to a disease is generated, for example, by preparing a large number of time-series data sets of operation information indicating the operations performed on operation input device 21 from the start to the end of surgery for that disease, and performing machine learning using as training data the relationship between first operation information regarding one operation in the time-series data and second operation information regarding the operation following the operation indicated by the first operation information in the time-series data. Alternatively, a trained model may be generated by performing machine learning using as training data the relationship between first operation information regarding all operations from the start of surgery to a certain point in time in the time-series data and second operation information regarding the operation following the operation indicated by the first operation information in the time-series data.

[0250] The training data, which is data for learning, is created by dividing time-series data of operation information indicating the details of operations performed on the operation input device 21 from the start to the end of surgery into multiple pieces of data each having a certain time range. Of the divided multiple pieces of data, one or more pieces of data can be used as first operation information, and one piece of data following the first operation information can be used as second operation information for learning.

[0251] For example, the time-series data of the operation information may be divided at predetermined times, and among the divided data, for example, two pieces of data that are adjacent in time may be set as the first operation information and the second operation information.

[0252] Furthermore, for example, the time-series data of operation information may be divided by the content of the operation. The content of the operation may be the object to be moved, the content of the movement, the object of work, or a combination thereof. In this case, among the divided data, for example, two pieces of data that are one after the other in terms of time may be set as the first operation information and the second operation information. In this case, the first operation information and the second operation information may differ from each other in the object to be moved, the content of the movement, the object of work, etc.

[0253] The input information may include condition information determined before the surgery, in addition to surgical situation information that may change in real time during the surgery. In other words, the trained model may be a model trained by machine learning to output recommended operation information when input information including condition information and surgical situation information is input. The condition information is stored in advance in memory 162, for example, before the surgery is performed.

[0254] The condition information may include at least one selected from the group consisting of surgery content information relating to the content of the surgery, instrument information indicating the type of surgical instrument used in the surgery, and information relating to the patient undergoing the surgery.

[0255] For example, if the condition information is surgical content information relating to the content of surgery, the trained model is generated by, for example, preparing a large number of time series data of operation information indicating the content of operations performed on the operation input device 21 from the start to the end of various types of surgery, and performing machine learning using as training data the relationship between the surgical content information relating to the content of the surgery from which the time series data was obtained, first operation information relating to one operation in the time series data, and second operation information relating to the operation following the operation indicated by the first operation information in the time series data.

[0256] The surgical situation information may include endoscopic image information captured by the endoscopic camera 151. That is, the situation of the surgery indicated by the surgical situation information may include the situation of the endoscopic image, for example, the situation of the surgical subject included in the endoscopic image, the situation of the surgical instrument 152, etc.

[0257] For example, the input information may include not only operation information but also endoscopic image information obtained when an operation corresponding to the operation information is performed. In other words, the trained model may be a model trained by machine learning to output recommended operation information when input information including an endoscopic image and operation information is input. In this case, the trained model is generated by, for example, preparing a large amount of time-series data of operation information indicating the details of operations performed on the operation input device 21 from the start to the end of the surgery and information on endoscopic images from the start to the end of the surgery, and conducting machine learning using as training data the relationship between first operation information related to one operation in the time-series data, endoscopic image information obtained when an operation corresponding to the first operation information is performed, and second operation information related to the operation following the operation indicated by the first operation information in the time-series data. When the input information includes image information, the input information may or may not include surgical content information.

[0258] After obtaining the recommended operation information, the processing circuit may cause the recommended operation information to be displayed on the display of the remote control device 20.

[0259] After obtaining the recommended operation information, the processing circuitry may determine whether to allow the recommended operation of the arm or medical instrument, which is an operation corresponding to the recommended operation, to be automatically executed. If the processing circuitry determines that the recommended operation is to be automatically executed, the processing circuitry may control the operation of the robotic arm or medical instrument to execute the recommended operation.

[0260] The processing circuit may determine whether or not determination target information, which is operation information indicating the content of the operation to be accepted by the operation input device after acquiring the recommended operation information, matches the acquired recommended operation information, and output warning information when it determines that the determination target information does not match the recommended operation information.

[0261] [Disclosure mode] Each of the following aspects is a disclosure of a preferred embodiment.

[0262] [Aspect 1] An endoscopic camera, Surgical instruments and a plurality of robot arms including a camera arm having a plurality of degrees of freedom and having the endoscopic camera attached to a tip end thereof, and an instrument arm having a plurality of degrees of freedom and having the surgical instrument attached to a tip end thereof; a display that displays an endoscopic image captured by the endoscopic camera; an operation input device that accepts user operations; A surgical assistance system comprising: The processing circuitry Identifying a surgical target in a patient's body that is a target for surgery using the surgical instrument; and and operating the plurality of robot arms in response to the user's operation on the operation input device.

[0263] [Aspect 2] the processing circuitry is configured to acquire image information indicative of an endoscopic image captured by the endoscopic camera within the patient's body; The surgical support system of aspect 1, wherein the processing circuit identifying the surgical target includes the processing circuit identifying the position of the surgical target in the endoscopic image based on the image information.

[0264] [Aspect 3] the processing circuitry is configured to execute acquiring surgical content information relating to the content of a surgical procedure performed using the surgical instrument; The surgical support system of aspect 2, wherein the processing circuit identifying the surgical target includes the processing circuit identifying the position of the surgical target in the endoscopic image based on the image information and the surgical content information.

[0265] [Aspect 4] The processing circuitry configured to perform obtaining relative position information indicative of a relative position between the patient and the robotic arm; A surgical assistance system described in any one of aspects 1 to 3, wherein the processing circuit identifying the surgical target includes the processing circuit identifying the position of the surgical target relative to the robot arm based on the relative position information.

[0266] [Aspect 5] A surgical assistance system according to any one of aspects 1 to 4, wherein the processing circuit is configured to identify an area specified by a user as the surgical target.

[0267] [Aspect 6] A surgical assistance system described in any of aspects 1 to 5, wherein the processing circuit is configured to control the movement of the robot arm based on the identified surgical target.

[0268] [Aspect 7] A surgical support system described in any of aspects 1 to 6, wherein the processing circuit is configured to control the operation of one or both of the camera arm and the endoscopic camera so that the surgical subject is located at a predetermined position in the endoscopic image based on the identified surgical subject.

[0269] [Aspect 8] A surgical assistance system described in any of aspects 1 to 7, wherein the processing circuit is configured to control the movement of one or both of the instrument arm and the surgical instrument based on the identified surgical target so that at least a portion of the surgical instrument is included in the endoscopic image.

[0270] [Aspect 9] The processing circuitry estimating a recommended operation that is recommended as an operation to be accepted by the operation input device based on the identified surgical target; displaying the recommended operation information indicating the estimated content of the recommended operation on the display; A surgical assistance system according to any one of aspects 1 to 8, configured to execute the following:

[0271] [Aspect 10] The processing circuitry determining whether to permit the robot arm to automatically execute a recommended action that corresponds to the recommended operation; If it is determined that the robot arm is permitted to automatically execute the recommended action, controlling the operation of the robot arm to execute the recommended action; A surgical assistance system as described in aspect 9, configured to perform the following.

[0272] [Aspect 11] A surgical support system described in any one of aspects 1 to 10, wherein the processing circuit is configured to superimpose a surgical target image showing the surgical target on the endoscopic image and display the superimposed image on the display.

[0273] [Aspect 12] A surgical assistance method for assisting surgery using a surgical assistance system including a robot arm having multiple degrees of freedom and a surgical instrument attached to a distal end thereof, and a processing circuit, comprising: acquiring, by the processing circuitry, image information indicative of an endoscopic image captured inside the patient's body; identifying, by the processing circuitry, a surgical target in the patient's body in the endoscopic image based on the image information, which is a target for surgery using the surgical instrument; A surgical assistance method comprising:

[0274] [Aspect 13] a plurality of robot arms each having a plurality of degrees of freedom; a plurality of medical instruments attached to the distal ends of the plurality of robot arms, respectively; an operation input device that accepts user operations; a processing circuit that operates the plurality of robot arms in response to an operation of the user on the operation input device, The processing circuitry acquiring surgical status information indicating the status of a surgery using the surgical instrument; inputting input information including the surgical situation information into a trained model that has undergone machine learning to estimate a recommended operation that is recommended as an operation to be accepted by the operation input device in the surgical situation indicated by the surgical situation information, thereby acquiring recommended operation information indicating the content of the recommended operation from the trained model; A surgical assistance system configured to perform the above steps.

[0275] [Aspect 14] the plurality of medical instruments includes an endoscopic camera; A surgical support system as described in aspect 13, wherein the surgical status information includes at least one selected from the group consisting of operation information indicating the content of the operation accepted by the operation input device and endoscopic image information captured by the endoscopic camera.

[0276] [Aspect 15] the plurality of medical instruments includes at least one surgical instrument; the input information includes condition information determined before the surgery is performed, A surgical support system as described in aspect 13 or 14, wherein the condition information includes at least one selected from the group consisting of surgical content information regarding the content of the surgery, instrument information indicating the type of surgical instrument used in the surgery, and information regarding the patient undergoing the surgery.

[0277] [Aspect 16] A surgical assistance system described in any one of aspects 13 to 15, wherein the processing circuit is configured to execute the following: displaying the recommended operation information on a display.

[0278] [Aspect 17] The processing circuitry determining whether to permit automatic execution of a recommended operation of the robot arm or the medical instrument, which is an operation corresponding to the recommended operation; If it is determined that the recommended action is permitted to be automatically executed, controlling the operation of the robot arm or the medical instrument to execute the recommended action; 17. A surgical assistance system according to any one of aspects 13 to 16, configured to execute the following:

[0279] [Aspect 18] The processing circuitry determining whether or not determination target information, which is operation information indicating the content of an operation to be accepted by the operation input device after the recommended operation information is acquired, matches the acquired recommended operation information; 18. A surgical support system according to any one of aspects 13 to 17, configured to output warning information if it is determined that the information to be judged does not match the recommended operation information. [Explanation of symbols]

[0280] 1A, 1B: Surgical support system 10: Surgical support robot 11:Support 12: Positioner 13: Arm base 14: Arm 15: Medical equipment 151: Endoscope camera 152 :Surgical instruments 16: First controller 161: Processor 162: Memory 20: Remote control device 21: Operation input device 22: Display device 23: Second controller 231: Processor 232: Memory 30: Operating table equipment 31:Operating table 40:Environmental information acquisition device 41: Object detection sensor P:Patient PP: Pivot position R:Operating room T: Trocar

Claims

1. An endoscopic camera, Surgical instruments and a plurality of robot arms including a camera arm having a plurality of degrees of freedom and having the endoscopic camera attached to a tip end thereof, and an instrument arm having a plurality of degrees of freedom and having the surgical instrument attached to a tip end thereof; a display that displays an endoscopic image captured by the endoscopic camera; an operation input device that accepts user operations; A surgical assistance system comprising: The processing circuitry Identifying a surgical target in a patient's body that is a target for surgery using the surgical instrument; and and operating the plurality of robot arms in response to the user's operation on the operation input device.

2. the processing circuitry is configured to acquire image information indicative of an endoscopic image captured by the endoscopic camera within the patient's body; The surgery support system according to claim 1 , wherein the processing circuitry identifying the surgical target includes the processing circuitry identifying the position of the surgical target in the endoscopic image based on the image information.

3. the processing circuitry is configured to execute acquiring surgical content information relating to the content of a surgical procedure performed using the surgical instrument; The surgical support system of claim 2, wherein the processing circuitry identifying the surgical target includes the processing circuitry identifying the position of the surgical target in the endoscopic image based on the image information and the surgical content information.

4. The processing circuitry configured to perform obtaining relative position information indicative of a relative position between the patient and the robotic arm; The surgical assistance system according to claim 1 or 2, wherein the processing circuit's identifying the surgical target includes the processing circuit's identifying the position of the surgical target relative to the robot arm based on the relative position information.

5. The surgery assistance system according to claim 1 or 2, wherein the processing circuitry is configured to identify a region designated by a user as the surgical target.

6. The surgical assistance system according to claim 1 or 2, wherein the processing circuit is configured to control the movement of the robot arm based on the identified surgical target.

7. 3. The surgical support system according to claim 1, wherein the processing circuit is configured to control the operation of one or both of the camera arm and the endoscopic camera based on the identified surgical target so that the surgical target is located at a predetermined position in the endoscopic image.

8. 3. The surgical assistance system according to claim 1, wherein the processing circuitry is configured to control the operation of one or both of the instrument arm and the surgical instrument based on the identified surgical target so that at least a portion of the surgical instrument is included in the endoscopic image.

9. The processing circuitry estimating a recommended operation that is recommended as an operation to be accepted by the operation input device based on the identified surgical target; displaying the recommended operation information indicating the estimated content of the recommended operation on the display; The surgical assistance system according to claim 1 or 2, configured to execute the following:

10. The processing circuitry determining whether to permit the robot arm to automatically execute a recommended action that corresponds to the recommended operation; If it is determined that the robot arm is permitted to automatically execute the recommended action, controlling the operation of the robot arm to execute the recommended action; The surgical assistance system according to claim 9 , configured to execute the following:

11. 3. The surgery support system according to claim 1, wherein the processing circuitry is configured to execute the following: superimposing a surgical target image showing the surgical target on the endoscopic image and displaying the superimposed image on the display.

12. A surgical assistance method for assisting surgery using a surgical assistance system including a robot arm having multiple degrees of freedom and a surgical instrument attached to a distal end thereof, and a processing circuit, comprising: acquiring, by the processing circuitry, image information indicative of an endoscopic image captured inside the patient's body; identifying, by the processing circuitry, a surgical target in the patient's body in the endoscopic image based on the image information, which is a target for surgery using the surgical instrument; A surgical assistance method comprising:

13. a plurality of robot arms each having a plurality of degrees of freedom; a plurality of medical instruments attached to the distal ends of the plurality of robot arms, respectively; an operation input device that accepts user operations; a processing circuit that operates the plurality of robot arms in response to an operation of the user on the operation input device, The processing circuitry acquiring surgical status information indicating the status of a surgery using the surgical instrument; inputting input information including the surgical situation information into a trained model that has undergone machine learning to estimate a recommended operation that is recommended as an operation to be accepted by the operation input device in the surgical situation indicated by the surgical situation information, thereby acquiring recommended operation information indicating the content of the recommended operation from the trained model; A surgical assistance system configured to perform the above steps.

14. the plurality of medical instruments includes an endoscopic camera; The surgical support system according to claim 13, wherein the surgical status information includes at least one selected from the group consisting of operation information indicating the content of the operation accepted by the operation input device and endoscopic image information captured by the endoscopic camera.

15. the plurality of medical instruments includes at least one surgical instrument; the input information includes condition information determined before the surgery is performed, 15. The surgical support system according to claim 13, wherein the condition information includes at least one selected from the group consisting of surgical content information relating to the content of the surgery, instrument information indicating the type of surgical instrument used in the surgery, and information about the patient undergoing the surgery.

16. The surgery assistance system according to claim 13 or 14, wherein the processing circuitry is configured to execute the following: displaying the recommended operation information on a display.

17. The processing circuitry determining whether to permit automatic execution of a recommended operation of the robot arm or the medical instrument, which is an operation corresponding to the recommended operation; If it is determined that the recommended action is permitted to be automatically executed, controlling the operation of the robot arm or the medical instrument to execute the recommended action; The surgical assistance system according to claim 13 or 14, configured to execute the following:

18. The processing circuitry determining whether or not determination target information, which is operation information indicating the content of an operation to be accepted by the operation input device after the recommended operation information is acquired, matches the acquired recommended operation information; The surgery assistance system according to claim 13 or 14, configured to output warning information when it is determined that the determination target information does not match the recommended operation information.

Images