Assistance system and assistance method

Abstract

To assist an operator who operates a surgery assistance robot.SOLUTION: An assistance system according to one aspect comprises: an operation manipulator which receives an operation of an operator and has a plurality of joints; and a processing circuit configured to control, in response to an operation received by the operation manipulator, an operation for at least one medical instrument included in a surgery assistance robot or an operation for at least one virtual medical instrument included in a virtual surgery assistance robot in a virtual space that simulates the surgery assistance robot. The processing circuit is configured to execute: acquiring operation information indicating a content of the operation received by the operation manipulator, and evaluating an operation skill of the operator indicated by the operation information on the basis of a prescribed operation evaluation rule.SELECTED DRAWING: Figure 12

Description

[Technical Field]

[0001] The present disclosure relates to an assistance system and an assistance method. [Background technology]

[0002] 2. Description of the Related Art A system is known in which a surgical support robot is operated by a surgeon to perform a medical procedure.

[0003] Patent Document 1 discloses a system for generating a treatment plan for performing a procedure with a robot-assisted manipulator. Specifically, the system disclosed in Patent Document 1 generates a treatment plan for performing a procedure with a robot-assisted manipulator based on a first plurality of treatment inputs, generates a performance index from the execution of the treatment, evaluates the performed treatment based on the performance index to generate treatment evaluation information, stores the treatment evaluation information, and generates a second treatment plan based on the stored treatment evaluation information and a second plurality of treatment inputs. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Special Publication No. 2023-551531 Summary of the Invention [Problem to be solved by the invention]

[0005] Improved systems and methods for assisting operators in operating surgical robots are desirable.

[0006] Therefore, an object of the present disclosure is to provide an assistance system and an assistance method for assisting an operator who operates a surgical assistance robot. [Means for solving the problem]

[0007] An assistance system according to one aspect of the present disclosure includes an operating manipulator having multiple joints that receives operations from an operator, and a processing circuit configured to control the operation of at least one medical instrument possessed by a surgical assistance robot, or the operation of at least one virtual medical instrument possessed by a virtual surgical assistance robot in a virtual space that simulates the surgical assistance robot, in accordance with the operations received by the operating manipulator. The processing circuit is configured to obtain operation information indicating the content of the operation received by the operating manipulator, and evaluate the operating skills of the operator corresponding to the operation information based on predetermined operation evaluation rules.

[0008] An assistance system according to another aspect of the present disclosure comprises a surgical assistance robot including an operating manipulator having a plurality of joints that receives operation from an operator, a plurality of medical instruments, a plurality of arms having a plurality of degrees of freedom and having the medical instruments attached to their tips, and a processing circuit configured to control the movement of the plurality of arms or the plurality of medical instruments in accordance with operation from the operator received by the operating manipulator, wherein the processing circuit is configured to evaluate, based on a predetermined interference evaluation rule, the possibility of interference between the arm or the medical instrument that is operated due to operation by the operator and a predetermined interference determination target, determine whether the evaluation result regarding the possibility of interference satisfies a predetermined condition, and, if it is determined that the predetermined condition is satisfied, present to the operator information indicating that the possibility of interference has exceeded an acceptable range.

[0009] An assistance method according to one aspect of the present disclosure is a method for assisting an operator in improving their operation skills with an operation manipulator for operating a medical instrument possessed by a surgical assistance robot or a virtual medical instrument possessed by a virtual surgical assistance robot in a virtual space simulating the surgical assistance robot, and includes controlling, by a processing circuit, the operation of the medical instrument or the operation of the virtual medical instrument in accordance with an operation of the operator received by the operation manipulator, acquiring, by the processing circuit, operation information indicating the content of the operation of the operator received by the operation manipulator, and evaluating, by the processing circuit, the operation skill of the operator corresponding to the operation information based on predetermined operation evaluation rules. [Effects of the Invention]

[0010] According to the present disclosure, it is possible to provide an assistance system and an assistance method for assisting an operator who operates a surgical assistance robot. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram for explaining the configuration of a surgery assistance system. [Figure 2] FIG. 2 is a diagram illustrating an example of the configuration of the surgery support robot of FIG. 1. [Figure 3] 3 is an enlarged perspective view of an arm base, an arm, and a medical instrument in the surgery support robot of FIG. 2. FIG. [Figure 4] FIG. 1 is an enlarged perspective view of an example tool of a surgical instrument. [Figure 5] FIG. 1 is a perspective view of an endoscopic camera. [Figure 6] FIG. [Figure 7] FIG. 2 is a perspective view of a pair of operating manipulators. [Figure 8] FIG. [Figure 9] 10 is a diagram showing a state in which an operator's finger is inserted into a finger insertion portion of the operating handle. FIG. [Figure 10]1 is a diagram for explaining the configuration of an operation training system which is an assistance system according to a first embodiment. FIG. [Figure 11] 11 is a block diagram showing an example of the configuration of a control system of the operation training system of FIG. 10. FIG. [Figure 12] 10 is a flowchart illustrating the flow of control of virtual surgery by the operation training system. [Figure 13] 10 is an example of a setting screen for virtual surgery. [Figure 14] 10 is an example of a setting screen for virtual surgery. [Figure 15] 10 is an example of a setting screen for virtual surgery. [Figure 16] 10 is an example of a display screen of a virtual endoscopic image. [Figure 17] 10 is an example of a display screen of the evaluation results of the virtual surgery. [Figure 18] 10 is an example of a display screen of the evaluation results of the virtual surgery. [Figure 19] FIG. 10 is a diagram illustrating an operating table device and an environmental information acquisition device of a surgery assistance system, which is an assistance system according to a second embodiment. [Figure 20] FIG. 10 is a block diagram showing an example of the configuration of a control system of a surgery support system, which is a support system according to a second embodiment. [Figure 21] 10 is a flowchart illustrating a control flow in surgery by the surgery assistance system. DETAILED DESCRIPTION OF THE INVENTION

[0012] In the first and second embodiments described below, an assistance system and an assistance method that help assist an operator in operating a surgical support robot are described. The assistance system according to the first embodiment is an operation training system for improving the operation skills of an operator who operates a surgical support robot. The assistance system according to the second embodiment is the surgical support system itself, and has a function of evaluating the operator's operation of the surgical support robot and assisting the operator according to the evaluation results.

[0013] Before describing the first and second embodiments, an example of the configuration of a surgery assistance system will be described.

[0014] [Configuration of surgical support system] FIG. 1 is a diagram for explaining the configuration of a surgery support system 1, showing various elements included in the surgery support system 1 arranged in an operating room R as viewed from above. As shown in FIG. 1, the surgery support system 1 includes a surgery support robot 10, a remote control device 20, and an operating table 31. The surgery support system 1 is a system in which an operator U1, such as a surgeon, operates the remote control device 20 to perform a surgical operation, such as an endoscopic surgery, on a patient P using the surgery support robot 10, as in robot-assisted surgery and robotic telesurgery.

[0015] 1 also shows a display device 61, an instrument table 62, and the like that may be included in the surgery assistance system 1. In addition to the operator U1 who operates the remote control device 20, Fig. 1 also shows a surgery assistant U2.

[0016] The surgery support system 1 is a system that uses a master-slave surgery support robot 10. A remote control device 20 constitutes the master machine, and the surgery support robot 10 constitutes the slave machine. The remote control device 20 is placed away from the surgery support robot 10, and the surgery support robot 10 is remotely controlled by the remote control device 20 during surgery. In the surgery support system 1, an operator U1 operates the remote control device 20 to input commands, and the surgery support robot 10 performs operations corresponding to the commands.

[0017] As shown in Figure 1, during surgery, the surgical support robot 10 is placed beside an operating table 31 on which a patient P lies in an operating room R. In Figure 1, each component included in the surgical support system 1 is placed in the operating room R, but some of the components of the surgical support system 1, such as the remote control device 20, may be placed outside the operating room R or in a remote location.

[0018] 1 includes an operating table device 30 including an operating table 31, and an environmental information acquisition device 40. The operating table device 30 and the environmental information acquisition device 40 will be described in a second embodiment below.

[0019] [Configuration of surgical assist robot] FIG. 2 illustrates an example of the configuration of a surgical support robot 10. The surgical support robot 10 constitutes an interface between the surgical support system 1 and the patient P. The surgical support robot 10 includes a support body 11, a positioner 12, an arm base 13, a plurality of arms 14, a plurality of medical instruments 15, and a controller 16. The surgical support robot 10 shown in FIG. 2 includes four arms 14, and a medical instrument 15 is attached to each arm 14.

[0020] The support 11, positioner 12, arm base 13, arm 14, and medical instrument 15 are connected in this order in a continuous manner from the support 11 to the medical instrument 15. In this specification and claims, the end facing the support 11 will be referred to as the "base end," and the end facing the opposite direction will be referred to as the "distal end."

[0021] (Support) The support body 11 supports the positioner 12. The support body 11 is configured to be freely movable. In this embodiment, the support body 11 is configured as a cart that can move within the operating room R. Specifically, the support body 11 includes a main body 111, a plurality of wheels 112 that support the main body 111, and a handle 113 that is connected to the main body 111. Note that, in FIG. 2, for simplicity of the drawing, only one wheel 112 is shown. The user U can grip the handle 113 to steer the support body 11 and move the surgical support robot 10 to any position. Note that, during surgery, the position of the support body 11 in the operating room R is fixed.

[0022] An interface device 114 is disposed near the handle 113 on the support 11. The interface device 114 is configured to receive operations from the user U, mainly for preparing for surgery before surgery. Specifically, the interface device 114 is configured to receive operations for moving or changing the posture of the positioner 12, the arm base 13, the multiple arms 14, or any combination thereof. The interface device 114 may include, as an input interface, a lever, a button, a touch panel, a joystick, or any combination thereof. The interface device 114 may include, as an output interface, a display.

[0023] For example, an image captured by a camera 139 included in the arm base 13 is displayed on a display included in the interface device 114. The camera 139 faces downward and captures an image of at least one of the operating table 31 and the patient P placed on the operating table 31. The input interface included in the interface device 114 accepts operations such as changing the magnification of the image captured by the camera 139 and displayed on the display. The user U can use the input interface to adjust the position of the surgical support robot 10 relative to the operating table 31 or the patient P while viewing the image displayed on the display included in the interface device 114.

[0024] (positioner) As shown in FIG. 2 , the positioner 12 extends from the support 11 and connects the support 11 to the arm base 13. The positioner 12 has a link structure. The positioner 12 is configured as a multi-axis joint robot arm having multiple degrees of freedom. In this embodiment, the positioner 12 is configured as a vertical multi-joint robot arm having seven degrees of freedom. The positioner 12 can freely change the position and orientation of the arm base 13 relative to the support 11 in three-dimensional space. The configuration of the positioner 12 is not particularly limited as long as it is configured to support the arm base 13, and may be, for example, a linear motion device, an elevating device, a fixing device, or the like. The fixing device may be a bracket or the like that fixes the arm base 13 to a ceiling, a wall, or the like.

[0025] The positioner 12 includes a base 120, a plurality of positioner links, and a mechanical interface 127. The base 120 is fixed to an upper portion of the main body 111. The plurality of positioner links are sequentially connected and extend from the base 120. The plurality of positioner links include a first link 121 rotatably connected to the base 120 via a first joint JP1, a second link 122 swingably connected to the first link 121 via a second joint JP2, a third link 123 swingably connected to the second link 122 via a third joint JP3, a fourth link 124 swingably connected to the third link 123 via a fourth joint JP4, a fifth link 125 swingably connected to the fourth link 124 via a fifth joint JP5, and a sixth link 126 swingably connected to the fifth link 125 via a sixth joint JP6. The mechanical interface 127 is rotatably connected to the sixth link 126 via a seventh joint JP7. The arm base 13 is connected to the mechanical interface 127. The positioner 12 is controlled by the controller 16.

[0026] (Arm base) Figure 3 is an enlarged perspective view of the arm base 13, the arm 14, and the medical instrument 15 of the surgical support robot 10 of Figure 2. Since each of the four arms 14 shown in Figure 2 has substantially the same structure, Figure 3 shows only one of the four arms 14 connected to the arm base 13, and omits the other three arms 14.

[0027] The arm base 13 functions as a "hub" that serves as a base for the multiple arms 14. The positioner 12 and the arm base 13 form a manipulator support that movably supports the multiple arms 14.

[0028] The arm base 13 includes a main body 131, a positioner coupling part 132 arranged on the upper part of the main body 131, and a plurality of manipulator coupling parts 133 arranged on the lower part of the main body 131. The positioner coupling part 132 is coupled to the mechanical interface 128 of the positioner 12. The main body 131 has a certain longitudinal direction and is arch-shaped with the longitudinal direction being the extension direction of the chord. The plurality of manipulator coupling parts 133 are distributed and arranged in the longitudinal direction of the main body 131. In this embodiment, four manipulator coupling parts 133 are arranged. The base end part of the arm 14 is detachably coupled to each manipulator coupling part 133.

[0029] (arm) The arm 14 may also be referred to as a patient-side manipulator 14. The multiple arms 14 are detachably attached to and supported by the arm base 13. In this embodiment, as shown in FIG. 3, four arms 14A, ..., 14D are arranged side by side in the longitudinal direction of the arm base 13. Hereinafter, when the four arms 14A, ..., 14D are referred to without distinction, they may be referred to as "arms 14."

[0030] Each arm 14 has a link structure. Each of the multiple arms 14 is configured as a robot arm. In this embodiment, each of the multiple arms 14 is configured as a vertical articulated robot arm. The position and orientation of the tip of each of the multiple arms 14 can be freely moved in three-dimensional space relative to the arm base 13.

[0031] The arm 14 includes a base 140 and a plurality of arm links connected in sequence and extending from the base 140. The base 140 has an interface 140a configured to be attachable to the manipulator connection portion 133. The plurality of arm links include a first link 141 rotatably connected to the base 140 via a first joint JA1, a second link 142 swingably connected to the first link 141 via a second joint JA2, a third link 143 swingably connected to the second link 142 via a third joint JA3, a fourth link 144 swingably connected to the third link 143 via a fourth joint JA4, a fifth link 145 swingably connected to the fourth link 144 via a fifth joint JA5, a sixth link 146 swingably connected to the fifth link 145 via a sixth joint JA6, and a seventh link 147 swingably connected to the sixth link 146 via a seventh joint JA7.

[0032] The seventh link 147 is connected to the tip link 148 via an eighth joint JA8. The eighth joint JA8 is a linear joint that linearly movesably connects the tip link 148 to the seventh link 147. Specifically, the seventh link 147 has a certain longitudinal direction, and the tip link 148 is connected to the seventh link 147 so as to be movable in the longitudinal direction of the seventh link 147 relative to the seventh link 147.

[0033] The distal link 148 has an interface 148a configured to allow attachment of the medical instrument 15. By attaching the medical instrument 15 to the interface 148a, the medical instrument 15 is held at the distal end of the arm 14.

[0034] In the arm 14 described above, there are seven joints between the base 140 and the seventh link 147. That is, the arm 14 has redundant degrees of freedom for setting the seventh link 147 to a desired position and posture in three-dimensional space.

[0035] An arm-side controller 17 is disposed on each arm 14. In this embodiment, the arm-side controller 17 is fixed to the tip link 148 of each arm 14. The arm-side controller 17 includes a plurality of operation buttons 171 and a joystick 172.

[0036] (Medical equipment) The multiple medical instruments 15 are used in surgery on patient P. A medical instrument 15 is held at the tip of each of the multiple arms 14. The multiple medical instruments 15 include an endoscopic camera 151 and multiple surgical instruments 152. Specifically, the endoscopic camera 151 is held at the tip of one of the multiple arms 14, and a surgical instrument 152 is held at the tip of each of the other three arms 14. Note that, hereinafter, the arm 14 of the four arms 14 to which the endoscopic camera 151 is attached may be referred to as the "camera arm 14," and the other three arms 14 to which the surgical instruments 152 are attached may be referred to as the "instrument arms 14."

[0037] 3 shows a surgical tool 152 at the tip of the arm 14. The surgical tool 152 may also be called an instrument. The surgical tool 152 is a tool used to perform a surgical procedure on a surgical target inside the body of a patient P.

[0038] In this specification and the claims, the term "surgical target" may refer to an affected area within the body of patient P, such as a lesion, or to a certain area within the body of patient P near the affected area that includes the affected area, or to a specific location within the body of patient P at or near the affected area. For example, the surgical target is a location that is the target of a surgical operation by the surgical instrument 152. For example, the surgical target may refer to an organ itself that includes the affected area, such as the small intestine or the stomach. For example, the surgical target may refer to a portion of the body of patient P on which a predetermined type of surgical operation is performed. The surgical target may also refer to only the portion of the body of patient P that comes into contact with the surgical instrument 152 (more specifically, the tool 152c described below) or a member grasped by the surgical instrument 152 (e.g., a suture needle) in order to perform a surgical operation. For example, in the case of performing surgery to remove a portion of the body of patient P, the surgical target may refer to the portion to be removed, or to the portion to be incised with a scalpel or the like. The surgical target may refer to a portion of bodily tissue on which a predetermined type of surgical operation is performed, or to the space within the body near the bodily tissue.

[0039] Each surgical instrument 152 has a base 152a configured to be removably attached to the interface 148a of the arm 14, a shaft 152b extending from the base 152a, and a tool 152c located at the tip of the shaft 152b.

[0040] There are multiple types of surgical instruments 152, each differing from the others in at least one of shape, size, and function. For example, the tool 152c of the surgical instrument 152 performs a surgical task according to its type. Examples of the types of surgical tasks include mechanical tasks such as grasping, blunt dissection, sharp dissection, ligation, suturing, and incision of the body tissue of the patient P, including blood vessels, as well as incision or coagulation of the body tissue using high-frequency current. Each surgical instrument 152 can function as forceps, a grasper, scissors, a stapler, a needle holder, or an electric scalpel.

[0041] FIG. 4 is an enlarged perspective view of an example of a tool 152c of the surgical instrument 152. The tool 152c shown in FIG. 4 functions as a forceps and includes a first link 152c1 rotatably connected to the shaft 152b via a first joint JT1, a second link 152c2 swingably connected to the first link 152c1 via a second joint JT2, and a pair of jaws 152c3 and 152c4 connected to the second link 152c2. The pair of jaws 152c3 and 152c4 are arranged to be able to open and close. That is, the base ends of the jaws 152c3 and 152c4 are connected to the second link 152c2 via a third joint JT3 and a fourth joint JT4 so as to be rotatable about the same line or lines parallel to each other.

[0042] The opening and closing movements and changes in orientation of the pair of jaws 152c3, 152c4 are achieved by driving them with the instrument driving device 153. Note that the tool 152c shown in Fig. 4 is an example, and the number of joints, shape, and the like are not limited to this. For example, depending on the type of surgical instrument 152, the tool 152c may not have any joints.

[0043] 5 is a perspective view of the endoscopic camera 151. For example, the endoscopic camera 151 includes an objective lens and a light guide. An endoscopic image captured by the endoscopic camera 151 is sent to the controller 16 and then sent to a controller 23 on the remote control device 20 (described later). However, the endoscopic image captured by the endoscopic camera 151 may also be sent to the controller 23 on the remote control device 20 without going through the controller 16.

[0044] Each arm 14 and each medical instrument 15 is controlled by a controller 16 .

[0045] [Configuration of remote control device] The configuration of the remote control device 20 will be described. Fig. 6 is a perspective view of the remote control device 20. The remote control device 20 constitutes an interface between the surgery support system 1 and the operator U1, and is used to receive input from the operator U1 and remotely operate the surgery support robot 10.

[0046] The remote control device 20 includes an operation input device 21 for receiving command input from the operator U1, a display device 22 for displaying endoscopic images captured by the endoscopic camera 151, a controller 23, and a speaker 24. The operation input device 21 includes a pair of left and right operation manipulators 211L and 211R, a plurality of operation pedals 212, and a touch panel 213. The operation manipulators 211L and 211R are devices used to manually operate the surgical support robot 10, particularly the arms 14 and medical instruments 15. The operation manipulators 211L and 211R receive operations to change the position and posture of the endoscopic camera 151 and the surgical instruments 152. For example, the operation manipulators 211L and 211R receive operations to operate the surgical instruments 152 of the corresponding arms 14. The operation input device 21, particularly the operation manipulators 211L and 211R, may also be referred to as operation input tools.

[0047] Next, the configuration of the operating manipulators 211L and 211R will be described in detail with reference to FIGS.

[0048] As shown in Fig. 7, the operating manipulators 211L and 211R are operated by the left and right hands of the operator U1, respectively. The operating manipulators 211L and 211R have a link structure with multiple joints. The operating manipulators 211L and 211R have the same configuration except that the operating handles 21b have a symmetrical structure.

[0049] Each operating manipulator 211 includes an operating arm 71 and an operating handle 72. The operating arm 71 has a link 71a, a link 71b, and a link 71c. The upper end of link 71a is attached to the main body of the remote control device 2 so as to be rotatable around the A1 axis along the vertical direction. The upper end of link 71b is attached to the lower end of link 71a so as to be rotatable around the A2 axis along the horizontal direction. One end of link 71c is attached to the lower end of link 71b so as to be rotatable around the A3 axis along the horizontal direction.

[0050] The operating arm 71 supports the operating handle 72 so that it can move within a predetermined three-dimensional operating range. Specifically, the operating arm 71 supports the operating handle 72 so that it can move up and down, left and right, and front and back. The arm 14 moves three-dimensionally in response to the three-dimensional operation of the operating arm 71.

[0051] The operating arm 71 of the right-handed operating manipulator 211R further includes a link 71d, a link 71e, and a link 71f. One end of link 71d is attached to the other end of link 71c so as to be rotatable around the A4 axis. One end of link 71e is received on the other end of link 71d so as to be rotatable around the A5 axis. One end of link 71f is attached to the other end of link 71e so as to be rotatable around the A6 axis.

[0052] As shown in FIG. 8, the operating handle 72 of the right-handed operating manipulator 211R has a support member 72a, a first lever 72b, and a second lever 72c. The support member 72a has a cylindrical shape extending in the Y direction and is connected to the proximal end of the operating arm 71. The distal end of the support member 72a is connected to the proximal end of the operating arm 71 so as to rotate around the A7 axis that is parallel to the Y direction, which is the longitudinal direction of the support member 72a. That is, the distal end of the support member 72a is connected to a link 71f so as to rotate around the A7 axis. The distal end of the support member 72a and the proximal end of the operating arm 71 are connected via a flange portion 73.

[0053] The first lever 72b has an elongated plate-like shape and is attached to the support member 72a so as to be rotatable relative to the support member 72a. The first lever 72b is attached to the support member 72a so as to be rotatable about an A8 axis perpendicular to the A7 axis. The first lever 72b is disposed opposite the second lever 72c via the support member 72a.

[0054] The second lever 72c has an elongated plate-like shape and is attached to the support member 72a so as to be rotatable relative to the support member 72a. The second lever 72c is attached to the support member 72a so as to be rotatable about an A9 axis perpendicular to the A7 axis. The second lever 72c is disposed to face the first lever 72b via the support member 72a.

[0055] Furthermore, a cylindrical finger insertion portion 72d and a rectangular finger pad 72e are provided on the first lever 72b and the second lever 72c. The operator inserts a finger into the finger insertion portion 72d and places the finger on the finger pad 72e to operate the operating handle 72. For example, as shown in Fig. 9, the operator's thumb O1 is inserted into the finger insertion portion 72d on the first lever 72b side, and the operator's middle finger O2 is inserted into the finger insertion portion 72d on the second lever 72c side.

[0056] 9, the operating handle 72 has clutch switches 72f and 72g attached to the first lever 72b and the second lever 72c, respectively. The clutch switches 72f and 72g perform a clutch function that does not transmit the operation of the operator to the arm 14. The clutch switches 72f and 72g have the same function.

[0057] The operation handle 72 also has multiple customization switches 72h. The multiple customization switches 72h are assigned functions to customize the functions to be executed. For example, functions related to the surgery on patient P, such as incision, suction, and water drainage, are assigned to the multiple customization switches 72h. The incision function is a function for using a high-frequency current with the surgical instrument 152 to incise the surgical site of patient P. The suction function is a function for using the surgical instrument 152 to aspirate cleaning water used to clean the surgical site of patient P or liquid such as blood generated at the surgical site of patient P. The water drainage function is a function for supplying cleaning water to the surgical site of patient P with the surgical instrument 152. Before the surgery on patient P, the operator operates the remote control device 2 to assign a desired function to the customization switch 72h at a desired position.

[0058] The multiple operation pedals 212 include, for example, a pedal that accepts operation of the endoscopic camera 151 and a pedal that accepts operation of the medical instrument 15, such as incision with the surgical instrument 152. For example, the multiple operation pedals 212 include a pedal that accepts a switching command for the arm 14 operated by each of the operation manipulators 211L and 211R. That is, the arm 14 of the surgery support robot 10 is associated with the operation manipulators 211L and 211R by operation of the operation pedals 212, for example, and the associated arm 14 moves in accordance with the operation of the operation manipulators 211L and 211R.

[0059] The controller 23 is communicably connected to the controller 16 on the surgical support robot 10 side. The controller 23 receives information on endoscopic images captured by the endoscopic camera 151 and displays it on the display device 22. The controller 23 acquires operation inputs accepted by the operation input device 21 and transmits them to the controller 16 on the surgical support robot 10 side. That is, the operator U1 commands the movement and operation of the medical instrument 15 by directly moving the operation manipulators 211L and 211R while checking the affected area on the endoscopic image displayed on the display device 22. The controller 23 can also display various information, such as information on the endoscopic image captured by the endoscopic camera 151, on the touch panel 213.

[0060] The controller 23 outputs audio information from the speaker 24. The audio information may be various types of audio information stored in advance in the memory 232, such as warning sound information indicating a warning sound.

[0061] First Embodiment Next, an operation training system 100, which is an assistance system according to the first embodiment, will be described with reference to Fig. 10. In the above-described surgery assistance system 1, the operator U1 is required to have mature operation skills in order to smoothly proceed with the surgery. The operation training system 100 shown in Fig. 10 is a system for improving the operation skills of the operator U1.

[0062] The operation training system 100 includes a remote control device 20 and a simulation device 50 communicably connected to the remote control device 20. The operation training system 100 executes a simulation in which a virtual surgery simulating a surgery is performed in a virtual space. That is, in response to the operation of an operator U1, a virtual surgery support robot simulating the above-mentioned surgery support robot 10 is operated in the virtual space to perform the virtual surgery. The operation training system 100 improves the operation skills of the operator U1 through the virtual surgery.

[0063] The remote control device 20 in the operation training system 100 has substantially the same configuration as the remote control device 20 in the above-described surgery support system 1, and therefore a description thereof will be omitted. Note that in the operation training system 100 of this embodiment, the interface device that constitutes the interface with the operator U1 may have a different configuration from the remote control device 20 in the surgery support system 1. The interface device may be a device that has substantially the same or similar configuration as the operation input device 21 included in the remote control device 20.

[0064] The simulation device 50 includes a simulator 51, which is an information processing device. The simulation device 50 also includes an input device 52 for inputting information necessary for performing the virtual surgery, and a display 53 for displaying a setting screen for the virtual surgery. Examples of the input device 52 include a keyboard, a mouse, and a touch panel. Note that the input device 52 and the display 53 are not essential. For example, the operation input device 21 of the remote control device 20 may function as an input device for inputting the simulation settings, or the display 22 may display a setting screen for the virtual surgery.

[0065] FIG. 11 is a block diagram showing an example of the configuration of a control system of an operation training system 100, which is an assistance system according to the first embodiment.

[0066] The simulator 51 can be realized as an information processing device such as a personal computer. The simulator 51 includes at least one processor 511 such as a CPU, and a memory 512. The memory 512 is configured with a storage device such as a semiconductor memory such as a volatile memory or a nonvolatile memory, a hard disk, or an SSD (Solid State Drive).

[0067] The memory 512 stores various information necessary for the simulation of virtual surgery. The simulator 51 executes various processes by at least one processor 511 executing the programs stored in the memory 512. The simulator 51 is connected to the controller 23 of the remote operation device 20 so as to be able to communicate with the controller 23.

[0068] The controller 23 of the remote control device 20 includes at least one processor 231 such as a CPU, and a memory 232. The memory 232 is configured with a storage device such as a semiconductor memory such as a volatile memory or a nonvolatile memory, a hard disk, or an SSD (Solid State Drive). The at least one processor 231 executes a program stored in the memory 232, causing the controller 23 to perform various processes.

[0069] The controller 23 controls the overall operation of the remote control device 20. Based on information, data, commands, etc. received from the simulation device 50, the controller 23 controls the operations to be performed by the operating manipulators 211L and 211R, etc., and the image display operation of the display device 22, etc.

[0070] For example, the controller 23 receives image information indicating a virtual endoscopic image that simulates an endoscopic image from the simulation device 50, and causes the virtual endoscopic image to be displayed on the display device 22. The controller 23 may also apply conversion processing or the like to the image information and cause the image information to be displayed on the display device 22.

[0071] The controller 23 transmits an operation command indicating the content of the operation accepted by the operation input device 21 to the simulation device 50. For example, the operation command may be a command regarding the position, posture, or movement of the virtual surgical support robot, or any combination thereof.

[0072] For example, each of the operating manipulators 211L and 211R includes servo motors MM1, ..., MMn (n is a natural number representing the number of active joints) and rotation sensors EM1, ..., EMn (n is a natural number representing the number of active joints) that are respectively arranged at the multiple joints included in each manipulator. The rotation sensors EM1, ..., EMn are, for example, encoders. Note that, for simplicity of illustration, the diagram shows the blocks of the servo motors MM1, ..., MMn and rotation sensors EM1, ..., EMn only for the operating manipulator 211L. The detection values ​​of the rotation sensors EM of the operating manipulators 211L and 211R detect the amount of rotation of each joint, and the processor 231 of the controller 23 generates an operation command based on the amount of rotation of each joint and sends it to the simulator 51. In this embodiment, each of the operating manipulators 211L and 211R does not necessarily have to include the servo motors MM1, ..., MMn.

[0073] Upon receiving the operation command, the simulator 51 operates the virtual surgery support robot in the virtual space based on the operation command. As a result of operating the virtual surgery support robot, the simulator 51 changes the image captured by the virtual endoscopic camera. The simulator 51 sends the image captured by the virtual endoscopic camera, which can change in response to the operation command, to the display 22. In this manner, the virtual surgery progresses.

[0074] Furthermore, the processor 511 of the simulator 51 evaluates the operation of the operator U1 based on the received operation command and predetermined operation evaluation rules. The operation command is an example of operation information indicating the content of the operation accepted by the operation input device 21. For example, the processor 511 evaluates the operating status of the robot resulting from the operation of the operator U1 or the operation skill of the operator U1 based on predetermined operation evaluation rules. The evaluation of the operation of the operator U1 will be described in detail later.

[0075] Each processing device of the controller 23 and the simulator 51 may execute each process under centralized control by a single processor, or may execute each process under distributed control through cooperation of multiple processors. The controller 23 and the simulator 51 may each be configured by at least one or a combination of two or more of a computer, a personal computer, a microcontroller, a microprocessor, a programmable logic device (PLD) such as an FPGA (field-programmable gate array), a programmable logic controller (PLC), and a logic circuit, for example.

[0076] The controller 23 and the simulator 51 may be configured integrally. In other words, the processing of one of the controller 23 and the simulator 51 may be performed by the other of the controller 23 and the simulator 51. For example, the operation training system 100 does not need to include the controller 23, and an operation command may be sent directly from the operation input device 21 to the simulator 51, or image information may be sent directly from the simulator 51 to the display 22. One or more processors and one or more memories included in the controller 23, the simulator 51, or any combination thereof is an example of a processing circuit.

[0077] [Control flow of virtual surgery] FIG. 12 is a flowchart illustrating the flow of virtual surgery in the operation training system 100.

[0078] The processor 511 of the simulator 51 sets surgical environment information (step S1). Specifically, the surgical environment information is information necessary for simulating a virtual surgery.

[0079] For example, the setting of surgical environment information is realized in the simulation device 50 by a user inputting various pieces of information via the input device 52 while viewing a setting screen displayed on the display 53. Note that the person who sets up the surgical environment information does not have to be the operator U1 of the operation input device 21 who performs the virtual surgery, but may be someone other than the operator U1, such as a person associated with the training center where the operation training system 100 is installed. For this reason, in the following description, the person who sets up the virtual surgery and the person who performs the virtual surgery may be simply referred to as the user without making a distinction between them.

[0080] The surgical environment information may include surgical content information indicating the content of the virtual surgery. For example, the surgical content information may include information indicating the name of a disease, information indicating a surgical procedure, or a combination thereof.

[0081] The surgical environment information may include surgical preparation information related to preparation for the virtual surgery, such as information about the position of the virtual patient, information about the orientation of the virtual operating table, information about the insertion position of a virtual medical instrument on the body surface of the virtual patient, information about the position of the virtual surgical robot relative to the virtual operating table, information about the orientation of a virtual arm included in the virtual surgical robot, or any combination thereof.

[0082] The surgical environment information may include peripheral object information regarding virtual peripheral objects arranged around the virtual surgical robot. The peripheral object information includes information indicating the positional relationship between the virtual peripheral object and the virtual surgical robot. The peripheral object information includes information indicating the positional relationship between the virtual peripheral object and elements of the virtual surgical robot, such as a virtual arm. The virtual peripheral object may include a virtual operating table and virtual peripheral equipment. The virtual peripheral object may also include a virtual patient. In other words, the peripheral object information may include patient information regarding the virtual patient in the virtual space.

[0083] The surgical environment information may include patient information about a virtual patient undergoing virtual surgery. The patient information may include, for example, body shape information about the body shape of the virtual patient. The patient information may include at least one of the virtual patient's height, chest circumference, and abdominal circumference. The patient information may be three-dimensional information about the virtual patient's body shape. For example, the patient information may include data obtained by 3D scanning a portion of the virtual patient's body or the entire body. The patient information may include information about weight, body fat percentage, etc. The patient information may include information about the virtual patient's age. The patient information may include information about the virtual patient's gender.

[0084] An example of a method for setting surgical environment information will be described below with reference to Figures 13 to 15. Figures 13 to 15 show examples of virtual surgery setting screens D1, D2, and D3. In the following description, the symbol for the model of each element will be indicated by adding "M" before the symbol for each element in real space, such as the surgery support system 1 or the patient P. For example, a virtual operating table, which is a three-dimensional model of the operating table 31, will be assigned the symbol "M31."

[0085] FIG. 13 shows a setting screen D1 for setting various information related to the patient and affected area who will be the subject of the virtual surgery to be performed. The setting screen D1 includes an input box B1 for specifying the name of the disease to be the subject of the virtual surgery, an input box B2 for specifying the name of the procedure for the virtual surgery, and an input box B3 for specifying the body shape of the patient who will be the subject of the virtual surgery. For each of the input boxes B1, B2, and B3, the user can select one from a plurality of predetermined options. The information specified in each of the input boxes B1 and B2 corresponds to surgery content information, and the information specified in each of the input boxes B3 corresponds to patient information.

[0086] The setting screen D1 shows a patient model MP, which is a three-dimensional model of a virtual patient, and a virtual affected area MV on the patient model MP. The virtual affected area MV is positioned at a location on the patient model MP according to the surgical content information set by the user. The position of the virtual affected area MV on the patient model MP may be adjusted by the user, or may be set by the processor 511 randomly or according to a predetermined rule.

[0087] For example, the state of the virtual affected area MV and changes in the state of the affected area MV in response to treatment on the virtual affected area MV may be randomly determined by the processor 511 for each virtual surgery. This is to help train the operator U1 to respond flexibly to situations. For example, a new three-dimensional model of the virtual affected area MV may be generated for each virtual surgery, or may be randomly selected from a plurality of predetermined models. As three-dimensional models of the virtual affected area MV, a plurality of sample models, such as a sample model for the operator U1 who is unfamiliar with virtual surgery, may be stored in the memory 512, and the user may be able to freely select a sample model to use in the virtual surgery.

[0088] 14 shows a setting screen D2 for setting some of the surgery preparation information. Specifically, the setting screen D2 allows the user to set the position of the virtual patient MP, the posture of the virtual operating table M31, and the positions r1, ..., r4 of the ports on the body surface of the virtual patient MP, which are the insertion positions of the virtual medical instruments M15.

[0089] 15 shows a setting screen D3 for setting some of the surgery preparation information. Specifically, the setting screen D3 sets the position of the virtual surgery-assist robot M10 relative to the virtual operating table M31, more specifically, the position of the virtual support body M11 relative to the virtual operating table M31. The setting screen D3 also sets the position of the virtual arm base M13 relative to the virtual operating table M31.

[0090] In the above-described method for setting surgical environment information, the surgical environment information is set by the user, but some or all of the information in the surgical environment information does not have to be set by the user and may be predetermined. For example, some or all of the information included in the surgical preparation information may be set automatically in accordance with information input by the user regarding the details of the surgery or other elements. For example, the position of the virtual patient MP, the posture of the virtual operating table, and insertion positions r1, r2, r3, and r4 of the virtual medical instrument M15 on the body surface of the virtual patient MP may be set automatically based on the disease name and surgical procedure.

[0091] By setting a part of the information included in the surgery preparation information, other information included in the surgery preparation information may be set. For example, when the user specifies the position of the virtual surgery-assist robot M10 relative to the virtual operating table M31, such as "to the right of the virtual patient MP," the postures of the virtual positioner M12 and the virtual arm M14 may be automatically set.

[0092] After the surgical environment information is set, the simulator 51 starts the virtual surgery (step S2).

[0093] FIG. 16 shows an example of a display screen D4 for a virtual endoscopic image. When the virtual surgery begins, the simulator 51 transmits an endoscopic image captured by a virtual endoscopic camera in the virtual space, as shown in FIG. 16, to the remote control device 20, which then displays the received endoscopic image on the display 22. The simulator 51 also controls the operation of the virtual surgical assistant robot in the virtual space in response to operations received by the operation input device 21. More specifically, the operator U1 operates the operation input device 21 to transmit operation information, which is an operation command indicating the content of the operation, to the simulator 51, and the simulator 51 operates the virtual surgical assistant robot based on the operation information. In this way, the operator U1 proceeds with the virtual surgery by operating the operation input device 21 while viewing the endoscopic image, thereby changing the imaging range of the virtual endoscopic camera and moving the virtual surgical instrument M152 to treat the virtual affected area MV.

[0094] Processor 511 executes real-time evaluation processing during the virtual surgery (step S3). Processor 511 determines whether the virtual surgery has ended (step S4), and repeats the real-time evaluation processing while it determines that the virtual surgery has not ended (step S4: No). When processor 511 determines that the virtual surgery has ended (step S4: Yes), processor 511 executes total evaluation processing for the virtual surgery (step S5), and then processor 511 outputs the evaluation result (step S6).

[0095] The real-time evaluation process in step S3 is a process for evaluating the operations of the operator U1 in real time during the virtual surgery. The total evaluation process in step S5 is a process for evaluating the operations of the operator U1 throughout the entire virtual surgery after the virtual surgery.

[0096] In one or both of the real-time evaluation process and the total evaluation process, the processor 511 acquires operation information and evaluates the operation of the operator U1 corresponding to the operation information based on predetermined operation evaluation rules. In this embodiment, the operation of the operator U1 is evaluated based on multiple evaluation items. The operation evaluation rules include multiple rules corresponding to the multiple evaluation items.

[0097] The information used to evaluate the operation of the operator U1 may differ depending on the evaluation item. For example, for a certain evaluation item, the processor 511 may acquire peripheral object information about virtual peripheral objects placed around the virtual surgery assist robot, and evaluate the operation of the operator U1 based on the peripheral object information and a predetermined operation evaluation rule.

[0098] Hereinafter, the method for evaluating the operation of the operator U1 by the processor 511 will be specifically described for each evaluation item.

[0099] (Accessibility) The operation evaluation rules include accessibility evaluation rules. The accessibility evaluation rules relate to the ease of access to a predetermined surgical target by a virtual surgery support robot operated due to the operation of the operator U1. Note that operating due to the operation of the operator U1 may mean operating based on an operation command. For example, the accessibility evaluation rules are rules for determining the ease of access to the virtual affected area MV of the virtual arm at a position after movement due to the operation of the operator U1.

[0100] For example, the range of motion of the virtual arm M14 may be limited due to virtual peripheral objects. The processor 511 calculates an index of ease of movement of the virtual arm based on position information of the virtual arm that moves in response to operation information, peripheral object information, and the accessibility evaluation rule. For example, when the processor 511 causes the virtual arm, after moving due to the operation of the operator U1, to perform multiple assumed next hypothetical motions, it calculates how many of the multiple hypothetical motions will cause interference between the virtual arm and the virtual medical instrument M152 attached thereto. The processor 511 calculates an index value corresponding to ease of access so that the index value corresponding to ease of access decreases as the number of interfering motions increases.

[0101] Furthermore, the virtual arm M14 must avoid certain postures, such as singular points. A singular point is a posture in which the degree of freedom in a specific direction is lost due to mechanical constraints of the manipulator. The processor 511 calculates an index of ease of movement of the virtual arm based on position information of the virtual arm operated by operation information and the accessibility evaluation rule. For example, when the processor 511 causes the virtual arm, after moving due to the operation of the operator U1, to perform multiple hypothetical next movements, it calculates how many of the hypothetical movements result in the virtual arm and the virtual medical instrument M152 attached thereto assuming a posture that should be avoided. The processor 511 calculates an index value corresponding to ease of access so that the index value corresponding to ease of access decreases as the number of times the posture that should be avoided increases.

[0102] (Possible interference between arms) The operation evaluation rules include interference evaluation rules, which relate to the possibility of interference between multiple virtual arms M14.

[0103] For example, the interference assessment rule is a rule that quantifies the possibility of interference between a virtual arm that has moved due to the operation of the operator U1 and another virtual arm. The processor 511 calculates an index value that indicates the possibility of interference between the multiple virtual arms M14 based on the interference assessment rule and the position information of all virtual arms including the virtual arm that has moved due to the operation information. For example, the processor 511 calculates the index value corresponding to the possibility of interference so that the smaller the distance between the virtual arm that has moved due to the operation of the operator U1 and another virtual arm, the larger the index value corresponding to the possibility of interference.

[0104] Alternatively, for example, the processor 511 may evaluate the possibility of interference between the multiple virtual arms M14 using an evaluation method similar to that for accessibility. That is, the processor 511 calculates how many of the multiple hypothetical actions will cause the virtual arms to interfere with each other when the virtual arms are caused to perform multiple hypothetical actions after the virtual arms are operated due to the operation of the operator U1. The processor 511 calculates an index value corresponding to the possibility of interference such that the more interfering actions there are, the larger the index value corresponding to the possibility of interference becomes.

[0105] (Total surgery time) The operation evaluation rules include time evaluation rules. The time evaluation rules relate to the total surgical time required for the virtual surgery performed using the virtual surgery robot. For example, the processor 511 counts the elapsed time from the start of the virtual surgery in step S2 to acquire time information indicating the total surgical time from the start of the virtual surgery to the time when it is determined in step S4 that the virtual surgery has ended. In other words, the time information indicates the time from the start of acquisition of operation information for the virtual surgery to the end of acquisition of the operation information.

[0106] Target time information indicating a target total surgery time according to the type of surgery is stored in memory 512. The target time information can be set in advance taking into consideration the time required by an experienced doctor to perform a similar surgery or virtual surgery in the past.

[0107] In the total evaluation process, the processor 511 calculates an index value indicating whether the virtual surgery performed by the operator U1 took too long based on the time information of the virtual surgery, the surgical content information of the virtual surgery, and the target time information corresponding to the surgical content.

[0108] The time evaluation rule may include, in addition to or instead of the rule regarding the total surgery time, a rule regarding the time required for each operation when the virtual surgery is divided into multiple operation steps. For example, the evaluation based on the time evaluation rule may be based on the time required for each operation step. For example, the processor 511 may divide the virtual surgery into multiple operation steps predetermined according to the operation content based on the operation content information, and count the elapsed time for each divided operation step. The multiple operation steps may be steps with different operation content, such as "resection of the lesion," "stopping bleeding," "extracting the resected lesion from the body," and "suturing the resected site."

[0109] The memory 512 may store target time information indicating a target time for each operation step according to the surgical content, and the processor 511 may compare the time counted for each operation step with the corresponding target time. This makes it possible to determine which operation step in the virtual surgery performed by the operator U1 is taking too long.

[0110] (Deviation from standard operation) The operation evaluation rule includes a comparison evaluation rule for comparing the acquired operation information with reference information indicating the standard of the operation accepted by the operating manipulators 211L, 211R corresponding to the operating status of the virtual surgery assist robot. In this embodiment, the reference information indicates the content of the operation recommended as the operation accepted by the operating manipulators 211L, 211R in the operating status of the virtual surgery assist robot, and the comparison evaluation rule relates to how much the acquired operation information deviates from the reference information indicating the content of the standard operation.

[0111] The processor 511 acquires reference information indicating the content of the reference operation, and compares the operation information indicating the operation content of the operator U1 with the reference information corresponding to the situation in which the operation information was acquired. The processor 511 may evaluate that the quality of the operation content of the operator U1 is poorer as the deviation between the operation of the operator U1 and the reference operation becomes greater.

[0112] The reference information is generated based on past operation information indicating the content of an operation for operating the surgery assist robot 10 in a surgery performed in the past or the virtual surgery assist robot M10 performed in the past.

[0113] For example, the process of generating the reference operation is performed using a trained model generated using a known machine learning algorithm. For example, the memory 512 may store a trained model that has undergone machine learning to estimate a reference operation, which is an operation to be performed for a certain surgical situation. The processor 511 acquires surgical situation information indicating the current situation of the surgery and inputs the surgical situation information into the trained model, thereby acquiring reference information indicating the content of the reference operation from the trained model.

[0114] The surgical situation information may include information indicating an endoscopic image. For example, the trained model is generated by machine learning using, as training data, endoscopic image information indicating an endoscopic image from a past surgery and past operation information corresponding to operations performed when the endoscopic image was captured. Alternatively, the trained model is generated by machine learning using, as training data, endoscopic image information indicating a virtual endoscopic image from a past virtual surgery and past operation information corresponding to operations performed when the virtual endoscopic image was captured.

[0115] The surgery status information may include surgery details information such as the name of the disease and the surgical procedure.

[0116] The evaluation of each evaluation item described above is performed by real-time evaluation processing or total evaluation processing. For example, evaluations of "ease of access," "possibility of interference between arms," ​​and "deviation from standard operation" are performed by real-time evaluation processing, while evaluation of "total surgery time" is performed by total evaluation processing after the virtual surgery. Furthermore, in the total evaluation processing, evaluation results obtained in the real-time evaluation processing throughout the entire virtual surgery can be used to comprehensively evaluate the operations throughout the entire virtual surgery.

[0117] In step S14, processor 511 may output the evaluation items and the evaluation results for the operations of the operator corresponding to the evaluation items in association with each other.

[0118] 17 is an example of a display screen D5 of the evaluation results of the virtual surgery. The display screen D5 of the evaluation results shows the results of the evaluation item "Possibility of Interference Between Arms" in a graph with time on the horizontal axis and an index value of the possibility of interference on the vertical axis. For example, by looking at this evaluation result, the operator U1 can understand at what point in the virtual surgery a situation where there was a high possibility of interference between the arms was reached.

[0119] Figure 18 is an example of a display screen D6 of the evaluation results of virtual surgery. The display screen D6 of the evaluation results shows multiple evaluation items, such as "ease of access," "possibility of interference between arms," ​​"deviation from standard operation," and "total surgery time," in association with the evaluation results corresponding to each evaluation item. In the example of Figure 18, the evaluation results for each evaluation item are shown using a predetermined multi-level rating scale, such as "Good," "Average," and "Below Average."

[0120] The display screens D5 and D6 for the evaluation results are displayed on the display 53 of the simulation device 50, but the evaluation results may also be displayed on the display 22 of the remote control device 20 or the display of the touch panel 213.

[0121] The above-described method for outputting the evaluation results is merely an example.

[0122] For example, the processor 511 may display on the display the result of comparing the operation corresponding to the operation information with the standard operation corresponding to the standard information as a result of evaluating the operator's operation based on the comparison evaluation rules for the operation information and the standard information. For example, the processor 511 may extract differences between the operation corresponding to the operation information and the operation corresponding to the standard information and display the differences on the display. For example, when the area incised by the operation of the operator U1 in the virtual surgery differs from the area to be incised by the standard operation corresponding to the standard information, the processor 511 may display a virtual endoscopic image on the display and indicate on the virtual endoscopic image the area to be incised by the standard operation corresponding to the standard information.

[0123] As described above, the operation training system 100 of this embodiment allows the operator U1 to perform a virtual surgery that recreates a surgical procedure. In this embodiment, the processor 511 of the simulator 51 evaluates operation information indicating the operation content for operating the virtual surgical support robot M10 in the virtual surgery based on predetermined operation evaluation rules and presents the evaluation results to the operator U1. This allows the operator U1 to objectively grasp his / her own current operation skills and to know points in his / her operation that need improvement, which can lead to the improvement of his / her operation skills. In this way, this embodiment can support the operator U1's operation of the surgical support robot from the perspective of improving his / her operation skills.

[0124] Second Embodiment Next, a surgery support system 1, which is a support system according to the second embodiment, will be described with appropriate reference to Figures 20, 21, and 22, and the above-mentioned Figures 1 to 9. The surgery support system 1 of this embodiment has a function of evaluating the operation of an operator U1 with respect to a surgery support robot 10 and supporting the operator U1 according to the evaluation result.

[0125] [Operating table equipment configuration] 19 is a diagram for explaining the operating table apparatus 30 and the environmental information acquisition device 40, and is a side view of the inside of the operating room R. Note that in FIG. 19, only the operating table apparatus 30 and the environmental information acquisition device 40 are shown, and other elements in the surgery support system 1 are omitted.

[0126] The operating table apparatus 30 includes an operating table 31 on which the patient P is placed, and a support structure 32 that supports the operating table 31. The operating table 31 may also be referred to as a table body 31. The support structure 32 includes a movement structure that moves the operating table 31 relative to the floor surface. More specifically, the support structure 32 is configured to be able to change the position, orientation, posture, or any combination thereof of the operating table 31. The position, orientation, posture, or any combination thereof of the operating table 31 may also be referred to as a table state.

[0127] Specifically, the support structure 32 includes a fixed element 32a that is fixed to the floor surface and at least one movable element 32b that is connected to the fixed element 32a and is displaceable relative to the fixed element 32a. The operating table 31 is fixed to an upper portion of the movable element 32b. At least one joint is interposed between the fixed element 32a and the movable element 32b. The joint is a prismatic joint or a rotary joint.

[0128] The table state may include the tilt of at least a portion of the operating table 31. For example, the support structure 32 is configured to be able to change the tilt of the operating table 31 relative to the horizontal plane, in other words, the tilt of the support surface of the operating table 31 on which the patient P rests. The support structure 32 may be configured to be able to tilt the support surface only in one predetermined direction, or may be configured to be able to tilt the support surface in multiple directions. For example, the support structure 32 may be configured to be able to move the movable element 32b so that the orientation of the top of the head of the patient P resting on the operating table 31 relative to the horizontal plane and / or the left-right orientation of the body of the patient P resting on the operating table 31 can be changed.

[0129] The operating table 31 may also have multiple bases, each having a support surface. The multiple bases may be connected to each other in a displaceable manner. For example, as illustrated in FIG. 19, the multiple bases 31a, 31b may be capable of changing their orientation relative to each other. For example, the base 31a on which the back of the patient P is placed and at least one second base 31b, separate from the first base 31a, on which the feet of the patient P are placed may be capable of changing their orientation relative to each other. A movable element 32b may be fixed to each of the multiple bases, or some of the multiple bases may be fixed to a fixed element 32a.

[0130] The operating table device 30 includes a drive device 33 (see FIG. 20), an operation input device 34 for receiving input of commands from a user U, and a third controller 35.

[0131] The drive device 33 drives the support structure 32 to displace the operating table 31. The drive device 33 includes at least one actuator. The drive device 33 includes at least one actuator that displaces at least one movable element 32b. The type of actuator is not particularly limited. For example, the actuator may be an electric motor, a hydraulic actuator, or a pneumatic actuator.

[0132] In this embodiment, the operation input device 34 is a touch panel, but may be realized by a known operation tool such as a lever, a button, a joystick, a pedal, or a motion capture device.

[0133] The operating table apparatus 30 includes a detector that detects table status information indicating the table status. In this embodiment, the detector includes at least one position sensor 36 that detects a physical quantity corresponding to the position of at least one movable element 32b. The third controller 35 is communicatively connected to the drive device 33, the operation input device 34, and the position sensor 36 via wired or wireless communication. For example, the position sensor 36 may be a sensor that detects the amount of linear movement of a linear joint, or a rotation angle sensor of a rotary joint, such as an encoder. Examples of sensors that can be used to detect the table status information include a gyro sensor, a tilt sensor, a geomagnetic sensor, an encoder, a potentiometer, a limit switch, a stroke sensor, a vision sensor, a 3D scanner, a laser tracker, a laser scanner, radar, a LiDAR (Light Detection and Ranging), a stereo camera, or any combination thereof.

[0134] [Configuration of environmental information acquisition device] The environmental information acquisition device 40 is a device for acquiring information about the environment within the operating room R. The environmental information acquisition device 40 is equipped with a plurality of object detection sensors 41. In this embodiment, the object detection sensors 41 detect three-dimensional information of objects within the operating room R. The object detection sensors 41 detect, for example, the three-dimensional position and three-dimensional shape of objects within the operating room R. The environmental information acquisition device 40 can also be referred to as a three-dimensional information acquisition device that acquires three-dimensional information within the operating room R.

[0135] In this embodiment, for example, the object detection sensor 41 is used to detect three-dimensional information of a patient P in the operating room R, as will be described later. Also, for example, the object detection sensor 41 is used to detect three-dimensional information of objects (which may include a patient) around the surgical support robot 10 in the operating room R, as will be described later.

[0136] The three-dimensional information of an object may be information indicating the shape and position of the object in the operating room R. For example, the three-dimensional information of a patient P may be information indicating the body shape of the patient P, the position and posture of the patient P in the operating room R.

[0137] The object detection sensor 41 is, for example, a vision sensor. The object detection sensor 41 may also be a distance measuring sensor or measuring device capable of acquiring three-dimensional information. The object detection sensor 41 may also be a three-dimensional scanner, a laser tracker, a laser scanner, a radar, a LiDAR (Light Detection And Ranging), or a stereo camera. The multiple object detection sensors 41 may be of different types.

[0138] The multiple object detection sensors 41 are arranged at intervals within the operating room R. The multiple object detection sensors 41 may be stationary sensors fixed within the operating room, or may be portable sensors that can be carried around within the operating room. For example, a given object detection sensor 41 may be supported on a side wall or ceiling within the operating room R. A given object detection sensor 41 may also be supported by a support structure such as a tripod at a predetermined position on the floor of the operating room R. A given object detection sensor 41 may also be fixed to the surgical support robot 10. Specifically, a given object detection sensor 41 may be fixed to any one of the support body 11, the positioner 12, the arm base 13, and the arm 14.

[0139] The environmental information acquisition device 40 may include a moving device 42. The moving device 42 is configured to change at least one of the position of the object detection sensor 41 within the operating room R and the orientation of the object detection sensor 41 within the operating room R. The moving device 42 enables a single moved object detection sensor 41 to detect objects from multiple positions or multiple directions in the operating room R.

[0140] The moving device 42 includes a moving structure 42a. The moving structure 42a is configured to support the object detection sensor 41 and to move the object detection sensor 41 by being driven by a driving device 42b (see FIG. 20) described below. For example, FIG. 19 shows, as an example of the moving structure 42a, a rotation mechanism that supports the object detection sensor 41 by suspending it from the ceiling of the operating room R. The rotation mechanism is configured to rotate and move the object detection sensor 41 about a certain vertical line C that passes through the ceiling of the operating room R. In the example of FIG. 19, the object detection sensor 41 is disposed at a position on the moving structure 42a that is a predetermined distance away from the vertical line C, but it may also be disposed on the vertical line C.

[0141] However, the configuration of the moving structure 42a is not particularly limited. The moving structure 42a may be a link mechanism having multiple joints, with its base end fixed to the operating room R and the object detection sensor 41 fixed to its tip. That is, the moving device 42 may be a multi-joint robot arm. In this case, the driving device 42b may be at least one servo motor that drives at least one joint of the multi-joint robot arm. That is, the object detection sensor 41 and the moving device 42 that supports it may form an arm-type three-dimensional measuring machine. Furthermore, the moving structure 42a may include wheels that rotate on the floor surface and a support that rotatably supports the wheels. In this case, the driving device 42b may be a motor that drives the wheels to rotate.

[0142] The environmental information acquisition device 40 includes a fourth controller 43 that controls the operation of the driving device 42b (see FIG. 20).

[0143] [Control system configuration of surgical support system] 20 is a block diagram showing an example of the configuration of a control system of the surgery support system 1, which is a support system according to the second embodiment. In the following description, the controller 16 of the surgery support robot 10 will be referred to as the "first controller 16," the controller 23 of the remote control device 20 will be referred to as the "second controller 23," the controller 35 of the operating table device 30 will be referred to as the "third controller 35," and the "controller 43" of the environmental information acquisition device 40 will be referred to as the "fourth controller 43."

[0144] [Control system configuration of surgical support system] (Configuration and functions of the first controller) In the surgery support robot 10, the first controller 16 is electrically connected to the positioner driving device 129, multiple arm driving devices 149, an endoscopic camera 151, and multiple instrument driving devices 153. The first controller 16 is also electrically connected to multiple arm-side controllers 17 and an interface device 114. Note that, in Fig. 20, for the sake of simplicity, only one block of each of the arm driving device 149, instrument driving device 153, and arm-side controller 17 is shown.

[0145] The positioner driving device 129 includes servo motors MP1, ..., MP7 and rotation sensors EP1, ..., EP7. The servo motors MP1, ..., MP7 are arranged at the joints JP1, ..., JP7, respectively. The servo motors MP1, ..., MP7 are motors that rotate the joints JP1, ..., JP7, respectively. The rotation sensors EP1, ..., EP7 are sensors that detect the amount of rotation of the servo motors MP1, ..., MP7, respectively. The rotation sensors EP1, ..., EP7 are, for example, encoders.

[0146] The arm driving device 149 includes servo motors MA1, ..., MA8 and rotation sensors EA1, ..., EA8. The servo motors MA1, ..., MA8 are arranged at the joints JA1, ..., JA8, respectively. The servo motors MA1, ..., MA8 are motors that rotate the joints JA1, ..., JA8, respectively. The rotation sensors EA1, ..., EA8 are sensors that detect the amount of rotation of the servo motors MA1, ..., MA8, respectively. The rotation sensors EA1, ..., EA8 are, for example, encoders.

[0147] The instrument driving device 153 is a device for driving joints JT1, ..., JTn (n is a natural number) included in the surgical instrument 152. The instrument driving device 153 includes servo motors MT1, ..., MTn (n is a natural number) and rotation sensors ET1, ..., ETn (n is a natural number). The servo motors MT1, ..., MTn are motors that rotate and drive the joints JT1, ..., JTn, respectively. The rotation sensors ET1, ..., ETn are sensors that detect the amount of rotation of the servo motors MT1, ..., MTn, respectively. The rotation sensors ET1, ..., ETn are, for example, encoders.

[0148] In this embodiment, the tool driving device 153 is disposed within the tip link 148. That is, the servo motors MT1, ..., MTn are disposed within the tip link 148, and the driving forces of the servo motors MT1, ..., MTn are transmitted to the joints JT1, ..., JTn via power transmission mechanisms disposed within the base 152a and the shaft 152b.

[0149] Each of the positioner drive unit 129, arm drive unit 149, and tool drive unit 153 includes a reducer and a power transmission mechanism corresponding to each servo motor of each drive unit. The reducer reduces the output of the corresponding servo motor to amplify the torque. The power transmission mechanism transmits the output of the corresponding servo motor to a corresponding link or the like. The power transmission mechanism may be composed of multiple gears, a transmission wire, a transmission belt, or any combination thereof.

[0150] (Controller configuration and functions) The first controller 16 includes at least one processor 161 such as a CPU, and a memory 162. The memory 162 is configured from a semiconductor memory such as a volatile memory or a nonvolatile memory, a hard disk, an SSD (Solid State Drive), or other storage device. The at least one processor 161 executes a program stored in the memory 162, causing the first controller 16 to perform various processes.

[0151] The first controller 16 includes servo amplifiers corresponding to the plurality of servo motors included in the positioner driving device 129, the arm driving device 149, and the tool driving device 153. Each servo amplifier is electrically connected to the corresponding servo motor. The servo amplifier calculates a drive current corresponding to a command value calculated by the processor 161 and supplies the drive current to the corresponding servo motor.

[0152] The first controller 16 controls the overall operation of the surgery assist robot 10. For example, the first controller 16 controls the operation of the positioner 12, the operation of each arm 14, and the operation of each medical instrument 15.

[0153] Various types of information required for robot-assisted surgery and its preparations are stored in the memory 162 of the first controller 16. The memory 162 stores information related to the content of the surgery, such as information indicating the surgical procedure, information indicating the surgical target, and information related to the surgical procedure.

[0154] Furthermore, for example, the memory 162 stores information regarding the configuration of elements included in the surgery support system 1 used in surgery. For example, the memory 162 may store information regarding the surgical instruments 152 used in surgery, information regarding the surgery support robot 10, information regarding the remote control device 20, etc. The information may be information regarding the type and dimensions of each element, such as the medical instruments 15, the surgery support robot 10, and the remote control device 20.

[0155] The memory 162 may store information about the surgical instrument 152 held at the tip of each arm 14. The information about the surgical instrument 152 may include information about the type, shape, dimensions, direction of movement, and range of movement of the surgical instrument 152.

[0156] The first controller 16 acquires image information indicating an endoscopic image captured by the endoscopic camera 151 and transmits it to the second controller 23.

[0157] (Configuration and function of the second controller) The first controller 16 of the surgery support robot 10 is communicably connected to the second controller 23 of the remote control device 20. Regarding the second controller 23, a description that overlaps with the first embodiment will be omitted.

[0158] The second controller 23 controls the overall operation of the remote control device 20. Based on information, data, commands, etc. received from the first controller 16, the second controller 23 controls the operations to be performed by the operation manipulators 211L and 211R, etc., and the image display operation of the display device 22, etc.

[0159] For example, the second controller 23 receives image information indicating an endoscopic image captured by the endoscopic camera 151 from the first controller 16, and displays the endoscopic image on the display device 22. The second controller 23 may perform conversion processing or the like on the image information and display it on the display device 22.

[0160] The second controller 23 transmits information corresponding to the operation input received by the operation input device 21 as operation information to the first controller 16. For example, the operation information may be a command regarding the position, posture, or movement of the surgical instrument 152, or any combination thereof.

[0161] For example, the detection values ​​of the rotation sensors EM of the operation manipulators 211L and 211R detect the amount of rotation of each joint, and the processor 231 of the second controller 23 generates operation information based on the amount of rotation of each joint and sends it to the first controller 16. Upon receiving the operation information, the first controller 16 controls the operations of the arm driving device 149 and the tool driving device 153 based on the operation information.

[0162] The second controller 23 also receives a position command from the first controller 16. The position command includes position and posture commands corresponding to the positions and postures of the positioner 12 and arm 14 of the surgical support robot 10. For example, the position command includes commands corresponding to the rotation angles of the joints of the positioner 12 and arm 14. Based on the received position command, the second controller 23 controls the servo motors MM of the operating manipulators 211L and 211R so that the positions and postures of the operating manipulators 211L and 211R move in accordance with the position and posture of the tip link 148 of the arm 14.

[0163] (Configuration and functions of the third controller) The first controller 16 of the surgery support robot 10 is connected to the third controller 35 of the operating table device 30 so as to be able to communicate with each other.

[0164] The third controller 35 includes at least one processor 351 such as a CPU, and a memory 352. The memory 352 is configured with a storage device such as a semiconductor memory such as a volatile memory or a nonvolatile memory, a hard disk, or an SSD (Solid State Drive). The at least one processor 351 executes a program stored in the memory 352, causing the third controller 35 to perform various processes.

[0165] The third controller 35 controls the overall operation of the operating table apparatus 30. For example, the third controller 35 controls the drive device 33 to change the attitude of the operating table 31. For example, the processor 351 of the third controller 35 controls the drive device 33 based on a command input to the operation input device 34 so that the position and attitude of the operating table 31 correspond to the command. Alternatively, for example, the third controller 35 controls the drive device 33 based on a command received from the first controller 16 so that the position and attitude of the operating table 31 correspond to the command.

[0166] The third controller 35 sends table status information indicating the position, orientation, or posture of the operating table 31 to the first controller 16. Based on the received table status information, the first controller 16 can control the operation of the positioner 12, the operation of the arm 14, and the operation of the medical instrument 15. For example, the first controller 16 can determine the preparatory posture based on the table status information, and control the operation of the positioner 12, the operation of the arm 14, and the operation of the medical instrument 15 so as to assume the determined preparatory posture.

[0167] The position of the operating table 31 indicated by the table status information includes information regarding the posture of the operating table 31. The table status information may include, for example, information indicating the inclination angle of the support surface on which the operating table 31 is placed relative to the horizontal plane. The table status information may also include information indicating the orientation of the operating table 31, such as the orientation of the operating table 31 in the operating room R. In this embodiment, the table status information indicating the position of the operating table 31 is a detection value by the position sensor 36, but the table status information may also be a command value for the drive device 33.

[0168] (Configuration and functions of the 4th controller) The first controller 16 of the surgery support robot 10 is connected to the fourth controller 43 of the environmental information acquisition device 40 so as to be able to communicate with each other.

[0169] The fourth controller 43 includes at least one processor 431 such as a CPU, and a memory 432. The memory 432 is configured with a storage device such as a semiconductor memory such as a volatile memory or a nonvolatile memory, a hard disk, or an SSD (Solid State Drive). The at least one processor 431 executes a program stored in the memory 432, causing the fourth controller 43 to perform various processes.

[0170] The fourth controller 43 sends information detected by the object detection sensors 41 to the first controller 16. The fourth controller 43 may process the information detected by the object detection sensors 41 and send it to the first controller 16. For example, the fourth controller 43 may combine the detection information from the multiple object detection sensors 41 to generate surrounding object information that is information about surrounding objects around the multiple arms 14, and send the surrounding object information to the first controller 16. Alternatively, the surrounding object information may be generated by the first controller 16 that receives the information detected by the object detection sensors 41.

[0171] The first controller 16 can control the operation of the positioner 12, the operation of the arm 14, and the operation of the medical instrument 15 based on the surrounding object information received from the environmental information acquisition device 40. For example, the first controller 16 can determine the ready posture based on the surrounding object information, and control the operation of the positioner 12, the operation of the arm 14, and the operation of the medical instrument 15 so as to assume the determined ready posture.

[0172] The peripheral object information is information relating to the positions and postures of objects around the surgical support robot 10. The fourth controller 43 can generate peripheral object information relating to the positions and postures of objects around the surgical support robot 10 based on the detection information of the object detection sensor 41 and position information indicating the position of the object detection sensor 41.

[0173] When the object detection sensor 41 is located at a fixed position within the operating room R, the position information indicating the position of the object detection sensor 41 is stored in advance in the memory 432 as known information.

[0174] If the position of the object detection sensor 41 can be displaced within the operating room R by the moving device 42, position information indicating the position of the object detection sensor 41 is calculated based on information related to the position and attitude of the moving device 42. Specifically, the moving device 42 includes a driving device 42b and a position sensor 42c that detects information corresponding to the position of the object detection sensor 41. If the driving device 42b is configured to include at least one servo motor that drives at least one joint of the articulated robot arm, the position sensor 42c may be a rotation sensor or the like that is disposed in each joint and detects the rotation angle of each joint.

[0175] Furthermore, if the object detection sensor 41 is movable within the operating room R, for example by being fixed to the arm 14 of the surgical support robot 10, position information indicating the position of the object detection sensor 41 can be obtained by an encoder included in the arm driving device, an encoder included in the positioner driving device, the object detection sensor 41 of the environmental information acquisition device 40, or a combination thereof.

[0176] Each of the first controller 16, the second controller 23, the third controller 35, and the fourth controller 43 may execute each process under centralized control by a single processor, or may execute each process under distributed control through cooperation of multiple processors. Each controller may be configured with at least one or a combination of two or more of a computer, a personal computer, a microcontroller, a microprocessor, a programmable logic device (PLD) such as a field-programmable gate array (FPGA), a programmable logic controller (PLC), and a logic circuit, for example.

[0177] Any combination of two or more of the first controller 16, the second controller 23, the third controller 35, and the fourth controller 43 may be integrated. In other words, the processing of one of the first controller 16, the second controller 23, the third controller 35, and the fourth controller 43 may be performed by another of the first controller 16, the second controller 23, the third controller 35, and the fourth controller 43. For example, the first controller 16 may control the entire operating table apparatus 30 instead of the third controller 35, in which case the third controller 35 may be omitted. For example, the first controller 16 may control the entire environmental information acquisition device 40 instead of the fourth controller 43, in which case the fourth controller 43 may be omitted. One or more processors and memories included in the first controller 16, the second controller 23, the third controller 35, the fourth controller 43, or any combination thereof are examples of processing circuits.

[0178] The first controller 16, the second controller 23, the third controller 35, the fourth controller 43, or any combination thereof may independently or in cooperation with each other execute an assistance process for assisting the operator U1 in operating the surgical assistance robot during surgery. The assistance process will be described in detail later.

[0179] [Surgical control] FIG. 21 is a flowchart illustrating the flow of control during surgery in the surgery assistance system 1.

[0180] 21 is started, first, members of the medical team including the operator U1 (hereinafter referred to as "users") prepare for surgery. For example, in preparation for surgery, the patient P's body position, the posture of the operating table, the insertion position of the medical instrument on the patient's body surface, the position of the surgical support robot relative to the operating table, and the posture of the arms included in the surgical support robot are set up so that the surgery can be performed.

[0181] Specifically, the user considers and determines the body position of the patient P and the posture of the operating table 31 according to the content of the surgery. After that, the user adjusts the operating table 31 so that the patient P assumes the determined posture, and has the patient P lie down on the operating table 31 so that the patient P assumes the determined body position.

[0182] The user also determines the positions of multiple ports on the body surface of the patient P and inserts a trocar into the determined positions. The trocar is a tubular member that is inserted into the body wall of the patient P to guide the medical instrument 15 into the body cavity. The multiple ports correspond to the multiple arms 14, respectively.

[0183] The user performs a so-called roll-in, which involves moving the surgical support robot 10 close to the operating table 31 so that it can assist with surgery. At this time, the position and posture of the arm base 13 relative to the patient P is adjusted. After that, depending on the position of the surgical support robot 10 relative to the patient P and the area to be operated on, such as the abdomen, the multiple arms 14 are deployed above the patient P so that the robot is in a predetermined preparation posture.

[0184] After the arm 14 reaches the preparation position, the pivot position is taught and the medical instrument is attached. The pivot position is a position that serves as a fulcrum when moving the medical instrument 15 during surgery. For example, the user U uses a pivot position teaching device to set the pivot position at or near the insertion position of the trocar on the body surface of the patient P. After teaching the pivot position, the user inserts the medical instrument 15 into the trocar, and then attaches the medical instrument 15, which is inserted into the trocar, to the arm 14. In this way, preparation for surgery is completed.

[0185] After preparations for surgery are made, the processor 161 of the first controller 16 acquires surgical environment information (step S21).

[0186] The surgical environment information in this embodiment is the same as the surgical environment information described in the first embodiment, except for whether it relates to elements in virtual space or elements in real space.

[0187] That is, the surgical environment information may include surgical content information indicating the content of the surgery. For example, the surgical content information may include information indicating the name of the disease, information indicating the surgical procedure, or a combination thereof.

[0188] The surgical environment information may include surgical preparation information related to preparation for surgery, such as information about the patient's position, information about the orientation of the operating table, information about the insertion position of a medical instrument on the patient's body surface, information about the position of a surgical assist robot relative to the operating table, information about the orientation of an arm included in the surgical assist robot, or any combination thereof.

[0189] The surgical environment information may include peripheral object information regarding peripheral objects located around the surgical support robot. The peripheral object information includes information indicating the positional relationship between the peripheral objects and the surgical support robot. The peripheral object information includes information indicating the positional relationship between the peripheral objects and elements of the surgical support robot, such as an arm. The peripheral objects may include an operating table and peripheral equipment. The peripheral objects may also include a patient. In other words, the peripheral object information may include patient information regarding the patient in the space.

[0190] The surgical environment information may include patient information about a patient undergoing surgery. The patient information may include, for example, body shape information about the patient's body shape. The patient information may include at least one of the patient's height, chest circumference, and abdominal circumference. The patient information may be three-dimensional information about the patient's body shape. For example, the patient information may include data obtained by 3D scanning a portion of the patient's body or the entire body. The patient information may include information about weight, body fat percentage, etc. The patient information may include information about the patient's age. The patient information may include information about the patient's gender.

[0191] In this embodiment, some of the surgical environment information is acquired by the environmental information acquisition device 40. For example, the environmental information acquisition device 40 sends three-dimensional information of objects in the operating room R to the first controller 16, and the processor 161 of the first controller 16 generates surgical preparation information, surrounding object information, and patient information based on the acquired three-dimensional information.

[0192] The methods for acquiring the surgery preparation information, peripheral object information, and patient information are not limited to these. For example, some of the surgery preparation information may be stored in advance in memory 162 before surgery preparation is performed. For example, the surgery preparation information may be stored in memory 162 in advance, and the user may proceed with preparations in the operating room as indicated by the surgery preparation information. For example, some of the surgical environment information may be calculated during surgery preparation or input by the user. The posture of the operating table 31 may be received from the operating table device 30.

[0193] After acquiring the surgical environment information, the surgery begins (step S22). That is, the processor 161 of the first controller 16 acquires image information indicating the endoscopic image captured by the endoscopic camera 151 and sends it to the remote control device 20, which then displays the received endoscopic image on the display 22. The processor 161 of the first controller 16 also controls the operation of the surgery support robot 10 in accordance with operations received by the operation input device 21. More specifically, when the operator U1 operates the operation input device 21, operation information, which is an operation command indicating the content of the operation, is sent to the first controller 16, and the first controller 16 operates the surgery support robot 10 based on the operation information. In this way, the operator U1 operates the operation input device 21 while viewing the endoscopic image, thereby proceeding with the surgery, changing the imaging range of the endoscopic camera 151 and moving the surgical instrument 152 to treat the affected area MV.

[0194] The processor 161 of the first controller 16 executes a real-time evaluation process during surgery (step S23).

[0195] The real-time evaluation process in step S23 is a process for evaluating the operation of the operator U1 in real time during surgery. In the real-time evaluation process during surgery, the processor 161 evaluates the operation of the operator U1, more specifically, the operating status of the surgical support robot 10 caused by the operation of the operator U1, based on the operation evaluation rules.

[0196] In this embodiment, the operation evaluation rule includes an interference evaluation rule regarding the arm 14, which moves due to the operation of the operator U1, and a predetermined object for interference determination. The processor 161 evaluates the possibility of interference between the arm 14 or the medical instrument 15, which moves due to the operation of the operator U1, and the predetermined object for interference determination, based on the interference evaluation rule. In this embodiment, the processor 161 quantifies the possibility of interference between the arm 14 or the medical instrument 15, which moves due to the operation of the operator U1, and the predetermined object for interference determination, based on the predetermined interference evaluation rule.

[0197] The objects to be determined for interference include the patient P and surrounding objects such as the operating table 31 around the arm 14. The objects to be determined for interference include the three arms 14 other than the arm 14 that moves due to the operation of the operator U1, and the medical instruments 15 attached to each of the other three arms 14. The objects to be determined for interference include areas inside the body of the patient P that are undesirable for the tool 152c of the surgical instrument 152 to come into contact with. The areas that are undesirable for contact with may be, for example, body tissues or organs unrelated to the affected area V.

[0198] Below are some examples of how to assess the possibility of interference.

[0199] For example, the processor 161 may calculate the distance between the arm 14 that moves due to the operation of the operator U1 and another arm, and calculate an index value corresponding to the possibility of interference so that the smaller the distance, the larger the index value corresponding to the possibility of interference.

[0200] For example, the processor 161 may determine whether or not an interference determination target exists in the movement direction of the arm 14 or the medical instrument 15 that moves due to the operation of the operator U1. Furthermore, when the processor 161 determines that an interference determination target exists in the movement direction of the arm 14 or the medical instrument 15, the processor 161 may calculate an index value corresponding to the possibility of interference such that the index value corresponding to the possibility of interference decreases as the movement speed of the arm 14 or the medical instrument 15 increases.

[0201] For example, when the processor 161 operates the arms 14 due to the operation of the operator U1 and then causes each of the arms 14 to perform a plurality of assumed next tentative actions, the processor 161 may calculate how many of the tentative actions will cause interference between the arms 14. The processor 161 calculates an index value corresponding to the possibility of interference such that the more interfering actions there are, the larger the index value corresponding to the possibility of interference becomes.

[0202] As shown in FIG. 21 , the processor 161 determines whether the evaluation result of the real-time evaluation process satisfies a predetermined condition (step S24). The predetermined condition is not particularly limited as long as it indicates that the degree of the possibility of interference, which is the evaluation result, exceeds an acceptable range. For example, the predetermined condition may include a condition that an index value corresponding to the possibility of interference exceeds a predetermined value as a result of operating the arm 14 due to the operation of the operator U1. For example, the predetermined condition may include a condition that an index value corresponding to the possibility of interference increases at a predetermined rate or more as a result of operating the arm 14 due to the operation of the operator U1.

[0203] When the processor 161 determines that the evaluation result of the real-time evaluation process satisfies a predetermined condition (step S24: Yes), it executes a support process to support the operation of the operator U1 during surgery (step S25). Note that determining whether the evaluation result of the real-time evaluation process satisfies a predetermined condition can also be one aspect of evaluating the operator's operation. A determination that the index value corresponding to the possibility of interference satisfies a predetermined condition can also be one aspect of the evaluation result.

[0204] The content of the support processing is processing according to the evaluation item. In this embodiment, since the evaluation item is the possibility of interference, the support processing may be processing that prompts the operator U1 to perform an operation that reduces the possibility of interference or an operation that suppresses the rate of increase in the possibility of interference. For example, the support processing may be processing that outputs the results of the evaluation. The support processing may be processing that controls a predetermined control object included in the surgery support system 1 based on the results of the evaluation.

[0205] The following describes examples of the support process.

[0206] In the assistance process, the processor 161 may control the arm 14 or the medical instrument 15 based on the interference possibility that is the evaluation result. Specifically, the processor 161 may control the arm 14 or the medical instrument 15 so as to reduce the interference possibility that is the evaluation result.

[0207] For example, in the assistance processing, the processor 161 may control the arm 14 itself, which is operated due to the operation of the operator U1, so as to assume a posture that reduces the possibility of interference, or alternatively, or in addition, may control the object to be determined as being subject to interference detection.

[0208] For example, if the target of interference determination is an arm 14 other than the arm 14 that operates due to the operation of the operator U1, in the assistance process, the processor 161 may control part or all of the arm 14 that operates due to the operation of the operator U1 and the other arm 14 that is the target of interference determination, so as to reduce the possibility of interference. In other words, in the assistance process, an arm 14 other than the arm 14 that operates due to the operation of the operator U1 may be operated.

[0209] For example, in the assistance process, the processor 161 may stop operating the arm 14 due to the operation of the operator U1 in order to stop an increase in the possibility of interference. For example, in the assistance process, the processor 161 may operate the arm 14 operated due to the operation of the operator U1 or the object to be detected for interference so that the distance between the arm 14 operated due to the operation of the operator U1 and the object to be detected for interference increases.

[0210] For example, the object to be controlled in the assistance process may be an element external to the surgical assistance robot 10.

[0211] For example, in the assistance processing, the processor 161 may generate a torque in the operating manipulator 211L or 211R to make it difficult to perform an operation that increases the possibility of interference. Specifically, the processor 161 may send a command to the second controller 23 to generate a torque that acts as an operation resistance in the operating manipulator 211L or 211R, in other words, a torque in the opposite direction to the operation direction, and the processor 231 of the second controller 23 may control the servo motor to apply the torque that acts as the operation resistance to the operating manipulator 211L or 211R based on the received command. The torque that acts as the operation resistance may be a torque of a preset value. The processor 161 may set the value of the torque that acts as the operation resistance so that the value increases as the likelihood of interference increases as a result of the evaluation.

[0212] For example, in the assistance processing, the processor 161 may cause the display 22 to display information indicating that the possibility of interference has exceeded the acceptable range. Specifically, the processor 161 may send information indicating that the possibility of interference has exceeded the acceptable range to the second controller 23, and the processor 231 of the second controller 23 may cause the display 22 to display information indicating that the possibility of interference has exceeded the acceptable range. The information indicating that the possibility of interference has exceeded the acceptable range may be displayed as a message such as "Interference may occur," or as an indicator indicating that the possibility of interference has exceeded the acceptable range.

[0213] For example, in the assistance process, the processor 161 may output information indicating that the possibility of interference has exceeded an acceptable range from the speaker 24. Specifically, the processor 161 may send information indicating that the possibility of interference has exceeded an acceptable range to the second controller 23, and the processor 231 of the second controller 23 may output audio information corresponding to the received information from the speaker 24. The audio information may be a voice such as "Interference may occur," or a warning sound.

[0214] The above support processes are merely examples. One or more of the support processes may be employed. Furthermore, other support processes may be employed instead of or in addition to the support processes described above.

[0215] The processor 161 determines whether the surgery has ended (step S26). For example, the processor 161 determines that the surgery has ended based on the user's operation input indicating that the surgery has ended.

[0216] If processor 161 determines that the surgery is not complete (step S26: No), the process returns to step S23. If processor 161 determines that the surgery is complete (step S26: Yes), the process ends control related to the surgery.

[0217] As described above, according to the surgery assistance system 1 of this embodiment, the processor 161 evaluates the interference possibility and executes the assistance process based on the interference possibility that is the evaluation result. By executing the assistance process, the operator U1 is prompted to perform an operation that reduces the interference possibility that is the evaluation result or an operation that suppresses the rate of increase in the interference possibility. Therefore, according to this embodiment, the operation of the surgery assistance robot by the operator U1 can be assisted from the viewpoint of reducing the interference possibility.

[0218] <Other embodiments> Although examples of embodiments of the present disclosure have been described above, the present disclosure is not limited to the above-described embodiments. That is, various modifications and improvements are possible within the scope of the present disclosure. For example, various modifications made to the embodiments and forms constructed by combining components of different embodiments are also included within the scope of the present disclosure.

[0219] The above embodiments have been described as examples of the technology disclosed in this application. However, the technology of the present disclosure is not limited to these and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made as appropriate. Furthermore, the components described in the above embodiments can be combined to create new embodiments. For example, some configurations or methods in one embodiment may be applied to another embodiment, and some configurations in an embodiment can be separated and arbitrarily extracted from other configurations in that embodiment. Furthermore, the components described in the accompanying drawings and detailed description include not only components essential for solving the problem, but also components that are not essential for solving the problem, and are used to illustrate the technology. Two blocks shown in order in a flowchart may be executed simultaneously or in reverse order, depending on the circumstances.

[0220] The functions of the elements disclosed herein can be performed using circuits or processing circuits, including general-purpose processors, special-purpose processors, integrated circuits, ASICs (Application Specific Integrated Circuits), conventional circuits, or any combination thereof, configured or programmed to perform the disclosed functions. A processor is considered a processing circuit or circuit because it includes transistors and other circuitry. In this disclosure, a circuit, unit, or means is hardware that performs the recited functions or hardware that is programmed to perform the recited functions. The hardware may be hardware disclosed herein or other known hardware that is programmed or configured to perform the recited functions. Where the hardware is a processor, which is considered a type of circuit, the circuit, means, or unit is a combination of hardware and software, and the software is used to configure the hardware or processor.

[0221] The configuration and shape of each element of the surgery assistance system are not limited to those described in the first and second embodiments. For example, the number of arms, the configuration of the base, and the number and arrangement of object detection sensors are not limited to those described in the above embodiments.

[0222] The remote control device 20 of the operation training system 100 of the first embodiment has the same configuration as the remote control device 20 of the surgery support system 1 used in actual surgery, but they may have different configurations. The remote control device 20 in the operation training system 100 may be configured to include a pair of manipulators having the same or similar configuration as the pair of left and right operation manipulators 211L and 211R, and a display that displays a virtual endoscopic image.

[0223] In the first and second embodiments, the operation information indicating the content of the operation received by the operating manipulator is exemplified by an operation command generated based on the amount of rotation detected by a rotation sensor, but the operation information is not limited to this. The operation information may be any information indicating the content of the operation received by the operating manipulator for operating the surgical assist robot or virtual surgical assist robot. The sensor included in the operating manipulator may be any sensor other than a rotation sensor as long as it detects the operation received by the operating manipulator.

[0224] Furthermore, the operation information indicating the content of the operation accepted by the operating manipulator may be information detected by a sensor external to the operating manipulator. For example, the operation information may be image information indicating the content of the operation accepted by the operating manipulator in addition to or instead of the above-mentioned operation command.

[0225] In this case, the image information may be an image of the operator's hand operating the operation manipulator. This type of image can be acquired from a camera located near the operation manipulator and communicatively connected to the circuit that executes the evaluation process. The camera may be supported by a part of the structure of the remote control device 20, or may be located separately from the remote control device 20.

[0226] The image information may also be an image captured of a surgical robot operating in response to an operator's operation received by an operating manipulator. This type of image can be acquired from a camera placed near the surgical robot and communicatively connected to a circuit that executes the evaluation process. The camera may be an object detection sensor of the environmental information acquisition device 40. The camera may be an endoscopic camera that captures images of a surgical instrument inside a patient's body.

[0227] The image information may also be an image of a virtual surgical robot operating in response to an operator's operation received by an operating manipulator. This type of image is captured by a virtual camera placed in the virtual space. The virtual camera may be positioned so that the entire field of view of the multiple virtual arms is included. The virtual camera may be a virtual endoscopic camera that captures an image of a virtual surgical instrument inside the body of a virtual patient.

[0228] Furthermore, the image information may be an endoscopic image captured by an endoscopic camera that operates in response to an operator's operation received by an operating manipulator, or an endoscopic image captured by a virtual endoscopic camera that operates in response to an operator's operation received by an operating manipulator. If the direction or field of view of the endoscopic camera or virtual endoscopic camera can be changed by the operator's operation, it is possible to understand from the endoscopic image how the operator is operating the endoscopic camera or virtual endoscopic camera. Surgical instruments or virtual surgical instruments do not necessarily need to be reflected in the endoscopic image.

[0229] The image information may include endoscopic image information captured by an endoscopic camera showing the operation of a surgical instrument in response to an operation received by the operating manipulator, or endoscopic image information captured by a virtual endoscopic camera showing the operation of a virtual surgical instrument in response to an operation received by the operating manipulator. The endoscopic image often shows the condition of the affected area. Therefore, when the processing circuit evaluates the operator's operating skill based on the operation information including such image information, it is easy to understand the condition of the surgical target and the type of operation the operator performed, thereby improving the accuracy of the evaluation of the operating skill.

[0230] In the first embodiment, the processing circuit may display the evaluation results of the real-time evaluation process on a display visible to the operator U1 during the virtual surgery. In the second embodiment, the processing circuit may display the evaluation results of the real-time evaluation process on a display visible to the operator U1 during the surgery.

[0231] Processing that is the same as or similar to the processing of the processing circuit described in one of the first and second embodiments can also be incorporated into the processing of the processing circuit described in the other of the first and second embodiments.

[0232] For example, the simulation of virtual surgery in the first embodiment may include processing such as step S25 described in the second embodiment. In other words, assistance processing similar to that in the second embodiment may be executed in the virtual surgery. In this case, the processing circuit may constantly execute assistance processing based on the evaluation results of the real-time evaluation processing during the virtual surgery, or, as in the second embodiment, may execute assistance processing only when it is determined that the evaluation results of the real-time evaluation processing satisfy a predetermined condition. In the first embodiment, the total evaluation processing may not be executed.

[0233] For example, after the surgery in the second embodiment, it is also possible to perform the total evaluation process of step S5 described in the first embodiment using operation information indicating the operation details of the surgeon from the start to the end of the surgery. In this case, the total evaluation process may be executed by a controller included in the surgery support robot or the remote operation device 20, or may be executed by a device other than the controller.

[0234] In the second embodiment, the processor 161 executes the assistance process when it is determined that the evaluation result of the real-time evaluation process satisfies a predetermined condition, but the processing circuit may execute the assistance process continuously during surgery. For example, if the evaluation result indicates that the possibility of interference is low, the torque value that becomes the operation resistance may be set to an extremely small value.

[0235] The evaluation items of the evaluation process described in the first and second embodiments are merely examples. For example, in the first embodiment, evaluation processes for evaluation items such as "ease of access," "possibility of interference between arms," ​​"deviation from standard operation," and "total surgery time" are shown, but in the first embodiment, some of these evaluation items may not be executed, or evaluation processes for evaluation items other than these may be executed.

[0236] Furthermore, in the second embodiment, the processor 161 evaluates the possibility of interference between the arm 14, which moves due to the operation of the operator U1, and a predetermined interference detection target. However, instead of or in addition to this evaluation, the processor 161 may evaluate other evaluation items during surgery. For example, in the second embodiment, the processor 161 may evaluate the operator's operation based on a time evaluation rule regarding the time required for each operation when the surgery is divided into multiple operation steps. For example, the evaluation based on the time evaluation rule may be based on the time required for each operation step. For example, the processor 161 may divide the ongoing surgery into multiple operation steps predetermined according to the operation content based on the acquired surgery content information, and count the elapsed time for each divided operation step. The multiple operation steps may be steps with different operation content, such as "resection of the lesion," "stopping bleeding," "extracting the resected lesion from the body," and "suturing the resected part."

[0237] The memory 162 may store target time information indicating a target time for each work step according to the surgical content, and the processor 161 may compare the time counted for each work step with the corresponding target time. The processor 161 may display the comparison result on a display visible to the operator U1 or the surgical assistant U2. The operator U1 or the surgical assistant can check whether the progress of the surgery is as planned by looking at the comparison result between the elapsed time for each work step and the target time.

[0238] As described in the first and second embodiments, the processing circuit of the assistance system may be configured to perform the following: outputting the evaluation result of the operator's operation based on a predetermined operation evaluation rule, controlling a predetermined control object based on the evaluation result, or outputting recommended operation information indicating the operation content recommended based on the evaluation result.

[0239] The evaluation results may be output using a communication device, a display, or a speaker. For example, the evaluation results may be transmitted to an information processing terminal such as a server or a personal computer via a communication network such as the Internet using a communication device and stored in the memory of the information processing terminal. For example, the evaluation results may be displayed on a display viewed by the operator, a display viewed by an administrator who manages the simulator that performs the virtual surgery simulation, or a display viewed by a surgical assistant.

[0240] The control object controlled based on the evaluation result may be various elements of the surgical robot. For example, the control object may be an arm or medical instrument operated in response to an operation command, an arm or medical instrument other than the arm or medical instrument operated in response to an operation command, or a combination thereof. The control object may also be an operating manipulator operated by an operator.

[0241] The recommended operation information, which indicates the operation content recommended based on the evaluation results, can be generated by machine learning using endoscopic image information showing endoscopic images from past surgeries and past operation information corresponding to the operation performed when the endoscopic images were captured, as training data, similar to the reference operation described in the above embodiment. The generation of the recommended operation is similar to the generation of the reference operation described in the above embodiment, and therefore a description thereof will be omitted. The processing circuit can output the recommended operation information using a display or speaker. For example, the recommended operation can be displayed on the endoscopic image viewed by the operator, such as on the display 22 of the remote control device, to provide operational assistance to the operator.

[0242] In the first embodiment, the reference information indicates the content of an operation recommended as an operation to be accepted by the operating manipulators 211L and 211R in the operating status of the virtual surgical support robot, but the reference information is not limited to this. The reference information may also be a discrimination criterion for determining whether or not the operation indicated by the operation information is an allowable operation. In this case, the comparison evaluation rule relates to whether or not the acquired operation information is allowable in the operating status of the surgical support robot or the virtual surgical support robot at the time the operation information is acquired.

[0243] For example, the evaluation process of operation information based on such comparative evaluation rules is performed using a trained model generated using a known machine learning algorithm. For example, if a problem occurs due to an operation accepted by the operating manipulators 211L and 211R in the operating situation of a certain surgical support robot in a past surgery, the operation can be determined to be an operation that is not permitted in that operating situation. In this way, training data is created by extracting operations that have caused problems in past surgeries and assigning a label indicating that the operation is not permitted to the combination of situation information indicating the operating situation of the virtual surgical support robot and operation information indicating the content of the operator's operation.

[0244] In the first and second embodiments, the processing circuit evaluates the operation of the operator U1 based on predetermined operation evaluation rules, but the processing circuit may also evaluate the preparation for the surgery or virtual surgery.

[0245] For example, the processor 511 of the first embodiment may acquire surgery preparation information related to preparations before performing a virtual surgery, such as information on the position of the virtual patient, information on the posture of the virtual operating table, information on the insertion position of a virtual medical instrument on the body surface of the virtual patient, information on the position of the virtual surgical assistant robot relative to the virtual operating table, information on the posture of a virtual arm included in the virtual surgical assistant robot, or any combination thereof. Furthermore, the processor 511 of the first embodiment may evaluate the surgery preparation based on predetermined preparation evaluation rules.

[0246] For example, the processor 161 of the second embodiment may acquire surgery preparation information related to preparations before surgery, such as information on the patient's position, information on the orientation of the operating table, information on the insertion position of a medical instrument on the patient's body surface, information on the position of a surgical assistant robot relative to the operating table, information on the orientation of an arm included in the surgical assistant robot, or any combination thereof. Furthermore, the processor 161 of the second embodiment may evaluate the surgery preparation information based on predetermined preparation evaluation rules.

[0247] The preparation evaluation rules may include evaluation rules similar to the operation evaluation rules. For example, the patient's position, the posture of the operating table, the insertion position of the medical instrument on the patient's body surface, the position of the surgical support robot relative to the operating table, and the posture of the arms included in the surgical support robot all significantly affect the ease of access of the arm 14 to the affected area and the possibility of interference with surrounding objects when the arm is moved. For this reason, the preparation evaluation rules preferably include the accessibility evaluation rules and interference evaluation rules described in the first and second embodiments.

[0248] In the first and second embodiments, the processing circuit of the assistance system evaluates the operation of the operator U1 based on a predetermined operation evaluation rule. However, the assistance system is not limited to this. That is, the processing circuit does not have to be configured to evaluate the operator's operation based on a predetermined operation evaluation rule. For example, the processing circuit may be configured to evaluate only the preparation for the surgery or virtual surgery described above, without evaluating the operator's operation.

[0249] That is, the support system is a support system used in surgery or virtual surgery performed by a surgical support robot that operates in response to an operator's operation or a virtual surgical support robot in a virtual space that simulates the surgical support robot, the support system comprises a processing circuit; The processing circuitry acquiring surgery preparation information related to preparation for a surgery to be performed using the surgical support robot or a virtual surgery to be performed using the virtual surgical support robot, the surgery preparation information including at least one selected from the group consisting of information on the position of a patient or a virtual patient, information on the posture of an operating table or a virtual operating table, information on an insertion position of a medical instrument on the body surface of a patient or a virtual patient, information on the position of a surgical support robot relative to the operating table or the position of a virtual surgical support robot relative to the virtual operating table, and information on the posture of an arm included in the surgical support robot or the posture of a virtual arm included in the virtual surgical support robot; The method may be configured to evaluate the preparation based on the surgical preparation information and predetermined preparation evaluation rules.

[0250] 13 to 15 are displayed on the display 53 of the simulation device 50, but may also be displayed on a display included in the remote control device 20. In this case, the user may input settings for surgical environment information via the operation input device 21, and the input information may be sent from the remote control device 20 to the simulation device 50.

[0251] The surgery assistance system 1 of the second embodiment does not necessarily have to include one or both of the operating table device and the environmental information acquisition device. For example, the operating table may be configured to be displaceable by human power.

[0252] The procedure for preparing for surgery is merely an example and does not limit the present disclosure. For example, after roll-in and before attaching the medical instrument to the arm, there may be no need to teach each arm. That is, in the second embodiment, the movement of the arm is restricted by teaching the pivot position, but the movement of the arm 14 may be restricted by another means, such as mechanically connecting the arm or the medical instrument 15 to a trocar.

[0253] The rotation sensor is not limited to an encoder, and may be any sensor that can detect the amount of rotation of a servo motor or the amount of rotation of a joint.

[0254] Although the support of the surgical support robot is configured as a cart that can move within the operating room, the support is not limited to this. The support of the surgical support robot may be configured to move within the operating room by being controlled by a first controller rather than being steered by the user U. For example, when the position of the surgical support robot 10 is predetermined in the simulation stage as in the second embodiment, the position information may be stored in memory, and movement control of the support of the surgical support robot may be performed based on the position information. Furthermore, the support of the surgical support robot may be fixed in the operating room so as to be immovable.

[0255] [Disclosure mode] Each of the following aspects is a disclosure of a preferred embodiment.

[0256] [Aspect 1] an operation manipulator having a plurality of joints that receives an operation from an operator; a processing circuit configured to control the operation of at least one medical instrument of a surgical assist robot, or the operation of at least one virtual medical instrument of a virtual surgical assist robot in a virtual space simulating the surgical assist robot, in accordance with an operation received by the operating manipulator; The processing circuitry acquiring operation information indicating the content of the operation accepted by the operating manipulator; and evaluating the operation skill of the operator indicated by the operation information based on predetermined operation evaluation rules.

[0257] [Aspect 2] The assistance system of aspect 1, wherein the processing circuit is further configured to output a result of the evaluation, control a predetermined control target based on the result of the evaluation, or output recommended operation information indicating an operation content recommended based on the result of the evaluation.

[0258] [Aspect 3] the manipulation manipulator includes a plurality of rotation sensors that detect rotation amounts at the plurality of joints, the operation information includes an operation command generated based on the amount of rotation detected by the rotation sensor, The assistance system of aspect 1 or 2, wherein the processing circuit evaluating the operator's operation skills includes the processing circuit evaluating the operator's operation skills based on the operation command and the operation evaluation rule.

[0259] [Aspect 4] the at least one medical instrument comprises an endoscopic camera and a surgical instrument, or the at least one virtual medical instrument comprises a virtual endoscopic camera and a virtual surgical instrument; The operation information is an image of the operator's hand operating the operation manipulator; an image of the surgical support robot that operates in response to an operation by the operator received by the operating manipulator; An image of the virtual surgery assist robot that operates in response to the operation of the operator received by the operating manipulator; an endoscopic image captured by the endoscopic camera, which operates in response to an operation by the operator received by the operating manipulator; and an endoscopic image captured by the virtual endoscopic camera, which operates in response to an operation by the operator received by the operation manipulator; and image information indicating at least one image selected from the group consisting of: An assistance system described in any one of aspects 1 to 3, wherein the processing circuit evaluating the operator's operating skills includes the processing circuit evaluating the operator's operating skills based on the image information and the operation evaluation rules.

[0260] [Aspect 5] The support system described in aspect 4, wherein the image information includes an image captured by the endoscopic camera of the surgical instrument that operates in response to the operation of the operator received by the operating manipulator, or endoscopic image information captured by the virtual endoscopic camera of the virtual surgical instrument that operates in response to the operation of the operator received by the operating manipulator.

[0261] [Aspect 6] The processing circuit is further configured to acquire peripheral object information regarding a peripheral object disposed around the surgical robot or a virtual peripheral object disposed around the virtual surgical robot; An assistance system described in any one of aspects 1 to 5, wherein the processing circuit evaluating the operator's operating skills includes the processing circuit evaluating the operator's operating skills indicated by the operation information based on the surrounding object information and predetermined operation evaluation rules.

[0262] [Aspect 7] The assistance system of aspect 6, wherein the surrounding object information includes patient information about a patient or a virtual patient in the virtual space.

[0263] [Aspect 8] An assistance system described in any of aspects 1 to 7, wherein the operation evaluation rule includes an accessibility evaluation rule regarding the ease of access of the surgical support robot or the virtual surgical support robot, which is operated due to the operation, to a specified surgical target.

[0264] [Aspect 9] the surgery support robot has a plurality of arms each having a plurality of degrees of freedom and each having a plurality of the medical instruments attached to a tip end thereof, or the virtual surgery support robot has a plurality of virtual arms simulating the plurality of arms; The assistance system according to any one of aspects 1 to 8, wherein the operation evaluation rule includes an interference evaluation rule regarding the possibility of interference between the plurality of arms or the plurality of virtual arms.

[0265] [Aspect 10] The support system described in aspects 1 to 9, wherein the operation evaluation rule includes a time evaluation rule regarding the surgical time required for surgery performed using the surgical support robot or virtual surgery performed using the virtual surgical support robot.

[0266] [Aspect 11] The processing circuit is further configured to acquire reference information indicating a reference for an operation accepted by the operating manipulator corresponding to an operating status of the surgical assist robot or the virtual surgical assist robot; 11. The assistance system according to any one of aspects 1 to 10, wherein the operation evaluation rule includes a comparison evaluation rule regarding a comparison between the operation information and the reference information.

[0267] [Aspect 12] The support system described in aspect 11, wherein the reference information is information generated based on past operation information indicating the content of operations received by the operating manipulator in a surgery previously performed using a surgical support robot or a virtual surgery performed using a virtual surgical support robot.

[0268] [Aspect 13] the reference information indicates the content of an operation recommended as an operation to be accepted by the operating manipulator in the operating situation of the surgical support robot or the virtual surgical support robot, The assistance system described in aspect 11 or 12, wherein the processing circuit is further configured to present to a user of the assistance system a comparison result between the operation corresponding to the operation information and the operation corresponding to the reference information, which is a result of the evaluation based on the comparison evaluation rule.

[0269] [Aspect 14] an operation manipulator having a plurality of joints that receives an operation from an operator; a surgical assistance robot including a plurality of medical instruments, a plurality of arms having a plurality of degrees of freedom and having the medical instruments attached to their distal ends, and a processing circuit configured to control the operation of the plurality of arms or the plurality of medical instruments in response to an operation by the operator received by the operating manipulator; The processing circuitry evaluating the possibility of interference between the arm or the medical instrument operated due to the operation of the operator and a predetermined interference determination target based on a predetermined interference evaluation rule; determining whether the evaluation result regarding the possibility of interference satisfies a predetermined condition; and When it is determined that the predetermined condition is satisfied, the assistance system is configured to present to the operator information indicating that the possibility of interference has exceeded an acceptable range.

[0270] [Aspect 15] A method for supporting improvement of an operator's operation skill with respect to an operation manipulator for operating a medical instrument possessed by a surgical support robot or a virtual medical instrument possessed by a virtual surgical support robot in a virtual space simulating the surgical support robot, comprising: controlling, by a processing circuit, the operation of the medical instrument or the operation of the virtual medical instrument in accordance with the operation of the operator received by the operating manipulator; acquiring, by the processing circuit, operation information indicating the content of the operation of the operator received by the operation manipulator; The assistance method includes evaluating, by the processing circuit, the operation skill of the operator indicated by the operation information based on predetermined operation evaluation rules. [Explanation of symbols]

[0271] 1: Surgical support system 10: Surgical support robot 11:Support 12: Positioner 13: Arm base 14: Arm 15: Medical equipment 151: Endoscope camera 152 :Surgical instruments 16: First controller 161: Processor 162: Memory 20: Remote control device 21: Operation input device 22: Display device 23: Second controller 231: Processor 232: Memory 30: Operating table equipment 31:Operating table 40:Environmental information acquisition device 41: Object detection sensor P:Patient

Claims

1. an operation manipulator having a plurality of joints that receives an operation from an operator; a processing circuit configured to control the operation of at least one medical instrument possessed by a surgical assist robot, or the operation of at least one virtual medical instrument possessed by a virtual surgical assist robot in a virtual space simulating the surgical assist robot, in accordance with an operation received by the operating manipulator; The processing circuitry acquiring operation information indicating the content of the operation accepted by the operating manipulator; and evaluating the operation skill of the operator corresponding to the operation information based on a predetermined operation evaluation rule.

2. 2. The assistance system according to claim 1, wherein the processing circuitry is further configured to output a result of the evaluation, control a predetermined control target based on the result of the evaluation, or output recommended operation information indicating an operation content recommended based on the result of the evaluation.

3. the manipulation manipulator includes a plurality of rotation sensors that detect rotation amounts at the plurality of joints, the operation information includes an operation command generated based on the amount of rotation detected by the rotation sensor, The assistance system according to claim 1 or 2, wherein the processing circuit evaluates the operation skill of the operator based on the operation command and the operation evaluation rule.

4. the at least one medical instrument comprises an endoscopic camera and a surgical instrument, or the at least one virtual medical instrument comprises a virtual endoscopic camera and a virtual surgical instrument; The operation information is an image of the operator's hand operating the operation manipulator; an image of the surgical support robot that operates in response to an operation by the operator received by the operating manipulator; An image of the virtual surgery assist robot that operates in response to the operation of the operator received by the operation manipulator; an endoscopic image captured by the endoscopic camera, which operates in response to an operation by the operator received by the operating manipulator; and an endoscopic image captured by the virtual endoscopic camera, which operates in response to an operation by the operator received by the operation manipulator; and image information indicating at least one image selected from the group consisting of: The assistance system according to claim 1 or 2, wherein the processing circuitry evaluating the operator's operation skills includes the processing circuitry evaluating the operator's operation skills based on the image information and the operation evaluation rules.

5. 5. The support system according to claim 4, wherein the image information includes an image captured by the endoscopic camera of the surgical instrument that operates in response to the operation of the operator received by the operating manipulator, or endoscopic image information captured by the virtual endoscopic camera of the virtual surgical instrument that operates in response to the operation of the operator received by the operating manipulator.

6. The processing circuit is further configured to acquire peripheral object information regarding a peripheral object disposed around the surgical robot or a virtual peripheral object disposed around the virtual surgical robot; 3. The assistance system according to claim 1, wherein the processing circuit evaluates the operation skill of the operator based on the surrounding object information and a predetermined operation evaluation rule.

7. The assistance system according to claim 6 , wherein the peripheral information includes patient information about a patient or a virtual patient in the virtual space.

8. The support system according to claim 1 or 2, wherein the operation evaluation rule includes an accessibility evaluation rule regarding the ease of access of the surgical support robot or the virtual surgical support robot, which is operated due to the operation, to a specified surgical target.

9. the surgery assist robot has a plurality of arms each having a plurality of degrees of freedom and each having a plurality of the medical instruments attached to a tip end thereof, or the virtual surgery assist robot has a plurality of virtual arms simulating the plurality of arms; The assistance system according to claim 1 or 2, wherein the operation evaluation rule includes an interference evaluation rule regarding the possibility of interference between the plurality of arms or the plurality of virtual arms.

10. The assistance system according to claim 1 or 2, wherein the operation evaluation rule includes a time evaluation rule regarding the surgical time required for a surgery performed using the surgical support robot or a virtual surgery performed using the virtual surgical support robot.

11. The processing circuit is further configured to acquire reference information indicating a reference for an operation accepted by the operating manipulator corresponding to an operating status of the surgical assist robot or the virtual surgical assist robot; The assistance system according to claim 1 , wherein the operation evaluation rule includes a comparison evaluation rule regarding a comparison between the operation information and the reference information.

12. The support system described in claim 11, wherein the reference information is information generated based on past operation information indicating the content of operations received by an operating manipulator in a surgery previously performed using a surgical support robot or a virtual surgery previously performed using a virtual surgical support robot.

13. the reference information indicates the content of an operation recommended as an operation to be accepted by the operating manipulator in the operating situation of the surgical support robot or the virtual surgical support robot, 12. The assistance system according to claim 11, wherein the processing circuitry is further configured to present to a user of the assistance system a comparison result between the operation corresponding to the operation information and the operation corresponding to the reference information, the comparison result being a result of evaluation based on the comparison evaluation rule.

14. an operation manipulator having a plurality of joints that receives an operation from an operator; a surgical assistance robot including a plurality of medical instruments, a plurality of arms having a plurality of degrees of freedom and having the medical instruments attached to their distal ends, and a processing circuit configured to control the operation of the plurality of arms or the plurality of medical instruments in response to an operation by the operator received by the operating manipulator; The processing circuitry evaluating the possibility of interference between the arm or the medical instrument operated due to the operation of the operator and a predetermined interference determination target based on a predetermined interference evaluation rule; determining whether the evaluation result regarding the possibility of interference satisfies a predetermined condition; and When it is determined that the predetermined condition is satisfied, the assistance system is configured to present to the operator information indicating that the possibility of interference has exceeded an acceptable range.

15. A method for supporting improvement of an operator's operation skill with respect to an operation manipulator for operating a medical instrument possessed by a surgical support robot or a virtual medical instrument possessed by a virtual surgical support robot in a virtual space simulating the surgical support robot, comprising: controlling, by a processing circuit, the operation of the medical instrument or the operation of the virtual medical instrument in accordance with the operation of the operator received by the operating manipulator; acquiring, by the processing circuit, operation information indicating the content of the operation of the operator received by the operation manipulator; The support method includes evaluating, by the processing circuit, the operation skill of the operator corresponding to the operation information based on a predetermined operation evaluation rule.

Images