Surgical assistance system, method for controlling surgical assistance system, and program

Abstract

A surgical assistance system (500) includes: a marker imaging unit (70) configured to capture an image of a marker member (M) which is detachably attached to at least one of a patient (P) and a trocar (T) inserted from the body surface (S) of the patient (P); and a control unit (350) configured to detect the marker member (M) on the basis of the captured image of the marker member (M) and set a pivot position (PP) serving as a fulcrum for the movement of a surgical instrument (1) on the basis of the detected marker member (M).

Description

Surgery support system, control method and program for surgery support system

[0001] The present disclosure relates to a surgery assistance system, a control method for a surgery assistance system, and a program.

[0002] Conventionally, surgical assistance systems including a robotic arm to which a surgical instrument is attached have been known. The surgical assistance system disclosed in Japanese Patent No. 6290372 includes a robotic arm to which a surgical instrument is attached and a processing device that controls the operation of the robotic arm. Furthermore, in Japanese Patent No. 6290372, a surgical instrument is inserted into a patient's body through a port disposed on the patient's body surface. Furthermore, the surgical assistance system disclosed in Japanese Patent No. 6290372 also includes an imaging system that images the port. Images of the port captured by the imaging system are input to the processing device. Then, based on the images of the port captured by the imaging system, the processing device defines the coordinates of the port as a fulcrum for movement of the surgical instrument. As a result, the surgical instrument is moved using the position of the port as a fulcrum.

[0003] Patent No. 6290372

[0004] However, in Japanese Patent No. 6290372, the port's coordinates are defined as the fulcrum for the movement of the surgical instrument based on an image of the port captured by an imaging system. In this case, it is conceivable that the port may be in a blind spot for other medical devices, and the imaging system may not be able to capture the port. In this case, it becomes impossible to set the fulcrum for the movement of the surgical instrument. Therefore, it is desirable to be able to reliably set the fulcrum for the movement of the surgical instrument.

[0005] The present disclosure provides a surgery assistance system, a control method for a surgery assistance system, and a program that can reliably set a pivot position that serves as a fulcrum for the movement of a surgical instrument.

[0006] A surgical support system according to a first aspect of the present disclosure includes a surgical support robot including a robotic arm to which a surgical instrument is attached; a marker imaging unit that images a marker member that is detachably attached to at least one of a patient on whom surgery is to be performed using the surgical support robot and a trocar inserted into the surface of the patient's body; and a control unit that detects the marker member based on an image of the marker member, and sets a pivot position that serves as a fulcrum for movement of the surgical instrument based on the detected marker member.

[0007] A surgical assistance system according to a first aspect of the present disclosure includes a marker imaging unit that images a marker member detachably attached to at least one of a patient and a trocar, and a control unit that detects the marker member based on the captured image of the marker member and sets a pivot position that serves as a fulcrum for movement of a surgical instrument based on the detected marker member. Because the marker member is detachable from the patient and the trocar, the marker member can be attached to a position that does not expose blind spots of other medical devices, etc. Therefore, the marker imaging unit can reliably image the marker member, and therefore the pivot position that serves as a fulcrum for movement of the surgical instrument can be reliably set based on the marker member detected from the captured image.

[0008] A control method for a surgical assistance system according to a second aspect of the present disclosure includes capturing an image of a marker member detachably attached to at least one of a patient on whom surgery is to be performed using a surgical assistance robot including a robotic arm to which a surgical instrument is attached and a trocar inserted into the surface of the patient's body, detecting the marker member based on the captured image of the marker member, and setting, by a control unit, a pivot position that serves as a fulcrum for movement of the surgical instrument based on the detected marker member.

[0009] A control method for a surgical assistance system according to a second aspect of the present disclosure includes capturing an image of a marker member detachably attached to at least one of a patient and a trocar, detecting the marker member based on the captured image of the marker member, and having a control unit set a pivot position serving as a fulcrum for movement of a surgical instrument based on the detected marker member. Because the marker member is detachable from the patient and the trocar, the marker member can be attached to a position that does not expose other medical devices or other blind spots. Therefore, the marker capturing unit can reliably capture an image of the marker member, thereby providing a control method for a surgical assistance system that can reliably set a pivot position serving as a fulcrum for movement of a surgical instrument based on the marker member detected from the captured image.

[0010] A program according to a third aspect of the present disclosure performs the following operations: capturing an image of a marker member detachably attached to at least one of a patient on whom surgery is to be performed using a surgical support robot including a robotic arm to which a surgical instrument is attached and a trocar inserted into the surface of the patient's body; detecting the marker member based on the captured image of the marker member; and setting a pivot position that serves as a fulcrum for movement of the surgical instrument based on the detected marker member.

[0011] A program according to a third aspect of the present disclosure includes imaging a marker member detachably attached to at least one of a patient and a trocar, detecting the marker member based on the captured image of the marker member, and having a control unit set a pivot position serving as a fulcrum for movement of a surgical instrument based on the detected marker member. Because the marker member is detachable from the patient and the trocar, the marker member can be attached to a position that does not expose other medical devices or other blind spots. Therefore, the marker imaging unit can reliably image the marker member, thereby providing a program that can reliably set a pivot position serving as a fulcrum for movement of a surgical instrument based on the marker member detected from the captured image.

[0012] According to the present disclosure, a pivot position that serves as a fulcrum for the movement of a surgical instrument can be reliably set.

[0013] 1 is a diagram illustrating the configuration of a surgery support system according to a first embodiment. FIG. 2 is a diagram illustrating the display unit of a medical cart according to the first embodiment. FIG. 3 is a diagram illustrating the configuration of a medical cart according to the first embodiment. FIG. 4 is a diagram illustrating the configuration of a robot arm according to the first embodiment. FIG. 5 is a diagram illustrating instruments. FIG. 6 is a perspective view illustrating the configuration of an arm operation unit according to the first embodiment. FIG. 7 is a diagram illustrating translational movement of a robot arm. FIG. 8 is a diagram illustrating rotational movement of a robot arm. FIG. 9 is a diagram illustrating an endoscope. FIG. 10 is a diagram illustrating a pivot position setting instrument. FIG. 11 is a diagram illustrating an operation unit according to the first embodiment. FIG. 12 is a diagram illustrating a wrist unit for a right hand according to the first embodiment. FIG. 13 is a diagram illustrating a wrist unit for a left hand according to the first embodiment. FIG. 14 is a perspective view illustrating a foot pedal according to the first embodiment. FIG. 15 is a control block diagram of a surgery support system according to the first embodiment. FIG. 16 is a control block diagram of a robot arm according to the first embodiment. FIG. 17 is a control block diagram of a positioner and a medical cart according to the first embodiment. FIG. 18 is a control block diagram of an operation unit according to the first embodiment. FIG. 19 is a diagram illustrating setting of a pivot position using a marker member affixed to a patient. FIG. 19 is a diagram illustrating setting of a pivot position using a marker member affixed to a trocar. FIG. 19 is a diagram illustrating an AR marker as a marker member. FIG. 20 is a diagram illustrating a state in which a marker member is imaged by a marker imaging unit arranged on an arm base. FIG. 1 is a diagram showing a patient, a trocar, and a marker member displayed on a display unit. FIG. 2 is a diagram for explaining a method for setting a pivot position from a marker member affixed to a patient. FIG. 3 is a diagram showing a pivot position set on the abdominal wall of a patient. FIG. 4 is a flow chart for explaining a control method of the surgery assistance system according to the first embodiment. FIG. 5 is a diagram showing a marker imaging unit arranged on a robot arm. FIG. 6 is a diagram showing two marker members arranged for one robot arm. FIG. 7 is a diagram showing a marker imaging unit arranged on an operating table. FIG. 8 is a diagram showing an endoscope as a marker imaging unit.

[0014] [First embodiment] (Configuration of surgery support system) The following describes the configuration of a surgery support system 500 according to the first embodiment. As shown in Fig. 1 , the surgery support system 500 includes a surgery support robot 100, a remote control device 200, a vision unit 300, and an image processing unit 400.

[0015] In this specification, as shown in Figure 4, the longitudinal direction of the surgical instrument 1 is defined as the Z direction. The distal end side of the surgical instrument 1 is defined as the Z1 side, and the proximal end side of the surgical instrument 1 is defined as the Z2 side. The direction perpendicular to the Z direction is defined as the X direction. The direction perpendicular to the Z direction and the X direction is defined as the Y direction.

[0016] As shown in FIG. 1 , the surgical support robot 100 is placed in an operating room. The remote control device 200 is placed at a location separated from the surgical support robot 100. The remote control device 200 receives operations on a surgical instrument 1. Specifically, an operator such as a doctor inputs commands to the remote control device 200 to cause the surgical support robot 100 to perform a desired operation. The remote control device 200 transmits the input commands to the surgical support robot 100. The surgical support robot 100 operates based on the received commands. The surgical support robot 100 is placed in an operating room, which is a sterilized sterile field.

[0017] (Configuration of surgical support robot) As shown in Figure 1, the surgical support robot 100 comprises a medical cart 10, a cart positioner operating unit 20, a positioner 30, an arm base 40, a plurality of robot arms 50, and an arm operating unit 60 provided on each robot arm 50.

[0018] 3 , the cart positioner operating unit 20 is supported by a cart positioner operating support unit 21 at the rear of the medical cart 10, and the medical cart 10 or the positioner 30 is moved by operating the cart positioner operating unit 20. The cart positioner operating unit 20 includes an input device 22 and an operating handle 23. The input device 22 receives operations to move and change the posture of the positioner 30, the arm base 40, and the multiple robot arms 50, mainly for preparing for surgery before the procedure.

[0019] As shown in Fig. 3, the input device 22 includes a display unit 22a, a joystick 22b, an enable switch 22c, an error reset button 22d, and a speaker 22e. The display unit 22a is, for example, a liquid crystal panel. As shown in Fig. 2, the display unit 22a displays numbers corresponding to the multiple robot arms 50. The display unit 22a also displays the type of surgical instrument 1 attached to each of the multiple robot arms 50. A check mark CM indicating that a pivot position PP, which will be described later, has been set is displayed on the display unit 22a.

[0020] 3, the joystick 22b is disposed near the display unit 22a of the input device 22. By selecting an operation mode displayed on the display unit 22a and operating the joystick 22b, the positioner 30 is moved three-dimensionally.

[0021] The enable switch 22c is disposed near the joystick 22b. The enable switch 22c permits or prohibits movement of the positioner 30. When the enable switch 22c is pressed down to permit movement of the positioner 30, the positioner 30 is moved by operating the joystick 22b.

[0022] The error reset button 22d resets an error in the surgery assistance system 500. The error may be, for example, a deviation abnormality error. The speaker 22e is provided as a pair. The pair of speakers 22e is provided near the location of the positioner 30 on the medical cart 10.

[0023] The operating handle 23 is disposed near the display unit 22a. The operating handle 23 has a throttle 23a that is gripped and rotated by an operator such as a nurse or technician to control the movement of the medical cart 10. Specifically, the operating handle 23 is disposed below the input device 22. The medical cart 10 moves forward when the throttle 23a is rotated from the front side to the back side. The medical cart 10 moves backward when the throttle 23a is rotated from the back side to the front side. The speed of the medical cart 10 changes depending on the amount of rotation of the throttle 23a. The operating handle 23 is configured to be rotatable left and right, indicated by the R direction, and the medical cart 10 rotates as the operating handle 23 is rotated.

[0024] An enable switch 23b that permits or prohibits movement of the medical cart 10 is disposed on the operating handle 23. When the enable switch 23b is pressed down to permit movement of the medical cart 10, the medical cart 10 is moved by operating the throttle 23a of the operating handle 23.

[0025] 1, the positioner 30 is, for example, a seven-axis articulated robot. The positioner 30 is placed on the medical cart 10. The positioner 30 adjusts the position of the arm base 40. The positioner 30 moves the position of the arm base 40 three-dimensionally.

[0026] The positioner 30 includes a base portion 31 and a plurality of link portions 32 connected to the base portion 31. The plurality of link portions 32 are connected to each other by joints 33.

[0027] The arm base 40 is attached to the tip of the positioner 30. The base ends of the multiple robot arms 50 are attached to the arm base 40. The multiple robot arms 50 can be folded for storage. The arm base 40 and the multiple robot arms 50 are covered with a sterile drape when in use. The robot arms 50 support the surgical instrument 1. A marker imaging unit 70, which will be described later, is disposed on the arm base 40. The marker imaging unit 70 is an example of an imaging unit for alignment.

[0028] 15 are arranged on the arm base 40. The status indicator 41 and the arm status indicator 42 are shown in FIG. 15. The status indicator 41 displays the status of the surgery assistance system 500. The arm status indicator 42 displays the status of the robot arm 50.

[0029] A plurality of robot arms 50 are provided. Specifically, four robot arms 50a, 50b, 50c, and 50d are provided. The robot arms 50a, 50b, 50c, and 50d have the same configuration.

[0030] As shown in FIG. 4 , the robot arm 50 includes an arm section 51, a first link section 52, a second link section 53, and a translational movement mechanism section 54. The robot arm 50 has joints JT1, JT2, JT3, JT4, JT5, JT6, JT7, and JT8. The joints JT1, JT2, JT3, JT4, JT5, JT6, and JT7 have A1, A2, A3, A4, A5, A6, and A7 axes as rotation axes, respectively. The joint JT8 has an A8 axis as a linear motion axis. The arm section 51 includes a base section 51 a and a link section 51 b.

[0031] The arm unit 51 is a seven-axis articulated robot arm. The first link unit 52 is disposed at the tip of the arm unit 51. The arm operating unit 60, which will be described later, is attached to the second link unit 53. The translational movement mechanism 54 is disposed between the first link unit 52 and the second link unit 53. A holder 55 for holding the surgical instrument 1 is disposed on the second link unit 53. The translational movement mechanism 54 translates the holder 55, to which the surgical instrument 1 is attached, between a first position and a second position. The first position is the end position on the Z2 side of the range of movement of the holder 55 by the translational movement mechanism 54 along the A8 axis. The second position is the end position on the Z1 side of the range of movement of the holder 55 by the translational movement mechanism 54 along the A8 axis.

[0032] A surgical instrument 1 is attached to the tip of each of the multiple robot arms 50. The surgical instrument 1 includes, for example, an interchangeable instrument 2, an endoscope 3 shown in FIG. 9 for capturing an image of the surgical site, and a pivot position setting instrument 4 shown in FIG. 10 for setting a pivot position PP. The instrument 2 includes a driven unit 2a, an end effector 2b, a wrist joint 2c shown in FIG. 5, and a shaft 2d. The end effector 2b is connected to the tip of the shaft 2d via the wrist joint 2c.

[0033] 1, an endoscope 3 is attached to the tip of one of the multiple robot arms 50, for example, robot arm 50c, and instruments 2 are attached to the tips of the remaining robot arms 50a, 50b, and 50d. Of the four robot arms 50 arranged adjacent to each other, it is desirable that the endoscope 3 be attached to one of the two robot arms 50b and 50c arranged in the middle.

[0034] 5, an end effector 2b having, for example, jaw members 2g and 2h is attached to the tip of the instrument 2. As the end effector 2b, scissors, graspers, needle holders, microdissectors, stable appliers, tackers, suction and cleaning tools, snare wires, clip appliers, and the like can be used.

[0035] The instrument 2 includes a first support member 2e and a second support member 2f. The first support member 2e is attached to a shaft 2d. The second support member 2f is supported by the first support member 2e so as to be rotatable about the A10 axis, and supports the end effector 2b so as to be rotatable about the A11 axis that intersects with the A10 axis. The shaft 2d rotates about the A9 axis. The wrist joint 2c is provided between the second support member 2f and the first support member 2e, with the A10 axis as its rotation axis.

[0036] 6 , the arm operating unit 60 is attached to the robot arm 50 and operates the robot arm 50. Specifically, the arm operating unit 60 is attached to the second link unit 53.

[0037] The arm operating unit 60 includes an enable switch 61 , a joystick 62 , a linear switch 63 , a mode switching button 64 , a mode indicator 65 , a pivot button 66 , and an adjustment button 67 .

[0038] When the enable switch 61 is pressed, it allows or disallows movement of the robot arm 50 using the joystick 62 and the linear switch 63. When the enable switch 61 is pressed while the arm operation unit 60 is being held by an operator such as a nurse or assistant, movement of the surgical instrument 1 by the robot arm 50 is permitted.

[0039] The joystick 62 is an operating tool for controlling the movement of the surgical instrument 1 by the robot arm 50. The joystick 62 controls the movement direction and movement speed of the robot arm 50. The robot arm 50 moves according to the direction and angle at which the joystick 62 is tilted.

[0040] The linear switch 63 is a switch for moving the surgical instrument 1 in the Z direction, which is the longitudinal direction of the surgical instrument 1. The linear switch 63 includes a linear switch 63a for moving the surgical instrument 1 in the direction of inserting it into the patient P, and a linear switch 63b for moving the surgical instrument 1 in the direction away from the patient P. Both the linear switch 63a and the linear switch 63b are push button switches.

[0041] The mode switching button 64 is a push button switch for switching between a mode for translating the surgical instrument 1 and a mode for rotating the surgical instrument 1. As shown in FIG. 7 , in the mode for translating the robot arm 50, the robot arm 50 is moved so that the tip 1a of the surgical instrument 1 moves on the X-Y plane. As shown in FIG. 8 , in the mode for rotating the robot arm 50, when the pivot position PP is not stored in the memory unit 351, the robot arm 50 is rotated around the center of the end effector 2b of the instrument 2 (as the surgical instrument 1) on the A11 axis or the tip of the end effector 2b as a fulcrum. When the pivot position PP is stored in the memory unit 351, the robot arm 50 is moved so that the surgical instrument 1 is rotated around the pivot position PP as a fulcrum. Note that the surgical instrument 1 is rotated with the shaft 1c of the surgical instrument 1 inserted into the trocar T. The mode switching button 64 is located on the Z-direction surface of the arm operating unit 60.

[0042] The mode indicator 65 displays the switched mode. When the mode indicator 65 is lit, it indicates the rotational movement mode, and when it is off, it indicates the translational movement mode. The mode indicator 65 also serves as a pivot position indicator that indicates that the pivot position PP has been set. The mode indicator 65 is located on the surface of the arm operation unit 60 facing in the Z direction.

[0043] The pivot button 66 is a push button switch for manually setting a pivot position PP, which serves as a fulcrum for the movement of the surgical instrument 1 attached to the robot arm 50, without using a marker member M, which will be described later.

[0044] The adjustment button 67 is a button for optimizing the position of the robot arm 50. After setting the pivot position PP for the robot arm 50 to which the endoscope 3 is attached, pressing the adjustment button 67 optimizes the positions of the other robot arms 50 and the arm base 40. The adjustment button 67 is a button different from the enable switch 61.

[0045] 1, the remote control device 200 is placed, for example, inside or outside an operating room. The remote control device 200 includes an operation unit 110, a foot pedal 120, a touch panel 130, a monitor 140, a support arm 150, a support bar 160, and an error reset button 161. The operation unit 110 constitutes an operation handle that allows an operator, such as a doctor, to input commands.

[0046] (Operation Unit) As shown in FIG. 11 , the operation unit 110 is a handle for operating the surgical instrument 1. The operation unit 110 also receives operations on the surgical instrument 1. When viewed from an operator such as a doctor, the operation unit 110 includes an operation unit 110L located on the left side and operated with the operator's left hand, and an operation unit 110R located on the right side and operated with the operator's right hand. The operation unit 110 includes an arm unit 111 and a wrist unit 112. The operation unit 110R includes an arm unit 111R and a wrist unit 112R. The operation unit 110L also includes an arm unit 111L and a wrist unit 112L.

[0047] The arm 111 has joints JT21, JT22, and JT23 shown in Figure 11, and JT24, JT25, JT26, and JT27 shown in Figures 12 and 13. The rotation axes of the joints JT21, JT22, JT23, JT24, JT25, JT26, and JT27 are defined as A21, A22, A23, A24, A25, A26, and A27 axes, respectively.

[0048] (Arm Unit) As shown in FIG. 11 , the arm unit 111R has link units 111a, 111b, and 111c. The upper end of link unit 111a is attached to the remote control device 200 so as to be rotatable around axis A21, which is aligned in the vertical direction. The upper end of link unit 111b is attached to the lower end of link unit 111a so as to be rotatable around axis A22, which is aligned in the horizontal direction. One end of link unit 111c is attached to the lower end of link unit 111b so as to be rotatable around axis A23, which is aligned in the horizontal direction. The wrist unit 112 is attached to the other end of link unit 111c so as to be rotatable around axis A24. Link unit 111a is connected to the remote control device 200 by joint JT21. Link units 111a and 111b are connected by joint JT22. The link portion 111b and the link portion 111c are connected by a joint JT23. The arm portion 111 supports a wrist portion 112. The arm portion 111L has a similar configuration to the arm portion 111R.

[0049] Wrist unit 112 includes wrist unit 112R, which is operated by the operator's right hand as shown in Fig. 12, and wrist unit 112L, which is operated by the operator's left hand as shown in Fig. 13. Fig. 12 shows the reference position of operation unit 110R, and Fig. 13 shows the reference position of operation unit 110L. Wrist unit 112R and wrist unit 112L have the same configuration.

[0050] The wrist unit 112 includes link units 112a, 112b, 112c, and a grip support member 112d that is operated by an operator such as a doctor. The base end of link unit 112a is connected to the tip end of the arm unit 111 and rotates around the A24 axis. The base end of link unit 112b is connected to the tip end of link unit 112a and rotates around the A25 axis. The base end of link unit 112c is connected to the tip end of link unit 112b and the grip support member 112d is connected to the tip end of link unit 112a and rotates around the A26 axis relative to link unit 112b. The grip support member 112d rotates around the A27 axis relative to link unit 112c. Each of link units 112a, 112b, and 112c has an L-shape.

[0051] The wrist section 112 includes a pair of grip members 112e that can be opened and closed by the operator. The grip members 112e are formed of elongated, plate-like lever members, and the proximal ends of each of the pair of grip members 112e are rotatably connected to the proximal end of the grip support member 112d. Cylindrical finger insertion sections 112f are disposed on the grip members 112e. The operator inserts their fingers into the pair of finger insertion sections 112f to operate the wrist section 112. The base ends of each of the pair of grip members 112e are connected to the grip support member 112d, and the opening angle between the jaw members 2g and 2h is changed by increasing or decreasing the angle between the pair of grip members 112e. A magnet is disposed on one of the grip members 112e, and a Hall sensor is disposed on the grip support member 112d. When the operator opens or closes the grip members 112e, the magnet and Hall sensor function as an angle detection sensor, and the Hall sensor outputs the opening angle. As an angle detection sensor, a Hall sensor may be disposed on the grip member 112e and a magnet may be disposed on the grip support member 112d. Alternatively, a magnet or a Hall sensor may be disposed on both of the grip members 112e.

[0052] As shown in FIG. 1 , the monitor 140 is a scope-type display device for displaying an image captured by the endoscope 3. The monitor 140 also has an alarm unit 141. The alarm unit 141 issues an error sound. The support arm 150 supports the monitor 140 so that the height of the monitor 140 is at the same height as the face of an operator such as a doctor. The touch panel 130 is mounted on a support bar 160. The surgical support robot 100 can be operated by the remote control device 200 by detecting the operator's head with a sensor provided near the monitor 140. The operator operates the operation unit 110 and the foot pedal 120 while visually checking the affected area on the monitor 140. This inputs commands to the remote control device 200. The commands input to the remote control device 200 are transmitted to the surgical support robot 100.

[0053] 14 , a plurality of foot pedals 120 are provided to perform functions related to the surgical instrument 1. The plurality of foot pedals 120 are arranged on a base 121. The foot pedals 120 include a switching pedal 122, a clutch pedal 123, a camera pedal 124, an incision pedal 125, a coagulation pedal 126, and a foot detector 127. The switching pedal 122, the clutch pedal 123, the camera pedal 124, the incision pedal 125, and the coagulation pedal 126 are operated by the operator's feet. The incision pedals 125 include an incision pedal 125R for the right robot arm 50 and an incision pedal 125L for the left robot arm 50. The coagulation pedals 126 include a coagulation pedal 126R for the right robot arm 50 and a coagulation pedal 126L for the left robot arm 50.

[0054] The switching pedal 122 switches the robot arm 50 operated by the operation unit 110. The clutch pedal 123 performs a clutch operation that temporarily disconnects the operational connection between the robot arm 50 and the operation unit 110. While the clutch pedal 123 is depressed by the operator, operations by the operation unit 110 are not transmitted to the robot arm 50. Furthermore, while the operator is depressing the camera pedal 124, the operation unit 110 can be used to operate the robot arm 50 to which the endoscope 3 is attached. While the operator is depressing the incision pedal 125 or the coagulation pedal 126, the electrosurgical device is activated.

[0055] (Vision Unit and Image Processing Unit) As shown in Fig. 1, the vision unit 300 and the image processing unit 400 are placed on a cart 210. The image processing unit 400 processes images captured by the endoscope 3. A display unit 220 is disposed on the cart 210. The image captured by the endoscope 3 is displayed on the display unit 220.

[0056] (Configuration of Control System) As shown in Fig. 15, the surgery assistance system 500 includes a first control device 310, an arm control device 320, a positioner control device 330, an operation control device 340, and a second control device 350. The surgery assistance system 500 also includes a storage unit 311 connected to the first control device 310 and a storage unit 351 connected to the second control device 350. The second control device 350 is an example of a control unit. The storage unit 311 stores a program 311a executed by the first control device 310. The storage unit 351 stores a program 351a executed by the second control device 350.

[0057] The first control device 310 is disposed inside the medical cart 10 so as to communicate with the arm control device 320 and the positioner control device 330, and controls the entire surgery assistance system 500. Specifically, the first control device 310 communicates with and controls each of the arm control device 320, the positioner control device 330, and the operation control device 340. The first control device 310 is connected to the arm control device 320, the positioner control device 330, and the operation control device 340 via a LAN or the like. The first control device 310 is disposed inside the medical cart 10.

[0058] An arm control unit 320 is provided for each of the plurality of robot arms 50. That is, a plurality of arm control units 320 corresponding to the number of the plurality of robot arms 50 are provided inside the medical cart 10.

[0059] As shown in Fig. 15 , the input device 22 is connected to the first control device 310 via a LAN or the like. The status indicator 41, arm status indicator 42, operating handle 23, throttle 23a, joystick 22b, and positioner control section 330 are serially connected via wiring 360 over a communication network that allows them to share information with each other. Note that Fig. 15 shows the status indicator 41, arm status indicator 42, and the like as if they were all connected to one wiring 360, but in reality, a wiring 360 is provided for each of the status indicator 41, arm status indicator 42, operating handle 23, throttle 23a, joystick 22b, stabilizer 24, and electric cylinder 25.

[0060] As shown in FIG. 16 , the arm unit 51 is provided with a plurality of servo motors SM1, an encoder EN1, and a reducer corresponding to each of the joints JT1, JT2, JT3, JT4, JT5, JT6, and JT7. The encoder EN1 detects the rotation angle of the servo motor SM1. The reducer decelerates the rotation of the servo motor SM1 to increase the torque. Inside the medical cart 10, a servo control unit SC1 for controlling the servo motor SM1 is disposed adjacent to the arm control unit 320. The servo control unit SC1 is electrically connected to the encoder EN1 for detecting the rotation angle of the servo motor SM1.

[0061] The second link section 53 is provided with a servo motor SM2, an encoder EN2, and a reducer for rotating a driven member disposed in the driven unit 2a of the surgical instrument 1. The encoder EN2 detects the rotation angle of the servo motor SM2. The reducer decelerates the rotation of the servo motor SM2 to increase the torque. The medical cart 10 is also provided with a servo control unit SC2 for controlling the servo motor SM2 that drives the surgical instrument 1. The servo control unit SC2 is electrically connected to an encoder EN2 for detecting the rotation angle of the servo motor SM2. Note that multiple servo motors SM2, encoders EN2, and servo control units SC2 are provided.

[0062] The translational movement mechanism 54 is provided with a servo motor SM3 for translationally moving the surgical instrument 1, an encoder EN3, and a reducer. The encoder EN3 detects the rotation angle of the servo motor SM3. The reducer decelerates the rotation of the servo motor SM3 to increase the torque. The medical cart 10 also has a servo control unit SC3 for controlling the servo motor SM3 for translationally moving the surgical instrument 1. The servo control unit SC3 is electrically connected to the encoder EN3 for detecting the rotation angle of the servo motor SM3.

[0063] The first control device 310 generates command values ​​that command the positions of the servo motors SM1, SM2, and SM3 based on the operation received by the remote operation device 200, and drives the servo motors SM1, SM2, and SM3 based on the command values. The first control device 310 then detects a deviation abnormality error when the difference between the command values ​​and the positions of the servo motors SM1, SM2, and SM3 detected by the sensors exceeds an allowable range.

[0064] 17 , the positioner 30 is provided with a plurality of servo motors SM4, an encoder EN4, and a reducer so as to correspond to a plurality of joints 33 of the positioner 30. The encoder EN4 is configured to detect the rotation angle of the servo motor SM4. The reducer is configured to decelerate the rotation of the servo motor SM4 to increase the torque.

[0065] The medical cart 10 is equipped with wheels, including front wheels as drive wheels and rear wheels steered by the operating handle 23. The rear wheels are located closer to the operating handle 23 than the front wheels. The medical cart 10 also includes a servo motor SM5 that drives each of the front wheels of the medical cart 10, an encoder EN5, a reducer, and a brake BRK. The reducer is configured to reduce the rotation of the servo motor SM5 and increase the torque. The operating handle 23 is also provided with a potentiometer P1 (see FIG. 3 ). The front wheel servo motor SM5 is driven based on the rotation angle detected by the potentiometer P1 in response to the twist of the throttle 23 a. The rear wheels of the medical cart 10 are dual-wheel type, and are steered based on the left and right rotation of the operating handle 23. 3 is disposed on the rotation axis of the operating handle 23, and a servomotor SM6, an encoder EN6, and a reducer are disposed on the rear wheels of the medical cart 10. The reducer is configured to reduce the rotation speed of the servomotor SM6 and increase the torque. The servomotor SM6 is driven based on the rotation angle detected by the potentiometer P2 in response to the left and right rotation of the operating handle 23. In other words, steering of the rear wheels by the left and right rotation of the operating handle 23 is configured to be power-assisted by the servomotor SM6.

[0066] The front wheels of the medical cart 10 are driven to move forward and backward, and the rear wheels are steered by rotating the operating handle 23, causing the medical cart 10 to rotate left and right.

[0067] As shown in Figure 17, the medical cart 10 is provided with a servo control unit SC4 for controlling the servo motor SM4 that moves the positioner 30. An encoder EN4 for detecting the rotation angle of the servo motor SM4 is electrically connected to the servo control unit SC4. The medical cart 10 is also provided with a servo control unit SC5 for controlling the servo motor SM5 that drives the front wheels of the medical cart 10. An encoder EN5 for detecting the rotation angle of the servo motor SM5 is electrically connected to the servo control unit SC5. The medical cart 10 is also provided with a servo control unit SC6 for controlling the servo motor SM6 that power-assists the steering of the rear wheels of the medical cart 10. An encoder EN6 for detecting the rotation angle of the servo motor SM6 is electrically connected to the servo control unit SC6.

[0068] As shown in Figures 16 and 17, brakes BRK are mounted on the joints JT1, JT2, JT3, JT4, JT5, JT6, and JT7 of the arm unit 51 and on the joint 33 of the positioner 30. Brakes BRK are also mounted on the front wheels of the medical cart 10, the arm base 40, and the translational movement mechanism 54. Control signals are transmitted unidirectionally from the arm control unit 320 to the brakes BRK mounted on the joints JT1, JT2, JT3, JT4, JT5, JT6, and JT7 of the arm unit 51 and on the translational movement mechanism 54. The control signals are signals that turn the brakes BRK on and off. The signal that turns the brakes BRK on includes a signal that keeps the brakes BRK engaged. The same applies to control signals sent from the positioner control unit 330 to the brakes BRK mounted on the joints 33 of the positioner 30 and on the arm base 40. At startup, all brakes BRK of the arm base 40, arm unit 51, and translational movement mechanism 54 are released, and the servo motor SM is driven to resist gravity, thereby maintaining the posture of the robot arm 50 and the posture of the arm base 40. When an error occurs in the surgery assistance system 500, the brakes BRK mounted on the arm base 40, arm unit 51, and translational movement mechanism 54 are turned on. When the error in the surgery assistance system 500 is resolved, the brakes BRK mounted on the arm base 40, arm unit 51, and translational movement mechanism 54 are turned off. A shutdown operation of the surgery assistance system 500 turns on the brakes BRK mounted on the arm base 40, arm unit 51, and translational movement mechanism 54. In addition, the brakes BRK of the front wheels of the medical cart 10 are always turned on, and are only released while the enable switch 23b is pressed down. Furthermore, the brakes BRK of the joints 33 of the positioner 30 are always on, and the brakes BRK are released only while the enable switches 22c are pressed.

[0069] As shown in FIG. 18 , servo motors SM7a, SM7b, SM7c, SM7d, SM7e, SM7f, and SM7g are disposed at joints JT21, JT22, JT23, JT24, JT25, JT26, and JT27 of the operation unit 110, respectively. Servo motor SM7a rotates link portion 111a around axis A21. Servo motor SM7b rotates link portion 111b around axis A22. Servo motor SM7c rotates link portion 111c around axis A23. Servo motor SM7d rotates link portion 112a around axis A24. Servo motor SM7e rotates link portion 112b around axis A25. Servo motor SM7f rotates link portion 112c around axis A26. Servo motor SM7g rotates grip support member 112d around axis A27. Servo controllers SC7a, SC7b, SC7c, SC7d, SC7e, SC7f, and SC7g are provided to control the servo motors. Encoders EN7a, EN7b, EN7c, EN7d, EN7e, EN7f, and EN7g are electrically connected to the servo controllers to detect the rotation angles of the servo motors. The servo motors, servo controllers, and encoders are provided in operation unit 110L and operation unit 110R, respectively.

[0070] The first control device 310 controls each servo motor via the operation control unit 340 to generate a torque that cancels out the gravitational torque generated in the rotation shaft of each servo motor, depending on the attitude of the operation unit 110. This enables the operator to operate the operation unit 110 with a relatively small force.

[0071] Furthermore, when the operator performs an operation to rotate the grip support member 112d of the operation unit 110 around the A27 axis shown in Figures 12 and 13, the shaft 2d of the instrument 2 rotates around the A9 axis shown in Figure 5. Furthermore, when the operator performs an operation to rotate the joints JT24, JT25, and JT26 of the operation unit 110 shown in Figures 12 and 13, the end effector 2b bends around the A10 axis or A11 axis shown in Figure 5.

[0072] Operation control unit 340 is disposed on the main body of remote control device 200. Operation control unit 340 controls operation unit 110. As shown in Fig. 15 , operation control unit 340 is disposed to correspond to each of operation unit 110L for the left hand and operation unit 110R for the right hand.

[0073] 15, the vision unit 300 and the image processing unit 400 are connected to a first control device 310 via a LAN or the like. The display unit 220 is connected to the vision unit 300.

[0074] (Setting of Pivot Position) Next, setting of the pivot position PP using the marker imaging unit 70 will be described. Here, in the first embodiment, a marker member M is attached to at least one of the patient P on whom surgery is performed using the surgical support robot 100 and the trocar T inserted into the body surface S of the patient P. The distance between the marker imaging unit 70 and the marker member M is, for example, approximately 1 m. In FIG. 19 , the marker member M is attached to the body surface S of the patient P, and in FIG. 20 , the marker member M is attached to the trocar T. The marker member M is detachable from at least one of the patient P and the trocar T. The marker member M is placed on at least one of the patient P and the trocar T by the operator. For example, an adhesive layer is formed on the marker member M, and the marker member M is attached to at least one of the patient P and the trocar T. The marker member M may be attached to both the body surface S of the patient P and the trocar T. The marker member M may be attached to a position that makes it easy for the marker imaging unit 70 to capture an image, for example, based on the judgment of the operator.

[0075] In the first embodiment, as shown in FIG. 21 , the marker element M is a rectangular AR marker. The AR marker serves as a landmark for displaying augmented reality, which allows digital content to be overlaid on real space. The AR marker is formed, for example, by combining white and black shapes. Information regarding the direction and distance from the marker element M at which a pivot position is to be set is associated with the marker element M. This associated information is stored in advance in at least one of the storage unit 311 and the storage unit 351. The length of one side of the marker element M is, for example, 20 mm or more and 30 mm or less.

[0076] Furthermore, in the first embodiment, as described above, a plurality of robot arms 50 to which surgical instruments 1 are attached are disposed. For example, four robot arms 50a, 50b, 50c, and 50d are disposed. A plurality of AR markers are disposed corresponding to the plurality of robot arms 50. In FIGS. 19 and 20 , four AR markers are affixed corresponding to the four robot arms 50a, 50b, 50c, and 50d. The plurality of AR markers are linked with at least one of information about the corresponding robot arm 50 and information about the type of surgical instrument 1 attached to the robot arm 50. For example, information indicating which of the four robot arms 50a, 50b, 50c, and 50d each of the four AR markers corresponds to is stored in advance in at least one of the storage unit 311 and the storage unit 351. In addition, information as to whether the surgical tool 1 attached to each of the four robot arms 50a, 50b, 50c, and 50d is an instrument 2 or an endoscope 3 is stored in advance in at least one of the memory unit 311 and the memory unit 351.

[0077] The marker imaging unit 70 is disposed in at least one of the robot arm 50, the operating table B on which the patient P is placed, and the arm base 40 to which the robot arm 50 is attached. In the first embodiment, as shown in FIG. 22 , the marker imaging unit 70 is disposed in the arm base 40 to which the robot arm 50 is attached. The marker imaging unit 70 attached to the arm base 40 also serves as an imaging unit for positioning the surgical support robot 100. Specifically, when the operator moves the surgical support robot 100 near the patient P placed on the operating table B, an image of the patient P captured by the marker imaging unit 70 is displayed in real time on the display unit 22a, as shown in FIG. 23 . As shown in FIG. 23 , a first mark MK1 and a second mark MK2 are displayed on the display unit 22a. The first mark MK1 is circular, and the second mark MK2 is cross-shaped. The display unit 22a displays the body surface S of the patient P photographed by the marker imaging unit 70 so as to overlap the first mark portion MK1 and the second mark portion MK2. While visually checking the display unit 22a, the operator moves the surgical support robot 100 so that the trocar T, into which the endoscope 3 is inserted, is positioned inside the first mark portion MK1.

[0078] In the first embodiment, the marker imaging unit 70 captures an image of a marker element M detachably attached to at least one of the patient P and a trocar T inserted into the body surface S of the patient P. The marker imaging unit 70 also includes a two-dimensional camera. The captured image of the marker element M captured by the marker imaging unit 70 is transmitted to at least one of the first control device 310 and the second control device 350. An example in which the image of the marker element M is transmitted to the second control device 350 will be described below. In this case, information linked to the marker element M is stored in the storage unit 351.

[0079] In the first embodiment, the second control device 350 detects the marker element M based on a captured image of the marker element M. The second control device 350 then sets a pivot position PP, which serves as a fulcrum for movement of the surgical instrument 1, based on the detected marker element M. Specifically, the second control device 350 reads information such as the shape of the marker element M from the captured image of the marker element M and reads information associated with the read marker element M stored in the storage unit 351. As described above, the information associated with the marker element M includes information regarding which robot arm 50 the read marker element M corresponds to, which of the instrument 2 and the endoscope 3 is attached to the corresponding robot arm 50, and, as shown in FIG. 24 , the direction and distance from the marker element M at which the pivot position PP is to be set. The second control device 350 also calculates the position and orientation of the marker element M from the captured image of the marker element M. The position and orientation of the marker element M are calculated by having the marker imaging unit 70 capture images of the marker element M arranged in various positions and orientations in advance, and calibrating the marker imaging unit 70 based on the captured images. For example, the marker element M is captured from multiple angles by the marker imaging unit 70 to identify the lens distortion and focal length of the marker imaging unit 70. The second control device 350 then calculates the position and orientation of the marker element M based on the known length of one side of the actually captured marker element M and the calibration results. The second control device 350 then sets the pivot position PP at coordinates separated by the read distance and in the direction read from the storage unit 351 based on the position and orientation of the marker element M in the captured image. Note that, in reality, the coordinates of the marker element M detected from the captured image are in the coordinate system of the marker imaging unit 70. Therefore, the coordinates of the marker element M are converted from the coordinate system of the marker imaging unit 70 to the coordinate system of the surgical support robot 100, and the pivot position PP is then set based on the direction and distance read from the storage unit 351.The second control device 350 may detect the coordinates of the marker member M from the captured image, and set the pivot position to a coordinate spaced from the detected coordinates by the read distance in the direction read from the storage unit 351, and then further moved a predetermined distance into the abdominal wall or pleura of the patient P. For example, as shown in Fig. 25, the pivot position PP is set to the center position in the thickness direction of the abdominal wall of the portion of the abdominal wall where the trocar T of the patient P is inserted. The accuracy of detecting the position of the marker member M is, for example, about ±10 mm.

[0080] Furthermore, in the first embodiment, the marker imaging unit 70 captures images of the marker element M in real time. The second control device 350 changes the pivot position PP based on the captured images of the marker element M captured in real time. Specifically, the marker imaging unit 70 captures moving images of the marker element M. The second control device 350 detects the marker element M in the captured images at predetermined time intervals. The second control device 350 then sets a new pivot position PP based on the newly detected marker element M. Note that the second control device 350 may change the pivot position PP when the error between the pivot position PP based on the newly detected marker element M and the currently set pivot position PP is equal to or greater than a predetermined threshold.

[0081] (Control Method of Surgery Support System) A control method of the surgery support system 500 will be described.

[0082] First, as a preliminary preparation, the operator attaches a marker member M to at least one of the patient P and the trocar T. If the surgery support robot 100 is provided with a plurality of robot arms 50, the operator attaches the marker members M in a number corresponding to the number of the plurality of robot arms 50.

[0083] 26 , in step S1, the operator operates the operation handle 23 of the cart positioner operation unit 20, thereby starting the movement of the surgical support robot 100. The surgical support robot 100 is moved to the vicinity of the patient P placed on the operating table B. The movement of the surgical support robot 100 is controlled by the first control device 310.

[0084] In step S2, the marker image capturing unit 70 starts capturing images. Note that the marker image capturing unit 70 captures images continuously. That is, the marker image capturing unit 70 captures moving images. Then, as shown in Fig. 22, when the surgery support robot 100 is moved near the patient P placed on the operating table B, the marker image capturing unit 70 captures an image of the marker member M detachably attached to at least one of the patient P and the trocar T inserted into the body surface S of the patient P, as shown in Fig. 23.

[0085] In step S3, the second control device 350 detects the marker element M based on the captured image of the marker element M. For example, the second control device 350 recognizes the shape of the marker element M from the captured image of the marker element M. Note that the detection of the marker element M may be performed automatically by the second control device 350 when the marker imaging unit 70 captures an image of the marker element M, or the second control device 350 may start the operation of recognizing the marker element M from the captured image based on an input operation by the operator.

[0086] In step S4, the second control device 350 sets a pivot position PP, which serves as a fulcrum for movement of the surgical instrument 1, based on the detected marker member M. For example, the second control device 350 reads out the shape of the marker member M recognized in step S3 and information associated with the marker member M that is stored in advance in the storage unit 351. Then, based on the read-out information, the second control device 350 sets a pivot position PP for the robot arm 50 that corresponds to the detected marker member M.

[0087] In step S5, the second control device 350 determines whether pivot positions PP have been set for all of the robot arms 50. If the answer is no in step S5, the operations of steps S3 and S4 are repeated. Note that, because the marker member M contains information indicating which robot arm 50 it corresponds to, it is also possible to perform detection of the marker member M for all of the robot arms 50 in step S3, and then set the pivot positions PP for all of the robot arms 50 simultaneously in step S4.

[0088] In step S6, the second control device 350 changes the pivot position PP based on the captured image of the marker member M, which is captured in real time. Note that the operation of step S6 is performed while the surgery is being performed by the surgery assistance system 500.

[0089] Effect of First Embodiment The surgery assistance system 500 includes a marker imaging unit 70 that images a marker element M attached to at least one of the patient P and the trocar T, and a second control device 350 that detects the marker element M based on the captured image of the marker element M and sets a pivot position PP that serves as a fulcrum for movement of the surgical instrument 1 based on the detected marker element M. As a result, the marker element M is detachable from the patient P and the trocar T, so that the marker element M can be attached to a position that does not expose other medical equipment or the like. Therefore, the marker imaging unit 70 can reliably image the marker element M, and therefore the pivot position PP that serves as a fulcrum for movement of the surgical instrument 1 can be reliably set based on the marker element M detected from the captured image.

[0090] Furthermore, when setting the pivot position PP using the pivot button 66, if the operator's level of skill is low, the pivot position PP may not be set correctly. As described above, the second control device 350 sets the pivot position PP based on the detected marker member M, allowing the pivot position PP to be set accurately and in a short time. If the patient P's body shifts after the pivot position PP is set, the trocar T and the surgical instrument 1 may come into contact with each other with relatively high pressure. In this case, when the surgical instrument 1 is moved, the friction between the trocar T and the surgical instrument 1 causes the surgical instrument 1 and the robot arm 50 holding the surgical instrument 1 to vibrate. By accurately setting the pivot position PP as described above, vibration of the surgical instrument 1 and the robot arm 50 holding the surgical instrument 1 can be suppressed.

[0091] The marker member M includes a rectangular AR marker, the marker imaging unit 70 includes a two-dimensional camera, and the second control device 350 sets the pivot position PP based on an image of the AR marker captured by the two-dimensional camera. This makes it possible to easily set the pivot position PP based on the image of the AR marker captured by the two-dimensional camera, for example, by linking the AR marker with information such as the direction and distance the pivot position PP should be shifted from the AR marker.

[0092] A plurality of robot arms 50 to which surgical instruments 1 are attached are arranged, a plurality of AR markers are arranged corresponding to the plurality of robot arms 50, and the plurality of AR markers are linked to at least one of information about the corresponding robot arm 50 and information about the type of surgical instrument 1 attached to the robot arm 50. This allows the second control device 350 to recognize which robot arm 50 and which type of surgical instrument 1 the imaged AR marker corresponds to, and therefore the pivot position PP can be appropriately set for the robot arm 50 and surgical instrument 1 linked to the imaged AR marker.

[0093] The marker imaging unit 70 is disposed on the arm base 40 to which the robot arm 50 is attached. This prevents the marker imaging unit 70 from interfering with other medical devices as the robot arm 50 moves, unlike when the marker imaging unit 70 is disposed on the robot arm 50.

[0094] The surgery support system 500 includes a marker image capture unit 70 as an image capture unit for alignment of the surgery support robot 100 attached to the arm base 40. When the surgery support robot 100 is moved near a patient P placed on the operating table B, an image of the patient P captured by the marker image capture unit 70 as an image capture unit for alignment is displayed in real time on the display unit 22a. In other words, the marker image capture unit 70 also serves as an image capture unit for alignment. This prevents the configuration of the surgery support system 500 from becoming complicated, unlike when the marker image capture unit 70 is provided separately from the image capture unit for alignment.

[0095] The marker imaging unit 70 captures an image of the marker element M in real time, and the second control device 350 changes the pivot position PP based on the captured image of the marker element M captured in real time. As a result, the pivot position PP is changed in real time, so that even if the patient P shifts from the operating table B, the surgical support robot 100 can be operated with the pivot position PP appropriately set. Furthermore, because fluctuations in the pivot position PP due to the patient P's breathing can be detected in real time, the surgical support robot 100 can be operated with the pivot position PP appropriately set, even if the pivot position PP shifts due to the patient P's breathing.

[0096] Second Embodiment The configuration of a surgery assistance system 500 according to a second embodiment will be described.

[0097] As shown in FIG. 27 , in the second embodiment, the marker image capture unit 71 is disposed on the robot arm 50. The marker image capture unit 71 is attached to the robot arm 50 when the pivot position PP is set, and is detached from the robot arm 50 after the pivot position PP is set. For example, in step S1 of the control method for the surgery assistance system 500 of the first embodiment, the marker image capture unit 71 is attached to one of the four robot arms 50 before the surgery assistance robot 100 is moved near the patient P placed on the operating table B. For example, the marker image capture unit 71 is attached to the second link unit 53 of the robot arm 50. The marker image capture unit 71 is detachable from the second link unit 53. The marker image capture unit 71 may be attached to a portion of the robot arm 50 other than the second link unit 53. Other configurations of the second embodiment are similar to those of the first embodiment.

[0098] [Effects of the Second Embodiment] The marker imaging unit 70 is disposed on the robot arm 50. As a result, even if the marker member M is disposed in a position that is in a blind spot of other medical equipment, the marker imaging unit 70 can be moved by the robot arm 50, and therefore the marker member M can be imaged.

[0099] The marker imaging unit 70 is attached to the robot arm 50 when the pivot position PP is set, and after the pivot position PP is set, it is detached from the robot arm 50. As a result, since the marker imaging unit 70 is detached from the robot arm 50 after the pivot position PP is set, it is possible to prevent the marker imaging unit 70 from interfering with other medical devices during surgery, etc.

[0100] [Modifications] The embodiments disclosed herein should be considered to be illustrative in all respects and not restrictive. The scope of the present disclosure is indicated by the claims, not by the description of the above embodiments, and further includes all modifications and variations within the meaning and scope of the claims.

[0101] In the above-described first and second embodiments, an example has been described in which the marker member M is disposed on either the patient P or the trocar T, but the present disclosure is not limited to this. For example, the marker member M may be disposed on both the patient P and the trocar T.

[0102] In the first and second embodiments, the marker member M is an AR marker, but the present disclosure is not limited to this. For example, if the marker imaging unit is a three-dimensional camera, the marker member M may be a simple pattern other than an AR marker.

[0103] In the first and second embodiments described above, an example has been shown in which a plurality of marker members M are arranged to correspond to a plurality of robot arms 50, but the present disclosure is not limited to this. For example, if only one robot arm 50 is arranged, only one marker member M may be arranged. In this case, the marker member M does not need to include information regarding which robot arm 50 the marker member M corresponds to.

[0104] Furthermore, in the first and second embodiments, an example was described in which the marker imaging unit 70 was a two-dimensional camera. However, the present disclosure is not limited to this. For example, the marker imaging unit 70 may be a three-dimensional camera. The three-dimensional camera may be, for example, a stereo camera or a ToF (Time of Flight) camera. The second control device 350 then sets the pivot position PP based on the captured image of the marker element M captured by the three-dimensional camera. That is, the second control device 350 calculates the distance to the marker element M, the position, and the orientation of the marker element M based on the captured image of the marker element M captured by the three-dimensional camera. The second control device 350 then sets the pivot position PP based on the calculated distance to the marker element M, the position, and the orientation of the marker element M, and the information read from the memory unit 351. In this way, since the marker imaging unit 70 is a three-dimensional camera, the three-dimensional camera can detect the three-dimensional shape of the subject, and therefore the pivot position PP, which serves as the fulcrum for the movement of the surgical instrument 1, can be appropriately set based on the image of the marker element M captured by the three-dimensional camera.

[0105] In the above first and second embodiments, an example has been described in which one marker member M is disposed for one robot arm 50. However, the present disclosure is not limited to this. For example, as shown in FIG. 28 , a plurality of marker members M may be disposed for one robot arm 50. In FIG. 28 , two marker members M are disposed for one robot arm 50. The second control device 350 then sets the pivot position PP based on the detected plurality of marker members M. For example, the second control device 350 sets the pivot position PP at the midpoint between the two marker members M. By disposing a plurality of marker members M for one robot arm 50 in this manner, even in cases where the pivot position PP cannot be set accurately based on a single marker member M, the pivot position PP can be set accurately based on the plurality of marker members M.

[0106] In the first embodiment, the marker image capturing unit 70 is disposed on the arm base 40, and in the second embodiment, the marker image capturing unit 71 is disposed on the robot arm 50. However, the present disclosure is not limited to this. For example, as shown in FIG. 29 , the marker image capturing unit 72 may be disposed on the operating table B on which the patient P is placed. For example, the marker image capturing unit 72 is disposed at a relatively high position so that multiple marker members M are within the field of view. In this way, by disposing the marker image capturing unit 72 on the operating table B on which the patient P is placed, the marker image capturing unit 72 moves together with the operating table B. Therefore, even if the operating table B is moved, the marker members M can be captured by the marker image capturing unit 72. Furthermore, the marker image capturing unit may be disposed at multiple positions among the arm base 40, the robot arm 50, and the operating table B.

[0107] 30 , the endoscope 3 disposed on the robot arm 50 may also serve as a marker imaging unit 70 that images the marker members M. In this case, the endoscope 3 images the multiple marker members M, and the second control device 350 sets pivot positions PP for the multiple robot arms 50 based on the images of the multiple marker members M. In this way, the endoscope 3 also serves as the marker imaging unit 70 that images the marker members M, which makes it possible to prevent the configuration of the surgery assistance system 500 from becoming complicated, unlike when the marker imaging unit 70 is disposed separately from the endoscope 3.

[0108] In addition, in the first and second embodiments described above, an example in which four robot arms 50 are arranged is shown, but the present disclosure is not limited to this. In the present disclosure, the number of robot arms 50 may be other than four.

[0109] In the first and second embodiments, the arm unit 51 and the positioner 30 are configured as a seven-axis articulated robot, but the present disclosure is not limited to this. For example, the arm unit 51 and the positioner 30 may be configured as an articulated robot with an axis configuration other than a seven-axis articulated robot. An example of an axis configuration other than a seven-axis articulated robot is a six-axis or eight-axis robot.

[0110] In the first and second embodiments, the surgical support robot 100 includes the medical cart 10, the positioner 30, and the arm base 40. However, the present disclosure is not limited to this. For example, the medical cart 10, the positioner 30, and the arm base 40 are not necessarily required, and the surgical support robot 100 may be configured with only the robot arm 50.

[0111] The functions of the elements disclosed herein can be performed using circuits or processing circuits, including general-purpose processors, special-purpose processors, integrated circuits, ASICs (Application Specific Integrated Circuits), conventional circuits, and / or combinations thereof, configured or programmed to perform the disclosed functions. A processor is considered a processing circuit or circuit because it includes transistors and other circuitry. In this disclosure, a circuit, unit, or means is hardware that performs the recited functions or hardware that is programmed to perform the recited functions. The hardware may be hardware disclosed herein or other known hardware that is programmed or configured to perform the recited functions. Where the hardware is a processor, which is considered a type of circuit, the circuit, means, or unit is a combination of hardware and software, and the software is used to configure the hardware and / or processor.

[0112] Aspects It will be appreciated by those skilled in the art that the exemplary embodiments described above are examples of the following aspects.

[0113] (Aspect 1) A surgical support system comprising: a surgical support robot including a robotic arm to which a surgical instrument is attached; a marker imaging unit that images a marker member that is detachably attached to at least one of a patient on whom surgery is to be performed using the surgical support robot and a trocar inserted into the body surface of the patient; and a control unit that detects the marker member based on an image of the marker member, and sets a pivot position that serves as a fulcrum for movement of the surgical instrument based on the detected marker member.

[0114] (Aspect 2) The surgical assistance system according to Aspect 1, wherein the marker member includes a rectangular shaped AR marker, the marker imaging unit includes a two-dimensional camera, and the control unit sets the pivot position based on an image of the AR marker captured by the two-dimensional camera.

[0115] (Aspect 3) A surgical assistance system as described in Aspect 2, wherein a plurality of the robot arms to which the surgical instruments are attached are arranged, a plurality of the AR markers are arranged to correspond to the plurality of robot arms, and the plurality of AR markers are linked to at least one of information about the corresponding robot arm and information about the type of the surgical instrument attached to the robot arm.

[0116] (Aspect 4) The surgical assistance system according to any one of Aspects 1 to 3, wherein the marker imaging unit includes a three-dimensional camera, and the control unit sets the pivot position based on an image of the marker member captured by the three-dimensional camera.

[0117] (Aspect 5) The surgical assistance system according to any one of Aspects 1 to 4, wherein a plurality of the marker members are arranged on one of the robot arms, and the control unit sets the pivot position based on the detected plurality of marker members.

[0118] (Aspect 6) The surgical assistance system according to any one of Aspects 1 to 5, wherein the marker imaging unit is disposed in at least one of the robot arm, an operating table on which the patient is placed, and an arm base to which the robot arm is attached.

[0119] (Aspect 7) The surgery assistance system according to Aspect 6, wherein the marker imaging unit is attached to the robot arm when the pivot position is set, and is detached from the robot arm after the pivot position is set.

[0120] (Aspect 8) A surgical support system as described in Aspect 7, comprising an imaging unit for positioning the surgical support robot attached to the arm base, wherein when the surgical support robot is moved near the patient placed on the operating table, an image of the patient captured by the imaging unit for positioning is displayed in real time on a display unit, and the imaging unit for positioning also serves as the marker imaging unit for capturing an image of the marker member.

[0121] (Aspect 9) The surgery assistance system according to aspect 7 or 8, wherein the surgical instrument includes an endoscope, and the endoscope also serves as the marker imaging unit that images the marker member.

[0122] (Aspect 10) The surgical assistance system according to any one of Aspects 1 to 9, wherein the marker imaging unit images the marker member in real time, and the control unit changes the pivot position based on the image of the marker member that is imaged in real time.

[0123] (Aspect 11) A control method for a surgical support system, comprising: capturing an image of a marker member detachably attached to at least one of a patient undergoing surgery using a surgical support robot including a robotic arm to which a surgical instrument is attached and a trocar inserted into the body surface of the patient; detecting the marker member based on the captured image of the marker member; and setting, by a control unit, a pivot position that serves as a fulcrum for movement of the surgical instrument based on the detected marker member.

[0124] (Aspect 12) A program that performs the following operations: capturing an image of a marker member that is detachably attached to at least one of a patient undergoing surgery using a surgical support robot including a robotic arm to which a surgical instrument is attached and a trocar inserted into the body surface of the patient; detecting the marker member based on the captured image of the marker member; and setting a pivot position that serves as a fulcrum for movement of the surgical instrument based on the detected marker member.

[0125] DESCRIPTION OF SYMBOLS 1 Surgical instrument 3 Endoscope 22a Display unit 40 Arm base 50 Robot arm 70 Marker imaging unit (imaging unit for alignment) 71 Marker imaging unit 72 Marker imaging unit 100 Surgery support robot 350 Second control device (control unit) 500 Surgery support system B Operating table M Marker member P Patient PP Pivot position S Body surface T Trocar

Claims

1. A surgical support system comprising: a surgical support robot including a robotic arm to which a surgical instrument is attached; a marker imaging unit that images a marker member detachably attached to at least one of a patient on whom surgery is to be performed using the surgical support robot and a trocar inserted into the body surface of the patient; and a control unit that detects the marker member based on an image of the marker member, and sets a pivot position that serves as a fulcrum for movement of the surgical instrument based on the detected marker member.

2. The surgical support system of claim 1, wherein the marker member includes a rectangular shaped AR marker, the marker imaging unit includes a two-dimensional camera, and the control unit sets the pivot position based on an image of the AR marker captured by the two-dimensional camera.

3. The surgical assistance system of claim 2, wherein a plurality of the robot arms to which the surgical instruments are attached are arranged, a plurality of the AR markers are arranged corresponding to the plurality of robot arms, and the plurality of AR markers are linked to at least one of information about the corresponding robot arm and information about the type of the surgical instrument attached to the robot arm.

4. The surgical support system of claim 1, wherein the marker imaging unit includes a three-dimensional camera, and the control unit sets the pivot position based on an image of the marker member captured by the three-dimensional camera.

5. The surgical assistance system of claim 1, wherein a plurality of the marker members are arranged on one of the robot arms, and the control unit sets the pivot position based on the detected plurality of marker members.

6. The surgical support system of claim 1, wherein the marker imaging unit is arranged in at least one of the robot arm, the operating table on which the patient is placed, and the arm base to which the robot arm is attached.

7. The surgical assistance system according to claim 6, wherein the marker imaging unit is attached to the robot arm when the pivot position is set, and is detached from the robot arm after the pivot position is set.

8. A surgical support system as described in claim 7, further comprising an imaging unit for aligning the surgical support robot attached to the arm base, wherein when the surgical support robot is moved near the patient placed on the operating table, an image of the patient taken by the imaging unit for alignment is displayed in real time on a display unit, and the imaging unit for alignment also serves as the marker imaging unit for imaging the marker member.

9. The surgical support system according to claim 7, wherein the surgical instrument includes an endoscope, and the endoscope also serves as the marker imaging unit that images the marker member.

10. The surgical support system of claim 1, wherein the marker imaging unit images the marker element in real time, and the control unit changes the pivot position based on the image of the marker element captured in real time.

11. A method for controlling a surgical support system, comprising: capturing an image of a marker element detachably attached to at least one of a patient undergoing surgery using a surgical support robot including a robotic arm to which a surgical instrument is attached and a trocar inserted into the surface of the patient's body; detecting the marker element based on the captured image of the marker element; and setting, by a control unit, a pivot position that serves as a fulcrum for movement of the surgical instrument based on the detected marker element.

12. A program that performs the following operations: capturing an image of a marker member that is detachably attached to at least one of a patient undergoing surgery using a surgical support robot including a robotic arm to which a surgical instrument is attached and a trocar inserted into the surface of the patient's body; detecting the marker member based on the captured image of the marker member; and setting a pivot position that serves as a fulcrum for the movement of the surgical instrument based on the detected marker member.

Images