Control device and control method, and surgery assisting robot system

The control device and method for surgical support robot systems enhance tool exchange efficiency by displaying priority lists of replaceable tools, addressing the complexity of multiple instrument use and reducing surgical time through automated selection and exchange processes.

WO2024236927A1PCT designated stage expired Publication Date: 2024-11-21SONY GROUP CORP
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2024/013719
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-16
Filing Date
2024-04-03
Publication Date
2024-11-21

AI Technical Summary

Technical Problem

Current surgical support robot systems face challenges in efficiently exchanging surgical tools during procedures, particularly when multiple instruments are used, leading to increased surgical time and complexity due to the lack of automated tool selection and replacement mechanisms.

Method used

A control device and method that includes an operation unit with followers, a console unit with readers operated by the operator, and a display device showing the patient's affected area, allowing for the detection of predetermined events to release the leader-follower linkage, display replaceable tools, and prioritize their selection on the screen, enabling seamless surgical tool exchange.

Benefits of technology

Facilitates smooth and efficient surgical tool exchange by displaying priority lists of replaceable tools, reducing operator intervention time and improving procedural efficiency in surgical support robot systems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2024013719_21112024_PF_FP_ABST
    Figure JP2024013719_21112024_PF_FP_ABST
Patent Text Reader

Abstract

Provided is a control device for smoothly replacing surgical tools in a surgery assisting robot system. This control device is for a surgery assisting robot system comprising: an operation unit including one or more followers to which a surgical tool is attached in a replaceable manner; a console unit including one or more leaders that are operated by an operator; and a display device for displaying, on a screen, a captured image of an affected area of a patient under surgery using the surgical tool. The control device controls the movement of the follower in accordance with the operation of the leader, and when a predetermined event is detected, disengages the coordination between the leader and the follower, displays information relating to a surgical tool that can be replaced with on the screen, and replaces the surgical tool attached to the follower.
Need to check novelty before this filing date? Find Prior Art

Description

Control device, control method, and surgical support robot system

[0001] The technology disclosed in this specification (hereinafter referred to as "the present disclosure") relates to a control device and control method for a surgical support robot system that performs treatment while exchanging multiple surgical tools, and the surgical support robot system.

[0002] Medical procedures are generally performed using surgical instruments such as forceps, tweezers, needle holders, scissors, electrosurgical scalpels, insufflation tubes, and energy treatment devices (hereinafter collectively referred to as "surgical instruments"), as well as imaging devices such as endoscopes. Recently, surgical robot systems have been used that support these surgical instruments and imaging devices. In the case of a master-slave (hereinafter referred to as a "leader-follower") surgical robot system, a medical professional such as a doctor (hereinafter referred to as an "operator") can operate the arms by manipulating a control device to perform surgery while observing images of the patient's body and other organs captured by an imaging device.

[0003] Typically, multiple surgical instruments are used in a single surgery. Therefore, in a surgical support robot system, the task of changing the surgical instruments frequently occurs during surgery. For example, a surgical system (see Patent Document 1) is known that assumes that an assistant will perform the surgical instrument change task. In this surgical system, it takes time from when the operator issues an instruction to change the surgical instrument to when the change is completed and the procedure can be resumed. Furthermore, as the number of surgical instruments used increases, it becomes difficult to smoothly proceed with the surgery while frequently switching between surgical instruments.

[0004] Although a surgical device that autonomously replaces surgical tools with desired tools has been proposed (see Patent Document 2), the surgeon must determine the tools required before surgery. In other words, this surgical device does not provide a means for the operator to select the next surgical tool to use from multiple replacement options during surgery. Furthermore, without a means for checking which surgical tools are available for replacement, as the number of surgical tools increases, it takes the operator a long time to select the surgical tool, resulting in longer surgery times.

[0005] JP 2021-19949 A JP 2022-512316 A JP 09-34620 A

[0006] An object of the present disclosure is to provide a control device and control method for smoothly changing surgical tools in a surgical support robot system, and a surgical support robot system.

[0007] The present disclosure has been made in consideration of the above-mentioned problems, and a first aspect thereof is a control device for controlling a surgical support robot system including an operation unit including one or more followers to which surgical tools are interchangeably attached, a console unit including one or more readers operated by an operator, and a display device that displays on a screen an image of the affected area of ​​a patient undergoing surgery with the surgical tools, the control device controls the operation of the follower in accordance with the operation of the leader, and when a predetermined event is detected, releases the link between the leader and the follower, displays information about interchangeable surgical tools on the screen, and replaces the surgical tool attached to the follower.

[0008] The control device detects the specified event based on the operation of a specified operator located on the console unit or elsewhere, or the movement of any of the one or more readers into a specified area.

[0009] The control device calculates the priority of each surgical tool that can be replaced with the surgical tool currently attached to the follower, and displays a list of replacement candidate surgical tools on the screen of the display device based on the priority.The control device also uses the leader that has released leader-follower control as a pointing device to control operations on the screen, such as selecting a surgical tool on the surgical tool list screen.

[0010] A second aspect of the present disclosure is a control method for a surgical support robot system including an operation unit including one or more followers to which surgical tools are interchangeably attached, a console unit including one or more readers operated by an operator, a display device that displays on a screen an image of the affected area of ​​a patient undergoing surgery with the surgical tools, and a control device that controls the operation of the followers in accordance with the operation of the leaders, the control method comprising the steps of: detecting a predetermined event that occurs on the surgical support robot system; releasing the link between the leader and the follower in response to the occurrence of the predetermined event and displaying information about interchangeable surgical tools on the screen; and replacing the surgical tool attached to the follower with a surgical tool selected based on the displayed information.

[0011] A third aspect of the present disclosure is a surgical support robot system comprising: an operation unit including one or more followers to which surgical tools are interchangeably attached; a console unit including one or more readers operated by an operator; a display device that displays on a screen an image of the affected area of ​​a patient undergoing surgery with the surgical tools; and a control device that controls the operation of the follower in accordance with the operation of the reader, wherein when the control device detects a predetermined event, it releases the link between the leader and the follower, displays information about the interchangeable surgical tools on the screen, and replaces the surgical tools attached to the follower.

[0012] However, the term "system" used here refers to a logical collection of multiple devices (or functional modules that realize specific functions), regardless of whether each device or functional module is contained within a single housing. In other words, both a single device consisting of multiple parts or functional modules and a collection of multiple devices are considered "systems."

[0013] According to the present disclosure, it is possible to provide a control device and control method, as well as a surgical support robot system, that initiates a surgical tool replacement sequence in response to a specified operation by an operator and displays candidate surgical tools on a screen to facilitate surgical tool replacement.

[0014] It should be noted that the effects described in this specification are merely examples, and the effects brought about by the present disclosure are not limited to these. Furthermore, the present disclosure may also bring about additional effects in addition to the effects described above.

[0015] Further objects, features, and advantages of the present disclosure will become apparent from the following detailed description based on the embodiments and accompanying drawings.

[0016] FIG. 1 is a diagram showing the external configuration of the surgical support robot system 100. FIG. 2 is a diagram showing the functional configuration of the surgical support robot system 100. FIG. 3 is a diagram showing an example of the functional configuration of the robot control device 130. FIG. 4 is a diagram showing the external configuration of the operation unit 120 and the robot control device 130. FIG. 5 is an enlarged view of the structure of the distal end side of the base arm 125. FIG. 6 is a diagram showing a series of operations (during surgery) of the second follower 122 and the surgical tool exchange unit 124 when exchanging a surgical tool. FIG. 7 is a diagram showing a series of operations (movement to the surgical tool exchange waiting area) of the second follower 122 and the surgical tool exchange unit 124 when exchanging a surgical tool. FIG. 8 is a diagram showing a series of operations (movement to the surgical tool exchange area and rotation of the tool turret 124C) of the second follower 122 and the surgical tool exchange unit 124 when exchanging a surgical tool. FIG. 9 is a diagram showing a series of operations of the second follower 122 and the surgical tool exchange unit 124 when replacing a surgical tool (removing a surgical tool). FIG. 10 is a diagram showing a series of operations of the second follower 122 and the surgical tool exchange unit 124 when replacing a surgical tool (attaching a surgical tool). FIG. 11 is a diagram showing a series of operations of the second follower 122 and the surgical tool exchange unit 124 when replacing a surgical tool (completion of surgical tool exchange and resumption of surgery). FIG. 12 is a diagram showing the external configuration of the console unit 110. FIG. 13 is a diagram showing the support unit 1203 as viewed from the side of an operator sitting in a chair 1204. FIG. 14 is a diagram showing the external configuration of the top surface of the foot switch unit 113. FIG. 15 is a flowchart showing the operation of the surgical support robot system 100. FIG. 16 is a diagram showing an example of a surgical tool exchange pattern. FIG. 17 is a diagram showing an example of the configuration of a screen including a surgical tool selection UI (first embodiment). Fig. 18 is a diagram showing a state in which UI parts for specific operations are placed in front of UI parts that are always displayed. Fig. 19 is a diagram showing a state in which UI parts that are always displayed are placed behind UI parts for specific operations. Fig. 20 is a diagram showing a state in which UI parts are displayed in a position that does not obstruct the view of surgical tools displayed on the upper side of the screen. Fig. 21 is a diagram showing a state in which UI parts are displayed in a position that does not obstruct the view of surgical tools displayed on the lower side of the screen.FIG. 22 is a diagram showing an example of switching the depth arrangement of UI parts by operating the reader. FIG. 23 is a diagram showing an example in which a first surgical tool selection UI section and a second surgical tool selection UI section are simultaneously displayed on the screen. FIG. 24 is a diagram showing the configuration of a gripper used in the operation section. FIG. 25 is a diagram showing the opening and closing operation of the gripper. FIG. 26 is a diagram showing changes in the state quantity of the gripper's grip axis. FIG. 27 is a diagram showing a selection screen for finalizing a provisional decision. FIG. 28 is a diagram showing a surgical tool exchange standby area (top view) set in the range of motion of the follower. FIG. 29 is a diagram showing a surgical tool exchange standby area (front view) set in the range of motion of the follower. FIG. 30 is a flowchart showing the processing operation (second embodiment) of the surgical support robot system 100. FIG. 31 is a diagram showing an example of the configuration of a screen including a surgical tool selection UI (third embodiment). FIG. 32 is a diagram showing a state in which an operator is gripping and rotating the gripper. FIG. 33 is a diagram showing a state in which an operator is closing the gripper. Fig. 34 is a diagram showing another example of the configuration of a screen including a surgical tool selection UI (third embodiment). Fig. 35 is a flowchart showing the processing operation (fourth embodiment) of the surgical support robot system 100. Fig. 36 is a diagram showing a method of operating a GUI configured by layers.

[0017] Preferred embodiments of the present disclosure will be described below in the following order with reference to the drawings. Note that the embodiments described below and their dimensions, materials, and other specific numerical values ​​are merely examples to facilitate understanding of the present disclosure and are not intended to limit the technical scope of the present disclosure. Furthermore, the configurations of the following embodiments can be combined with the configurations of other embodiments.

[0018] A. System Configuration B. Functional Configuration of Robot Control Device C. Structure of Operation Unit D. Structure of Console Unit E. Surgical Tool Exchange Sequence and User Interface (First Embodiment) F. Second Embodiment G. Third Embodiment H. Fourth Embodiment

[0019] A. System Configuration Fig. 1 shows the external configuration of a leader-follower type surgical support robot system 100 to which the technology disclosed in this specification is applied. Fig. 2 also shows a schematic diagram of the functional configuration of the surgical support robot system 100.

[0020] 1 and 2, the surgical robot system 100 includes a console unit 110, an operation unit 120, a robot control device 130, and an external operation terminal 140. The surgical robot system 100 can use an external network 160. The console unit 110 and the operation unit 120 correspond to the master device and the slave device, respectively, in a master-slave system. The robot control device 130 links the console unit 110 and the operation unit 120 to achieve leader-follower control.

[0021] An operator such as a doctor or medical professional operates the console unit 110 to instruct the operation of the surgical support robot system 100. On the other hand, the operation unit 120 is equipped with a robot arm equipped with surgical tools required for surgery, and is configured to operate in response to the operation of the console unit 110 by the operator, and is placed in the operating room near the operating table 150 on which the patient lies in order to perform surgery on the patient.

[0022] When an operator inputs an operation command or instruction, the console unit 110 transmits the operation command to the operation unit 120 via the robot control device 130. The operation unit 120 operates each arm that supports the surgical tools and imaging device in accordance with the operation command received via the robot control device 130, and performs surgery on the patient.

[0023] Console unit: The console unit 110 is used when an operator remotely operates the operation unit 120 from a location away from the operating table 150 in an operating room (or outside the operating room), for example. As shown in FIG. 2, the console unit 110 is equipped with a first reader 111, a second reader 112, a foot switch unit 113, and a display device 114. It is assumed that the first reader 111 and the second reader 112 are operated by the operator's left and right hands, respectively, and the foot switch unit 113 is operated by the operator's left or right foot or both. Although not shown, the console unit 110 is further equipped with a communication device capable of communicating various information with the robot control device 130.

[0024] The first reader 111 and the second reader 112 are each configured as an articulated robot arm, supporting a first operating unit 111C corresponding to right-handed operation and a second operating unit 112C corresponding to left-handed operation at their distal ends. The first operating unit 111C and the second operating unit 112C can be moved vertically and horizontally, i.e., two-dimensionally, at each height position by an operator. However, since the technical scope of this disclosure is not limited to the degree of freedom configuration and structure of a specific robot arm, detailed description of the robot arm will be omitted in this specification. The first operating unit 111C and the second operating unit 112C each have, for example, a gripper consisting of a pair of grips (not shown) rotatably supported on a grip shaft and operable to open and close.

[0025] The first reader 111 and the second reader 112 are each equipped with electric motors 111A and 112A for driving the joints of the robot arm (first operating unit 111C and second operating unit 112C), and sensors 111B and 112B for detecting the state of the robot arm. The sensors 111B and 112B detect the state of the robot arm, such as the torque reference value, joint angle, and joint angular velocity of each joint. The torque reference value substantially corresponds to the current value input to the electric motors 111A and 112A. The joint angle is obtained from an encoder attached to the output shaft of the electric motors 111A and 112A, and the joint angular velocity is obtained by differentiating the joint angle with respect to time. The electric motors 111A and 112A are driven by drive signals from the robot control device 130 to perform the desired operation based on the position detection results detected by the sensors 111B and 112B, respectively. However, in this embodiment, the robot arms serving as the first reader 111 and the second reader 112 may include passive joints, and may be of a vertically multi-joint type or a horizontally multi-joint type.

[0026] The foot switch section 113 is configured so that the operator can use his or her feet to operate the operation unit 120 and to activate the functions of the operation unit 120. As shown in Fig. 2, the foot switch section 113 includes a first surgical tool exchange button 113A, a second surgical tool exchange button 113B, and a clutch button 113C.

[0027] The display device 114 displays 2D or 3D images of the affected area captured by the imaging unit 123 on the operation unit 120 side. Referring to FIG. 1 , the display device 114 is attached near the top of the console unit 110. The display device 114 may be an open-type 2D or 3D monitor with a flat screen or an immersive 2D or 3D viewer into which the operator peers. The console unit 110 is positioned at approximately the same height as the head of an operator seated in a chair. Therefore, the operator can operate the first reader 111 and the second reader 112 with their left and right hands and the foot switch unit 113 with their feet while observing the 2D or 3D image of the affected area displayed on the display device 114. While the display device 114 is integrated with the console unit 110 in the example shown in FIG. 1 , it may be physically separated from the console unit 110 or may be a head-mounted display (HMD) worn on the operator's head.

[0028] 2, the operation unit 120 includes a first follower 121, a second follower 122, an imaging unit 123, a surgical tool exchange unit 124, and a base arm 125. Although not shown, the operation unit 120 further includes a communication device capable of communicating various information with the robot control device 130.

[0029] The base arm 125 may be a single robot arm that commonly supports all of the first follower 121, the second follower 122, the imaging unit 123, and the surgical tool exchange unit 124, or may be composed of multiple robot arms that each support one of these. The degree of freedom configuration of the base arm 125 is not particularly limited. As a modified example, the imaging unit 123 may be supported by a robot arm independent of the base arm 125 (for example, a robot arm other than the base arm 125 that supports the first follower 121 and the second follower 122).

[0030] The first follower 121 is a robot arm that supports a first surgical tool 121A as an end effector at its distal end. It includes an electric motor 121B that drives each joint of the robot arm and a sensor unit 121C that detects the state of the robot arm. The sensor unit 121C detects the state of the robot arm, such as a torque reference value, joint angle, and joint angular velocity of each joint. The torque reference value substantially corresponds to the current value input to the electric motor 121B. The joint angle is obtained from an encoder attached to the output shaft of the electric motor 121B, and the joint angular velocity is obtained by differentiating the joint angle with respect to time. Similarly, the second follower 122 is a robot arm that supports a second surgical tool 122A as an end effector at its distal end. It includes an electric motor 122B that drives each joint of the robot arm and a sensor unit 122C. The robot arms that support the first follower 121 and the second follower 122 are not limited to a specific configuration or structure of degrees of freedom, and therefore a detailed description of the robot arms will be omitted in this specification.

[0031] The first follower 121 and the second follower 122 are movable up and down in conjunction with the movement of the first leader 111 and the second leader 112 by the operation of the operator, and are also movable horizontally, i.e., two-dimensionally, at each height position. In this embodiment, a surgical tool exchange area for automatic exchange of surgical tools is provided in part of the movable area of ​​the first follower 121 and the second follower 122, and a surgical tool exchange waiting area is provided near this surgical tool exchange area, where the followers are moved before exchanging surgical tools and where they wait until the surgical tool is determined.

[0032] In the surgical support robot system 100 according to this embodiment, the operation unit 120 is provided with two followers, but may have three or more followers or only one. To perform leader-follower control, the number of leaders on the console unit 110 side and the number of followers on the operation unit 120 side are the same. The number of followers corresponds to the number of surgical tools that can be used simultaneously in the surgical support robot system 100. In addition, in this embodiment, for the sake of simplicity of explanation, the first follower 121 and the second follower 122 are assumed to have the same configuration, but of course each follower may have a different configuration.

[0033] In response to the operation state of the console unit 110 by the operator, the electric motors 121B and 122B are driven on the operation unit 120 side, and at least one of the positions and postures of the first surgical tool 121A and the second surgical tool 122A of the first follower 121 and the second follower 122 is controlled. The first follower 121 and the second follower 122 preferably have six degrees of freedom for the positions and postures of the first surgical tool 121A and the second surgical tool 122A and a degree of freedom for opening and closing the surgical tools, but may also be devices that can control at least one of the positions and postures of the first surgical tool 121A and the second surgical tool 122A. Furthermore, when the first surgical tool 121A and the second surgical tool 122A are surgical tools such as tweezers or scissors, the first follower 121 and the second follower 122 may be devices that can drive the opening and closing of such surgical tools and do not have to have an arm-like configuration.

[0034] Imaging Unit: The imaging unit 123 is a unit for capturing images of the patient's affected area, etc., and includes a camera 123A, a light source 123B, and a camera arm 123C, as shown in FIG. 2. The camera arm 123C is a robotic arm equipped with an electric motor and an encoder (neither of which are shown) at its joints. It supports the camera 123A and the light source 123B and can move the camera 123A and the light source 123B to predetermined positions by driving the motor. The camera arm 123C supports the camera 123A so that the patient's affected area is within the field of view of the camera 123A and the light emitted by the light source 123B falls on the field of view of the camera 123A. The position and attitude of the camera arm 123C may be configured to be controlled by leader-follower control from the console unit 110 side.

[0035] Surgical Tool Exchange Unit: The surgical tool exchange unit 124 includes a plurality of surgical tools (including a first replacement surgical tool 124A and a second replacement surgical tool 124B) to be exchanged for the first surgical tool 121A and the second surgical tool 122A attached to the first follower 121 and the second follower 122, respectively, a tool turret 124C that holds these plurality of surgical tools, an electric motor 124D that rotates the tool turret 124C, and an encoder 124E that detects the rotational position of the electric motor 124D (or the tool turret 124C). When exchanging the first surgical tool 121A attached to the first follower 121 for the first replacement surgical tool 124A held by the tool turret 124C, the tool turret 124C is rotated using the electric motor 124D and the encoder 124E so as to place the first replacement surgical tool 124A in a predetermined position. The same applies to the case where the second surgical tool 121B attached to the second follower 122 is replaced with the second replacement surgical tool 124B.

[0036] Base Arm: The base arm 125 is a robot arm that supports all of the first follower 121, the second follower 122, the imaging unit 123, and the surgical tool exchange unit 124. The base arm 125 includes an electric motor 125A that drives each joint and a sensor unit 125B that detects the state of the base arm 125. The base arm 125 drives the electric motor 125A in response to an operation state by an operator using the console unit 110, thereby moving the first follower 121, the second follower 122, and the imaging unit 123, which are indicated at the distal end, to predetermined positions. The sensor unit 125B detects the torque reference value, joint angle, joint angular velocity, etc. of each joint as the state of the base arm 125. The torque reference value substantially corresponds to the current value input to the electric motor 125A. The joint angle is obtained from an encoder attached to the output shaft of the electric motor 125A, and the joint angular velocity is obtained by time-differentiating the joint angle.

[0037] 2, the robot control device 130 includes a CPU (Central Processing Unit) 131, a memory unit 132 having a RAM (Random Access Memory) and a ROM (Read Only Memory), a bus 133, and an input / output (IO) unit 134 that converts and inputs / outputs various signals. The components of the CPU 131, the input / output unit 134, and the memory unit 132 are connected via the bus 133 so that signals can be transmitted to one another. The robot control device 130 may have a processing circuit such as a GPU (Graphics Processing Unit), a DSP (Digital Processing Unit), or an ASIC (Application Specific Integrated Circuit) instead of or in addition to the CPU 131. The robot control device 130 may be configured using one or more general computer devices such as personal computers (PCs).

[0038] The CPU 131 functions as an arithmetic processing unit, and reads and executes various programs stored in a memory unit 132 such as a ROM or a RAM, or an external storage device (not shown).

[0039] The RAM or ROM in the memory unit 132 is a computer-readable semiconductor recording medium that records programs and other data for executing control of the surgical assistance robot system 100. Of these, the ROM stores data such as programs and device constants used by the CPU 131 in a non-volatile manner, while the RAM temporarily stores programs used by the CPU 131 and variables that change sequentially during program execution.

[0040] The program causes the CPU 131, the RAM and ROM in the memory unit 132, and the input / output unit 134 to operate cooperatively, causing them to function as an arithmetic processing unit (described later), thereby executing each function of the surgical assistance robot system 100. The program is written in, for example, a low-level language such as machine language or assembly language, or a high-level language such as C language.

[0041] The input / output unit 134 includes a communication device, a D / A converter, an A / D converter, a motor drive circuit, etc. (all of which are not shown). The input / output unit 134 also includes an interface that connects various sensors, such as external devices, actuators, and encoders, to the robot control device 130. However, in Fig. 2, the components within the input / output unit 134 are not shown to simplify the drawing.

[0042] The communication device included in the input / output unit 134 is configured to be able to send and receive signals, etc. between the Internet and other communication devices in accordance with a specified protocol such as TCP / IP, EtherCAT (registered trademark) or EtherNet / IP (registered trademark) used as an industrial network.

[0043] The input / output unit 134 may be configured to connect the robot control device 130 to a storage device that stores various data or a drive device that is a reader / writer for recording media. The storage device may be implemented by, for example, a magnetic storage device such as a hard disc drive (HDD), a semiconductor storage device such as a solid state drive (SSD), an optical storage device, or a magneto-optical storage device.

[0044] 1 and 2, the robot control device 130 is configured as a physically integrated unit (i.e., a single housing) for the control devices for the console unit 110, the operation unit 120, and the imaging unit 123. As a variation, a device similar to the robot control device 130 may be configured by physically separating dedicated control devices for each unit (i.e., each as a separate housing), and the control devices may be connected to each other by wire or wirelessly, or a device similar to the robot control device 130 may be configured by a collection of multiple control devices.

[0045] 4 shows the external structure of the robot control device 130 that is physically integrated with the operation unit 120. In the example shown in FIG. 4, a housing 400 of the robot control device 130 that houses the above-mentioned components 131 to 134 is connected at its top surface 401 to a support column 411 at the base of the base arm 125 of the operation unit 120. In addition, the bottom of the housing 400 of the robot control device 130 is provided with a plurality of (e.g., four) legs 402 with wheels 403 as a means of transportation. Therefore, an operator or an assistant can manually push the housing 400 of the robot control device 130 and the operation unit 120 to move them in and out of the operating room.

[0046] The housing 400 of the robot control device 130 and the operation unit 120 may be physically separated. In this case, the housing 400 of the robot control device 130 may be fixedly installed in a predetermined location inside or outside the operating room. The operation unit 120, which is separate from the robot control device 130, may be directly attached and fixed to the floor of the operating room, or may have wheeled legs (not shown) at the bottom ends of the support columns 411 as a means for movement. As yet another variation, the robot control device 130 may be physically integrated with the console unit 110 instead of the operation unit 120.

[0047] External Network: The external network 160 is a wired or wireless transmission path for information transmitted from devices connected to the network. The network referred to here may include public networks such as the Internet, telephone networks, and satellite communication networks, as well as various LANs (Local Area Networks) and WANs (Wide Area Networks) including Ethernet (registered trademark). The network may also include dedicated line networks such as IP-VPN (Internet Protocol-Virtual Private Network).

[0048] The robot control device 130 is connected to an external network 160 via a communication device (described above) within the input / output unit 134. The robot control device 130 may share data regarding surgical information such as the surgical tools used and the operating status of the surgical support robot system 100 with multiple other surgical support robot systems and upper control devices (not shown) that are connected to each other via the external network 160. The upper control device may be, for example, a PC, various servers, or a combination of these, and monitors the operating status of each surgical support robot system and performs statistical data analysis.

[0049] External operation terminal: The external operation terminal 140 (not shown in FIG. 1) is externally connected to the robot control device 130, and is configured to receive and display the operating status of the surgical support robot system 100, as well as to allow an assistant to input commands and set parameters. As shown in FIG. 2, the external operation terminal 140 includes an input unit 141 and a display unit 142.

[0050] The input unit 141 consists of, for example, one or a combination of two or more of a keyboard, mouse, button, switch, infrared or other remote control means using radio waves, and is configured to be operated by the operator and assistant to input commands to the surgical support robot system 100 (for example, instructions to attach the initial surgical tools when starting up the system) and to set parameters adjusted for each operator.

[0051] The display unit 142 receives and displays the operating status of the surgical support robot system 100 from the robot control device 130, allowing the operator and assistant to visually and intuitively recognize this information. The display unit 142 may be a display device such as a liquid crystal panel or a lamp, or may be a sound device that displays information by warning sound or voice, and may be set to issue a warning when a system error occurs, for example. The display unit 142 may also be configured as an integrated unit with an input unit, like a tablet terminal, and may be equipped with an application for input and status display.

[0052] 1 and 2, the main components of the surgical support robot system 100 have been described, but each component may be configured using general-purpose parts or may be configured using hardware specialized for the function of each component. The configuration used for each component can be changed as appropriate depending on the technical level at the time of implementing the present disclosure.

[0053] B. Functional Configuration of the Robot Control Device In the above section A, it was mentioned that the CPU 131, the RAM and ROM in the memory unit 132, and the input / output unit 134 operate cooperatively by executing a program within the robot control device 130 to function as the arithmetic processing unit of the surgery support robot system 100. In this section B, the functional configuration of the robot control device 130 will be described.

[0054] 3 shows an example of the functional configuration of the robot control device 130. In the example shown, the robot control device 130 includes a calculation processing unit 310 and an input / output interface 320. The calculation processing unit 310, as the calculation processing unit of the surgery support robot system 100, includes an input / output processing unit 311, a drive control unit 312, an image generation unit 313, and a display control unit 314.

[0055] The input / output processing unit 311 acquires signals output from the camera 123A, sensors, etc., performs signal processing such as filtering, and outputs the signals to each functional unit. The input / output processing unit 311 also outputs the outputs from each functional unit to other system components such as actuators.

[0056] The drive control unit 312 controls the motion of the first follower 121 and the second follower 122 by leader-follower control based on the operation performed by the operator on the first leader 111 and the second leader 112. This makes it possible to change the position and posture of the first surgical tool 121A and the second surgical tool 122A attached to the first follower 121 and the second follower 122, respectively (when a surgical tool that performs opening and closing operations is used, the leader-follower control is also performed for the opening and closing operations). Furthermore, the drive control unit 312 controls the motion of the first leader 111 and the second follower 112 based on sensor information from the first follower 121 and the second follower 122, thereby presenting a haptic sensation to the operator. The drive control unit 312 may apply a bilateral control method so that the operator can remotely operate the console unit 110 while recognizing the state of the operation unit 120, but details of the leader-follower control will be omitted. The drive control section 312 is also configured to drive and control the camera arm 123C to move the imaging unit 123 and change the field of view of the camera 123A.

[0057] The image generation unit 313 generates a surgical tool selection UI, which is a GUI (Graphical User Interface) for selecting a surgical tool, in accordance with input from the operator and the state of the surgical support robot system 100. Details of the configuration of the surgical tool selection UI and operations on this UI will be described later. In addition, in order to present meaningful replacement candidate surgical tools on this surgical tool selection UI, the image generation unit 313 calculates the priority of surgical tools that are replacement candidates for the first surgical tool 121A and the second surgical tool 122A attached to the first follower 121 and the second follower 122, respectively; details of the calculation method will also be described later.

[0058] The display control unit 314 superimposes the image of the patient's affected area captured by the camera 123A on the surgical tool selection UI generated by the image generation unit 313, and displays them on the screen of the display device 114 of the console unit 110 or the display unit 142 of the external operation terminal 140. The display control unit 314 displays the GUI so as not to interfere with viewing the image of the affected area captured by the camera 123A, and details will be given later. The display control unit 314 may also include a function for adjusting the imaging magnification of the camera 123A and the light source 123B.

[0059] Some or all of the functions for executing the processing of each of the functional modules 311 to 314 of the arithmetic processing unit 310 can be realized using a dedicated hardware circuit, an integrated circuit customized for a specific application such as an ASIC, or a programmable logic circuit such as an FPGA (Field Programmable Gate Array). Furthermore, there is no particular limit to the number of computers (such as PCs) that execute the programs related to the processing in the arithmetic processing unit 310. For example, multiple computers, including a higher-level control device, may cooperate with each other to execute the programs related to the processing in the arithmetic processing unit 310.

[0060] Furthermore, the storage device storing various programs for realizing the functions of the arithmetic processing unit 310 may be, but is not limited to, a memory incorporated in the robot controller 130. The programs may be recorded in a computer-readable storage device that is distributed separately from the robot controller 130 to provide the programs. Alternatively, the programs may be downloaded via a network such as the external network 160 and installed or updated in a computer that operates as the robot controller 130.

[0061] In this embodiment, the surgical support robot system 100 has two operating modes: a procedure mode and a surgical tool exchange mode. In the procedure mode, a surgical operation is performed by driving the first follower 121 and the second follower 122, each equipped with a surgical tool, on the operation unit 120 side in conjunction with the operation of the first leader 111 and the second leader 112 by an operator on the console unit 110 side through leader-follower control. In the surgical tool exchange mode, the leader-follower control is interrupted, and the surgical tool of at least one of the first follower 121 and the second follower 122 is exchanged through a surgical tool exchange sequence (described later).

[0062] The operator can instruct a transition of the operation mode by performing a predetermined operation on the foot switch unit 113, the first reader 111, the second reader 112, etc. The robot control device 130 performs processes for controlling leader-follower control in the procedure mode, transition of the operation mode in response to an operation by the operator, presentation of a surgical tool selection UI (described later) in the surgical tool exchange mode, execution of a surgical tool exchange sequence for a selected surgical tool, etc.

[0063] The surgical support robot system 100 may have further operating modes other than the procedure mode and the surgical tool exchange mode, but as these are not directly related to the present disclosure, their description will be omitted in this specification.

[0064] C. Structure of the Operation Unit As explained in Section A above with reference to Fig. 2, the operation unit 120 includes a first follower 121, a second follower 122, an imaging unit 123, a surgical tool exchange unit 124, and a base arm 125. In this Section C, the structure of the operation unit 120 will be explained in detail.

[0065] 4 shows the external structure of the operation unit 120 that is physically integrated with the robot control device 130. The operation unit 120 is made up of a base arm 125 that commonly supports at its distal end the first follower 121, second follower 122, imaging unit 123, and surgical tool exchange unit 124, which are the main components shown in FIG.

[0066] The base arm 125 is a single robot arm having a multi-link structure in which, from the base side, a support 411, a first link 412, and a second link 413 are connected. The support 411 is a columnar member extending in the vertical direction, and its base is connected to the top surface 401 of the housing 400 of the robot control device 130. The support 411 may be configured to be extendable and contractible in the vertical direction. The first link 412 is connected to the upper end of the support 411 at its base side via a first joint 421. The second link 413 is connected to the distal end of the first link 412 at its base side via a second joint 422, and at its distal end, commonly supports the first follower 121, the second follower 122, the imaging unit 123, and the surgical tool exchange unit 124, which are main components of the operation unit 120, via a third joint 423.

[0067] The first joint 421, the second joint 422, and the third joint 423 have degrees of freedom of rotation about yaw axes 421A, 422A, and 423A, respectively, but may also have degrees of freedom of rotation about other axes. In addition, one or more links may be further connected to the distal end side of the second link 413, and the distal end of the most distal link may be configured to support the first follower 121, the second follower 122, the imaging unit 123, and the surgical tool exchange unit 124.

[0068] The first joint 421, the second joint 422, and the third joint 423 are assumed to be active joints that are rotationally driven by electric motors, but some or all of the joints may be passive joints. An operator or an assistant may manually operate the rotation angles of the passive joints to adjust the positions and attitudes of the first follower 121, the second follower 122, and the imaging unit 123. However, the base arm 125 may include a passive joint equipped with a locking mechanism, and may be a vertical multi-joint type or a horizontal multi-joint type, etc.

[0069] 2, for simplicity, the base arm 125 is illustrated in an abstract manner as including an electric motor 125A that drives each of the joints 421 to 423 and a sensor unit 125B that detects the state of the base arm 125. The sensor unit 125B includes an encoder that detects the rotational position of each joint. The base arm 125 drives the electric motor 125A in accordance with the state of operation by the operator at the console unit 110, and can move the first follower 121, the second follower 122, and the imaging unit 123, which are indicated at the distal end, to predetermined positions.

[0070] FIG. 5 shows an enlarged view of the structure supporting the first follower 121, the second follower 122, the imaging unit 123, and the surgical tool exchange unit 124 on the distal end side of the base arm 125 (second link 413).

[0071] The first follower 121 includes a first mounting part 501 to which a first surgical tool 121A serving as an end effector is replaceably mounted, and a first robot arm 511 that supports at its distal end the first mounting part 501. As shown in Fig. 2, the first follower 121 includes an electric motor 121B that drives each joint of the first robot arm 511, and a sensor part 121C that detects the state of the first robot arm 511. The sensor part 121C includes an encoder that detects the joint angle.

[0072] Similarly, the second follower 122 includes a second robot arm 512 that supports, at its distal end, a second mounting portion 502 to which a second surgical tool 122A is replaceably attached, as shown in Fig. 5. In addition, as shown in Fig. 2, the second follower 122 includes an electric motor 122B that drives each joint of the second robot arm 512, and a sensor unit 122C that detects the state of the second robot arm 512.

[0073] 5, the first robot arm 511 and the second robot arm 512 have the same configuration, but of course they may have different configurations for each follower. The bases (or roots) of the first robot arm 511 and the second robot arm 512 are each connected to the distal end of the base arm 125.

[0074] The first surgical tool 121A and the second surgical tool 122A are, for example, tweezers, a needle holder, scissors, an electric scalpel, or any other type of treatment tool. The first mounting portion 501 and the second mounting portion 502 are equipped with a mechanism for attaching and detaching the surgical tools, and are configured so that the first surgical tool 121A and the second surgical tool 122A can be attached and detached by driving the actuators, respectively. When surgical tools that open and close, such as tweezers, a needle holder, or scissors, are attached as the first surgical tool 121A and the second surgical tool 122A, the first follower 121 and the second follower 122 can drive such surgical tools to open and close.

[0075] In response to the operation states of the first reader 111 and the second reader 112 by the operator on the console unit 110, the electric motors 121B and 122B are driven on the operation unit 120 side, and at least one of the positions and postures of the first surgical tool 121A and the second surgical tool 122A of the first follower 121 and the second follower 122 is controlled, respectively. It should be understood that there is interlocking between the first leader 111 and the first follower 121, and between the second leader 112 and the second follower 122, so that when the leader is operated, the followers move cooperatively. It is preferable that the first robot arm 511 and the second robot arm 512 have six degrees of freedom for the position and posture of the first surgical tool 121A and the second surgical tool 122A, respectively, and six degrees of freedom for opening and closing the surgical tools, but they may also be devices that can control at least one of the positions and postures of the first surgical tool 121A and the second surgical tool 122A.

[0076] The imaging unit 123 is a unit for capturing images of the patient's affected area, etc., and as shown in Figure 5, it is equipped with a camera 123A that can capture moving or still images, a light source 123B that illuminates the field of view of the camera 123A, and a camera arm 123C.

[0077] The camera 123A is configured, for example, using an RGB camera and is an imaging device such as a 3D endoscope or microscope capable of capturing three-dimensional images, but various known configurations can be applied. The camera 123A has a zoom mechanism and is capable of changing the imaging magnification. The light source 123B is configured, for example, by a light source such as an LED (Light Emitting Diode) or laser, and provides illumination light when the camera 123A captures an image of the surgical site. However, the configurations of the camera 123A and the light source 123B are not particularly limited as long as they can acquire video or images and output that information, and therefore detailed description thereof will be omitted in this specification.

[0078] The video and still image data acquired by the camera 123A is transmitted to the robot control device 130, where it undergoes various common image processing steps to display the video and still images. The video and still image data after image processing is transmitted to the display device 114 of the console unit 110, the display unit 142 of the external operation terminal 140, or another display monitor (not shown) installed in the operating room, where it is displayed as an image of the surgical site. However, the various image processing steps do not necessarily have to be performed by the robot control device 130; a processing circuit for image processing may be provided within the imaging unit 123, and the display data after image processing may be transmitted to the display monitor.

[0079] The camera arm 123C is a robotic arm equipped with electric motors and encoders (neither of which is shown) at its joints. It supports the camera 123A and the light source 123B at its distal end, and can move the camera 123A and the light source 123B to predetermined positions by driving the motors. As can be seen from FIG. 5 , the camera arm 123C has a base (or root) connected to the distal end of the base arm 125 (strictly speaking, connected to the distal end of the second link via the third joint 423), and supports the camera 123A and the light source 123B facing downward so that the patient's affected area is within the field of view of the camera 123A. In this embodiment, all of the joints of the camera arm 123C are active joints driven by electric motors, but some or all of the joints may be changed to passive joints as appropriate. The operator or assistant may manually adjust the rotation angle of the passive joints. However, since the present disclosure is not limited to a specific structure of the camera arm 123C, a detailed description of the structure of the camera arm 123C will be omitted.

[0080] In this embodiment, the imaging unit 123 is attached to the operation unit 120 together with the first follower 121 and the second follower 122, but it may also be attached to a robot arm separate from the operation unit 120, or may be placed in another position that allows a bird's-eye view of the patient's affected area and its surroundings.

[0081] 2, the surgical tool exchange unit 124 includes a plurality of surgical tools (including a first replacement surgical tool 124A and a second replacement surgical tool 124B) that are replaceable with the first surgical tool 121A and the second surgical tool 122A that are attached to the first follower 121 and the second follower 122, respectively, a tool turret 124C that holds the plurality of surgical tools, an electric motor 124D that rotates the tool turret 124C, and an encoder 124E that detects the rotational position of the electric motor 124D (or the tool turret 124C). As shown in Fig. 5, the base of the surgical tool exchange unit 124 is connected to the base arm 125 (in the example shown in Fig. 5, it is connected to the distal end of the second link via the third joint 423 together with the camera arm 123C). The surgical tool replacement unit 124 is installed within a range in which the first follower 121 and the second follower 122 can move to replace surgical tools, but it is preferable that it be installed near the first follower 121 and the second follower 122 to shorten the replacement time.

[0082] The tool turret 124C is, for example, fan-shaped, and when replacing a surgical tool, it rotates by driving the electric motor 124D in conjunction with the operation of the follower, and is configured to position the first replacement surgical tool 124A or the second replacement surgical tool 124B to be replaced at a predetermined rotational position.

[0083] 6 to 11 show a series of operations of the second follower 122 and the surgical tool exchange unit 124 when exchanging a surgical tool. In Fig. 6 to 11, the second exchange surgical tool 124B to be exchanged is shown filled in gray. For convenience of drawing, the structure supporting the first follower 121 and the second follower 122 in Fig. 6 to 11 is different from that in Fig. 4, and the imaging unit 123 is not shown.

[0084] Fig. 6 shows a state in which surgery is being performed using the second surgical tool 122A attached to the second follower 122. When replacing the second surgical tool 122A, the drive control unit 312 first moves the second follower 122 to a surgical tool replacement standby area closer to the surgical tool replacement unit 124, as shown in Fig. 7. Figs. 28 and 29 show the surgical tool replacement areas set in the ranges of motion of the first follower 121 and the second follower 122 at the distal end of the base arm 125, as viewed from above and from the front, respectively. In Figs. 28 and 29, a surgical tool replacement standby area 2801 for the first follower 121 and a surgical tool replacement standby area 2802 for the second follower 122 are indicated by hatched areas, respectively. 29 corresponds to the operator interrupting the procedure and safely withdrawing from the affected area of ​​the patient. A surgical tool exchange sequence may be initiated for the follower in such a situation (see Section H below).

[0085] Here, the surgical tool exchange standby area is an area to which the follower is moved before exchanging a surgical tool and where it waits until the surgical tool is determined. By moving the follower to the surgical tool exchange standby area before exchanging a surgical tool, the time from deciding on the surgical tool to actually completing the surgical tool exchange can be shortened. The surgical tool exchange standby area is preferably set within the range of motion of the follower, far enough from the affected area to ensure safety during tool exchange, and near the surgical tool exchange unit 124 (tool turret 124C).

[0086] Next, the drive control unit 312 moves the second follower 122 to the surgical tool exchange area, as shown in Fig. 8. At the same time, the drive control unit 312 rotates the tool turret 124C so that the second follower 122 is in a position where it is easy to remove the second replacement surgical tool 124B. Then, when the movement of the second follower 122 is complete, the drive control unit 312 removes the second surgical tool 122A from the second mounting unit 502 and returns it to the tool turret 124C, as shown in Fig. 9.

[0087] Next, as shown in Fig. 10, the drive control unit 312 attaches the second replacement surgical tool 124B from the tool turret 124C to the second mounting unit 502, and then moves the second follower 122 to its original position, completing the surgical tool exchange, as shown in Fig. 11. Thereafter, the surgery is resumed using the second replacement surgical tool 124B.

[0088] D. Structure of the Console Unit As explained in Section A above with reference to Figure 2, the console unit 110 includes a first reader 111, a second reader 112, a foot switch unit 113, and a display device 114, and the operator operates the console unit 110 to instruct the surgery support robot system 100 to perform certain operations. In this Section D, the structure of the console unit 110 will be explained in detail.

[0089] 12 shows the external configuration of the console unit 110. The console unit 110 is a structure that is approximately L-shaped when viewed from the side, and has a bottom portion 1201 at its lowest end that is U-shaped when viewed from above, with a base portion 1202 connected to the center of the bottom portion 1201 in an approximately vertical direction.

[0090] An O-shaped or ring-shaped support portion 1203 is attached near the middle of the base portion 1202. The support portion 1203 is at approximately the same height as the elbows of an operator sitting on a chair 1204. However, the height of the chair 1204 may be adjusted so that the support portion 1203 is at approximately the same height as the elbows of the operator.

[0091] A display device 114 is attached to the upper end of the base unit 1202. In the example shown in Fig. 12, the display device 114 is an immersive 2D and 3D viewer into which the operator peers. Therefore, the operator can operate the first reader 111 and the second reader 112 with their left and right hands, and operate the foot switch unit 113 with their feet, while peering into the screen of the display device 114 and observing a 2D or 3D image of the affected area. Although the foot switch 113 is not shown in Fig. 12, the foot switch 113 may be installed, for example, inside the U-shaped base unit 1202.

[0092] FIG. 13 shows the support unit 1203 as seen from the side of an operator sitting on a chair 1204. A first reader 111 and a second reader 112 are disposed near the base of the support unit 1203. The first reader 111 and the second reader 112 are readers that the operator operates using the left and right hands, respectively, and have the same structure and are symmetrical. As can be seen from FIG. 12, the operator can use the front edge of the ring-shaped support unit 1203 as a hand rest or wrist rest 1205, and can rest both arms or elbows on the support unit 1203 to operate the first operating unit 111C at the tip of the first reader 111 and the second operating unit 112C at the tip of the second reader 112 using the left and right hands.

[0093] 13 , the first reader 111 and the second reader 112 are each configured as an articulated robot arm, and support a first operating unit 111C corresponding to right-handed operation and a second operating unit 112C corresponding to left-handed operation at the distal end of each robot arm. However, since the technical scope of the present disclosure is not limited to the structure of a specific robot arm, detailed description of the robot arm will be omitted in this specification. Also, in this embodiment, for the sake of simplicity, the first reader 111 and the second reader 112 have the same configuration, but of course, each reader may have a different configuration.

[0094] The first reader 111 and the second reader 112 support, at their distal ends, a first operating unit 111C for right-handed operation and a second operating unit 112C for left-handed operation. As shown in Fig. 2, the first reader 111 and the second reader 112 are equipped with sensor units 111B and 112B, respectively, that detect the state of the robot arm. The first reader 111 and the second reader 112 are configured to compensate for their own weight through mechanical or motor control, so that they will not fall even if the operator lets go of them.

[0095] There is interlocking between the first leader 111 and the first follower 121, and between the second leader 112 and the second follower 122 (as described above). Therefore, leader-follower control is possible, and in accordance with the operation state of the first leader 111 and the second leader 112 by the operator on the console unit 110, the electric motors 121B and 122B are driven on the operation unit 120 side, and the positions and attitudes of the first surgical tool 121A and the second surgical tool 122A of the first follower 121 and the second follower 122, respectively, are controlled.

[0096] The electric motors 111A and 112A are driven by control signals from the robot control device 130 so as to perform a target operation based on the position detection results converted into electric signals. However, in this embodiment, the robot arms serving as the first reader 111 and the second reader 112 may include passive joints, and may be of a vertically multi-joint type or a horizontally multi-joint type.

[0097] The first operation unit 111C and the second operation unit 112C are, for example, composed of a pair of grips rotatably supported by a grip shaft and operable to open and close, and can open and close the grippers by driving the electric motors 111A and 112A, respectively. Therefore, the first operation unit 111C and the second operation unit 112C can present haptic sensations to the operator's right and left hands by driving the electric motors 111A and 112A based on control signals input from the robot control device 130. Presenting haptic sensations here means transmitting force information to the operator's right and left hands so that the operator can approximate the forces acting on the first follower 121 and the second follower 122, respectively, and the results of their operation inputs.

[0098] The foot switch unit 113 is configured so that the operator can use their feet to operate the operation unit 120 and activate the functions of the operation unit 120. As shown in FIG. 2, the foot switch unit 113 includes a first surgical tool change button 113A, a second surgical tool change button 113B, and a clutch button 113C. Although the presence of the foot switch unit 113 is difficult to see in FIG. 12, it is located inside the U-shape of the bottom 1201. Each button 113A to 113C is located on the top surface of the foot switch unit 113, so that an operator sitting in a chair 1204 can press the buttons with their feet. FIG. 14 shows the external configuration of the top surface of the foot switch unit 113 so that the arrangement of each button 113A to 113C can be seen.

[0099] The clutch button 113C is operated to execute the clutch function. Specifically, the clutch button 113C is used to collectively interrupt the interlocking between the first leader 111 and the first follower 121 and the interlocking between the second leader 112 and the second follower 122, and to stop the operation of the first surgical tool 121A and the second surgical tool 122A on the operation unit 120 side. While the clutch button 113C is being operated, the first follower 121 and the second follower 122 will not operate even if the first leader 111 and the second reader 112 are operated.

[0100] For example, when an operator operates the first leader 111 and the second leader 112 and either the first leader 111 or the second leader 112 reaches near the end of the range in which they can move, the operator can operate the clutch button 113C to temporarily disconnect the link between the first leader 111 and the first follower 121, and between the second leader 112 and the second follower 122, and return the first leader 111 and the second leader 112 to near the center of the range in which they can move.

[0101] When the operation of the clutch button 113C is stopped, the leader-follower control is resumed, and the operator can resume the operation of the first operation unit 111C of the first leader 111 and the second operation unit 112C of the second leader 112 near the center of the movable range. Note that, in this embodiment, the configuration is such that the operation of one clutch button 113C can temporarily disconnect two sets of leader-follower control collectively, but the configuration may also be such that the leader-follower control between the first leader 111 and the first follower 121, and between the second leader 112 and the second follower 122 can be temporarily disconnected individually.

[0102] The first surgical tool change button 113A is operated to execute the surgical tool change function of the first follower 121. Specifically, by operating the first surgical tool change button 113A, the link between the first leader 111 and the first follower 121 is released, and the operation of the first follower 121 is switched to an operation for executing the surgical tool change function. Similarly, the second surgical tool change button 113B is operated to execute the surgical tool change function of the second follower 122, and the link between the second leader 112 and the second follower 122 is released, and the operation of the second follower 122 is switched to an operation for executing the surgical tool change function.

[0103] 2 and 14 , the configuration of the foot switch unit 113 is not limited to the examples shown in FIGS. 2 and 14 , and other switches may be included. The switch may be a button or another mechanical operator such as a joystick. For example, the foot switch unit 113 may further include a switch for switching on and off the supply of high-frequency current to the surgical tool, a camera switch for switching the operation of the camera arm, and the like. For example, when the camera switch is operated, the camera arm 123C may be driven by a follower operation by the operator, so that at least one of the position and posture of the camera 123A can be changed. Furthermore, the first surgical tool change button 113A and the second surgical tool change button 113B may be located on the first reader 111 and the second reader 112, respectively, rather than on the foot switch unit 113.

[0104] E. Surgical Tool Exchange Sequence and User Interface (First Embodiment) This section E describes the operation of the surgical support robot system 100 according to the first embodiment. Figure 15 shows the operation of the surgical support robot system 100 in the form of a flowchart. This operation is mainly performed by the arithmetic processing unit 310 (see Figure 3) in the robot control device 130. Below, the operation of the surgical support robot system 100 will be described with reference to Figure 15.

[0105] First, the drive control unit 312 determines the state of the clutch button 113C (step S1501). If the clutch button 113C is not pressed (No in step S1501), the leader-follower control continues (step S1502). If the clutch button 113C is released (No in step S1501), the leader-follower control resumes (step S1502).

[0106] On the other hand, when the clutch button 113C is pressed (Yes in step S1501), the drive control unit 312 temporarily suspends the leader-follower control (step S1503). After that, the operator may perform a normal clutch button operation to correct the leader position (to return the leader that has reached the end of its movable range to the center of its movable range), or may perform an input related to replacing a surgical tool.

[0107] The image generation unit 313 calculates the order in which replacement candidate surgical tools are to be displayed in the surgical tool selection UI (step S1504). That is, the image generation unit 313 calculates, based on a predetermined algorithm, the surgical tool that is likely to be replaced from among the surgical tools that are replacement candidates for the surgical tool currently attached to the target follower.

[0108] For example, the image generation unit 313 records the order in which surgical tools are replaced for each operator and estimates the surgical tool that is likely to be replaced based on past patterns of frequency. Figure 16 illustrates an example of a surgical tool replacement pattern when three types of surgical tools are used and three time-series data are used. Based on the data for the currently used surgical tool and the surgical tool used previously, two patterns are predicted for the next tool that may be replaced. The example shown in Figure 16 illustrates a case in which a first replacement pattern in which the surgical tools are used in the order of surgical tool A → surgical tool B → surgical tool C, and a second replacement pattern in which the surgical tools are used in the order of surgical tool A → surgical tool B → surgical tool A, are predicted. If the frequency of the first replacement pattern is high, it can be determined that the surgical tool most likely to be replaced after surgical tool B is surgical tool C.

[0109] The image generation unit 313 may use an algorithm other than that for calculating replacement candidate surgical tools based on past surgical tool replacement patterns. For example, the image generation unit 313 may calculate the priority of each replacement candidate surgical tool using a machine learning model that has been deep-learned to infer replacement candidate surgical tools from currently used surgical tools and the surgical scene (or task).

[0110] When the image generation unit 313 completes the series of calculations in step S1504, it generates a surgical tool selection UI that displays a list of surgical tools that are candidates for replacement in the calculated display order. Then, the display control unit 314 displays the surgical tool selection UI generated by the screen generation unit 313 on the screen of the display device 114 (step S1505).

[0111] Specifically, the display control unit 314 displays, on the screen of the display device 114, the surgical tool selection UI of the follower operated by the operator, either the first follower 121 or the second follower 122, and a mouse pointer that moves in conjunction with the operating follower. In the example shown in FIG. 16 , based on the first replacement pattern determined to be high frequency, a list of replacement candidate surgical tools is displayed on the screen in the order of surgical tool C, surgical tool A, and surgical tool B, and the mouse pointer is displayed on surgical tool C, which is likely to be replaced next in this first replacement pattern. Note that the movement of the mouse pointer is not limited to a two-dimensional movement on a plane. For example, if the display device 114 is equipped with a 3D viewer, the screen display in step S1505 may reflect the movement in three-dimensional space of the leader corresponding to the follower operated by the operator.

[0112] It should be noted that in the surgical support robot system 100 according to this embodiment, the operator can display the surgical tool selection UI on the screen and smoothly start the surgical tool exchange sequence simply by operating the clutch button 113C with his / her foot. The surgical tool selection UI displays a list of surgical tools that are expected to be exchanged, so the operator can quickly select the surgical tool they want to exchange after operating the clutch button 113C. In this embodiment, it can be said that the function of displaying the surgical tool selection UI is assigned to the clutch button 113C.

[0113] The configuration of the screen for displaying the surgical tool selection UI will now be described with reference to the drawings. FIG. 17 shows an example of the configuration of the display screen of the display device 114. The illustrated display screen 1700 is divided into a first display area 1701 near the center of the screen, a second display area 1702 adjacent to the outside of the first display area 1701, and a third display area 1703 at the outermost periphery of the screen. The first display area 1701 always displays images of the patient's affected area captured by the camera 123A during surgery, and the display of UI parts is prohibited. Furthermore, the second display area 1702 can partially display UI parts when the surgical tool is stopped by operating the clutch button 113C, for example. The third display area 1703 is where the UI parts reside because it is less likely to interfere with the viewing of images of the affected area, etc.

[0114] During the surgical tool replacement sequence, a surgical tool selection UI is arranged as one of the UI parts in the third display area 1703. In the example shown in FIG. 17 , a first surgical tool selection UI section 1711 for selecting a surgical tool to be replaced with the first follower 121 for right-hand operation and a second surgical tool selection UI section 1712 for selecting a surgical tool to be replaced with the second follower 122 for left-hand operation are arranged near the right and left upper ends of the third display area 1703, respectively. Both the first surgical tool selection UI section 1711 and the second surgical tool selection UI section 1712 display a list of replaceable surgical tools in a pull-down menu format. The first surgical tool selection UI section 1711 and the second surgical tool selection UI section 1712 are shown enlarged near the left and right upper ends of FIG. 17 . The menu item of the provisionally selected surgical tool in the pull-down menu is highlighted. In the example shown in FIG. 17, "Micro scissors" in the pull-down menu of the second surgical tool selection UI section 1712 is provisionally selected and highlighted.

[0115] Near the bottom right and left ends of the third display area 1703, there are arranged, as other UI parts, a first surgical tool display UI section 1713 and a second surgical tool display section 1714, which display the surgical tools currently attached to the first follower 121 and the second follower 122, respectively. Near the bottom left and right ends of Fig. 17, the first surgical tool display UI section 1713 and the second surgical tool display section 1714 are shown enlarged. In the example shown in Fig. 17, the first surgical tool display UI section 1713 displays the surgical tool "Micro scissors" currently attached to the first follower 121, and the second surgical tool display section 1714 displays the surgical tool "Tweesors-L" currently attached to the second follower 122.

[0116] Near the bottom center of the third display area 1703, there are disposed, as UI parts that are always displayed, an imaging magnification setting UI section 1715 and an operation magnification setting UI section 1716 for setting or changing the imaging magnification of the imaging unit 123 and the operation magnification in leader-follower control, respectively. It is preferable that the imaging magnification and operation magnification can be flexibly adjusted according to the type of surgery, such as laparoscopic surgery, brain surgery, or ophthalmic surgery, and the phase of each surgery (the task to be performed). The imaging magnification setting UI section 1715 and the operation magnification setting UI section 1716 are each shown enlarged near the bottom center of FIG.

[0117] During the surgical tool replacement sequence, leader-follower control is suspended for the follower to be replaced, and the corresponding leader is disassociated with the follower and operates as a pointing device instead. Therefore, by operating this leader, the operator can move the mouse pointer on the screen 1700 and select the surgical tool to be replaced from the pull-down menus of the first surgical tool selection UI unit 1711 and the second surgical tool selection UI unit 1712. Menu items in the pull-down menus that are moused over may be highlighted using various techniques, such as changing color, flashing, or animation. Additionally, menu items corresponding to currently attached surgical tools or tools that cannot be replaced may be grayed out to indicate that they cannot be selected. In addition to the mouse-over method described above, surgical tool selection in the pull-down menus may also be performed using a dial. In the example shown in FIG. 17 , the mouse pointer is positioned on the tentatively selected menu item "Micro Scissors" in the pull-down menu of the second surgical tool selection UI unit 1712.

[0118] The drive control unit 312 may drive each electric motor so that the operable direction of the reader used as a pointing device is restricted to one direction, forward and backward, for selecting a surgical tool, or two directions, including up and down for changing the settings of the imaging magnification and the operating magnification. This allows the operator to move the mouse cursor on the surgical tool selection UI more accurately, and enables the operator to select a surgical tool more quickly and smoothly.

[0119] UI parts required for specific operations, such as the first surgical tool selection UI section 1711 and the second surgical tool selection UI section 1712, may be displayed using the second display area 1702 in addition to the third display area 1703. In the example shown in FIG. 17 , parts of the first surgical tool selection UI section 1711 and the second surgical tool selection UI section 1712 are arranged in the second display area 1702.

[0120] The first surgical tool selection UI unit 1711 and the second surgical tool selection UI unit 1712 display a plurality of replaceable surgical tools, for example, in a pull-down menu, in order of likelihood of being replaced next, on the screen 1700 of the display device 114. Therefore, even if the number of surgical tools increases, the operator can make a selection without spending a lot of time using the pull-down menu that places the surgical tool predicted to be replaced next at the top.

[0121] The screen 1700 shown in Fig. 17 is configured so that the UI parts 1711 to 1716 are arranged outside the first display area 1701 in the center of the screen, preventing the UI parts from obscuring the surgical field and interfering with the operator's procedure. Considering the surgical field and the path of the tube in the image of the patient's affected area captured by camera 123A, the arrangement of the UI parts 1711 to 1716 as shown in Fig. 17 is preferable. Furthermore, because surgical tools are often located from the bottom right to the center in the image of the affected area, it is more preferable to arrange the first surgical tool selection UI section 1711 and the second surgical tool selection UI section 1712 in the upper part of the screen 1700.

[0122] Here, multiple layers may be provided from the front side to the back side of the screen 1700, and the arrangement of UI parts may be managed for each layer, with UI parts arranged in the front layer being superimposed on UI parts arranged in the back layer and displayed simultaneously on the screen 1700. Furthermore, at least some of the UI parts may be displayed semi-transparently so as not to obscure the surgical field and interfere with the operator's procedure.

[0123] For example, UI parts required for specific operations such as surgical tool selection, such as the first surgical tool selection UI unit 1711 and the second surgical tool selection UI unit 1712, may be positioned closer to the operator. Furthermore, UI parts that are always displayed, such as the imaging magnification setting UI unit 1715 and the operation magnification setting UI unit 1716, or an image of the affected area captured by the camera 123A, may be displayed further back than the UI parts required for specific operations, or some of the UI parts may be displayed semi-transparently. Displaying UI parts such as the first surgical tool selection UI unit 1711 and the second surgical tool selection UI unit 1712 semi-transparently prevents them from obscuring the surgical field and interfering with the operator's procedure. Figure 18 shows how the surgical tool selection UI unit 1801, a UI part required for specific operations such as surgical tool selection, is positioned closer to the operator than the operation magnification setting UI unit 1802, a UI part that is always displayed. 19 also shows how an operation magnification setting UI section 1902, which is a UI part that is always displayed, is positioned further back than a surgical tool selection UI section 1901, which is a UI part required for specific operations. The follower, who is not changing surgical tools, may be grasping the affected tissue. For this reason, from a safety standpoint, it is preferable to position UI parts on the screen so as not to obstruct the operator's observation of the affected area as much as possible.

[0124] The display positions of the UI parts are not particularly limited in the second display area 1702 and the third display area 1703. For example, the display control unit 314 may be configured to recognize the position of the surgical tool displayed on the screen (e.g., object recognition) and automatically change the display positions of the UI parts so as not to obstruct the visibility of the surgical tools. Figure 20 shows how, when surgical tools 2001 and 2002 are displayed on the left and right sides from the top to the center of the screen, the display control unit 314 places the surgical tool selection UI unit 2003, which is a UI part used during a specific operation, in the lower left side of the second display area 1702 and the third display area 1703, where the surgical tools 2001 and 2002 are not obstructed from being viewed. 21 also shows how, when surgical tools 2101 and 2102 are displayed on the left and right sides from the bottom to the center of the screen, the display control unit 314 places a surgical tool selection UI unit 2103, which is a UI part for specific operations, in the upper left side of the screen in the second display area 1702 and the third display area 1703, where the visibility of the surgical tools 2101 and 2102 is not obstructed. Such operation of the display control unit 314 makes it possible to ensure that the UI parts (pull-down menu) related to surgical tool replacement are easily visible while minimizing the operator's ability to observe the affected area, as shown in FIGS.

[0125] 18 and 19 show examples in which UI parts required for specific operations are arranged closer to the user than UI parts that are always displayed, but when changing settings such as the imaging magnification or operation magnification, it is easier to operate if the imaging magnification setting UI section 1715 and operation magnification setting UI section 1716, which are always displayed UI parts, are switched to be displayed closer to the user. Therefore, in this embodiment, an operation is adopted in which the depth arrangement of UI parts is switched by operating the reader.

[0126] Figure 22 shows an example of screen operation in which the operator moves the reader vertically by a certain distance or more, switching the display of the UI part (operation magnification setting UI part) at the back of the screen to a position closer to the UI part (first surgical tool selection UI part) 2201 that was originally positioned in the foreground.

[0127] 22, the operation magnification setting UI section 2202, which is a UI part that is always displayed, was initially located behind the surgical tool selection UI section 2201, which is a UI part required for specific operations, making it difficult to align the mouse pointer and operate it. Therefore, when it is desired to change the setting of the operation magnification in the left-hand leader-follower control, the operator moves the left leader downward by a certain distance or more, and the display of the operation magnification setting UI section 2202 is switched so that it is positioned in front of the surgical tool selection UI section 2201, thereby enabling the operation magnification to be changed.

[0128] Although not shown in the figures, when it is desired to change the setting of the operation magnification in the right-hand leader-follower control, it is possible to switch the right-hand operation magnification setting UI section to the front side of the second surgical tool selection UI section by similarly operating the second reader 112 so as to move it downward by a certain distance or more. Also, when it is desired to change the setting of the imaging magnification rather than the operation magnification, it is possible to configure the imaging magnification setting UI section, which is always displayed, to be switched to the front side by operating the reader.

[0129] Depending on the situation, such as changing surgical tools or the work (surgery), it may be necessary to change settings such as the operation magnification in leader-follower control. As explained above with reference to Figure 22, by moving the leader up or down, the UI part for changing settings is switched to the front, enabling the setting change operation and enabling smooth setting changes. In addition, because the UI part to be operated next is visible at the back, the operator can intuitively perform operations according to the flow.

[0130] Furthermore, by utilizing the reader's ability to move both vertically and horizontally at each height, the operation of the GUI configured in layers can be switched depending on the height of the reader. For example, as shown on the left side of FIG. 36 , multiple GUIs are configured from the front to the back on a first layer 3601, a second layer 3602, and a third layer 3603. In this case, from bottom to top, two-dimensional movement 3611 of the reader at the first height may be assigned to the operation of the GUI configured on the first layer 3601, and similarly, two-dimensional movement 3612 of the reader at the second height may be assigned to the operation of the GUI configured on the second layer 3602, and two-dimensional movement 3613 of the reader at the third height may be assigned to the operation of the GUI configured on the third layer 3603. In this case, it is possible to configure the GUI in layers and switch layers on the reader side to operate the UI parts arranged on each layer without performing the operation of switching the depth arrangement of the UI parts as shown in FIG. 22 .

[0131] Note that Figure 22 shows an example in which the operator switches the depth direction placement of only the surgical tool selection UI section on the side where the reader is operated, but the display control section 314 may also be configured to display the first surgical tool selection UI section 1711 and the second surgical tool selection UI section 1712 on the screen simultaneously when the clutch button 113C is operated.

[0132] 23 shows an example in which a first surgical tool selection UI section 2301 and a second surgical tool selection UI section 2302 are displayed on the screen simultaneously. As shown in Fig. 23, the menu items in the pull-down menu of the second follower 122 on the operating side may be highlighted, and menu items corresponding to currently attached surgical tools or tools that cannot be replaced may be grayed out to clearly indicate that they cannot be selected.

[0133] During the surgical tool exchange sequence, leader-follower control is suspended for the follower to be replaced with a surgical tool, and the corresponding leader is disengaged from the follower and operates instead as a pointing device. Therefore, by operating the disengaged leader, the operator can move the mouse pointer and select the surgical tool to be replaced from the pull-down menus of the first surgical tool selection UI unit 2301 and the second surgical tool selection UI unit 2302. When using the reader as a pointing device, the drive control unit 312 may drive each electric motor so that the corresponding reader's operable direction is limited to one direction (forward and backward) for selecting the surgical tool, or two directions including up and down for changing the imaging magnification and operation magnification settings. This allows the operator to more accurately move the mouse cursor on the surgical tool selection UI, enabling the operator to select the surgical tool more quickly and smoothly.

[0134] When the first surgical tool selection UI section 2301 and the second surgical tool selection UI section 2302 are configured to be displayed on the screen simultaneously as shown in Figure 23, the first surgical tool selection UI section 2301 of the first follower 121 for the right hand can be selected only by operating the first reader 111 on the right hand side, and the second surgical tool selection UI section 2302 of the second follower 122 for the left hand can be selected only by operating the second reader 112 on the left hand side, or it can be configured so that either the first surgical tool selection UI section 2301 or the second surgical tool selection UI section 2302 can be selected with either hand.

[0135] Referring again to FIG. 15 , the surgical tool replacement sequence will be described. In step S1505, with the mouse over a menu item on the pull-down menu corresponding to the surgical tool to be replaced, the input / output processing unit 311 determines whether the surgical tool replacement has been tentatively determined (step S1506). For example, when the first operation unit 111C and the second operation unit 112C are grippers consisting of a pair of grips rotatably supported by a grip shaft, as shown in FIG. 24 , the operator can indicate the tentative determination by gripping the grippers. That is, the operator can tentatively determine the replacement of the surgical tool by closing the grippers by a certain angle or more, as shown in FIG. 25 (Yes in step S1506). If the operator does not close the grippers, i.e., if the replacement surgical tool has not been tentatively determined (No in step S1506), the process returns to step S1501, waits for the next operation of the clutch button 113C, and repeats the above process.

[0136] When the operator tentatively decides to replace the surgical tool, erroneous operation can be prevented by providing the operator with tactile feedback such as a clicking sensation, thereby improving operability and safety. In the case where the first operation unit 111C and the second operation unit 112C are grippers 2400 each consisting of a pair of grips 2401 and 2402 as shown in Fig. 24, when the gripper 2400 is closed to a certain extent or more, the drive control unit 312 drives the electric motors 111A and 112A to open and close the gripper 2400, thereby providing a clicking sensation (force tactile sensation) to the operator's fingertips from each of the grips 2401 and 2402.

[0137] FIG. 26 is a graph showing changes in the state quantities of the grip axis of the gripper 2400 used in the first operation unit 111C and the second operation unit 112C. FIG. 26(a) shows the joint angle of the grip axis, FIG. 26(b) shows the joint angular velocity of the grip axis, and FIG. 26(c) shows the joint torque of the grip axis. The horizontal axis in FIGS. 26(a) to 26(c) is the time axis, and time is shown synchronized in each graph. In FIG. 26, A shows the start position of the gripper closing operation, B shows the position where control begins so that the operator feels resistance, C shows the position where resistance is reduced and a signal is input, and D shows the start position of the gripper 2400 opening operation. Furthermore, to give the operator a sense of operation, an electric motor is driven to bias the gripper 2400 in the opening direction.

[0138] Referring to FIGS. 26( a) to 26(c), only the biasing torque is applied at position A, but when the joint angle reaches B, the torque is controlled to increase in the direction opposite to the movement. As a result, the angular velocity decreases. Referring to FIG. 26(b), the movement of the gripper 2400 continues in the closing direction while decelerating until the joint angle reaches C. Then, when the joint angle reaches C, the biasing torque is controlled to decrease, resulting in a large increase in the angular velocity. This series of controls makes it possible to provide a clicking sensation to the fingertips of the operator holding the gripper 2400. Note that after the joint angle reaches D, the operation direction is reversed to open the gripper, but the biasing torque is maintained at a constant value in the opening direction.

[0139] The calculation processing unit 310 (drive control unit 312) can be configured so that parameters such as the angle and the intensity of the click sensation that produce an optimal click sensation can be changed for each operator. Parameters that have been adjusted in advance for each operator can be loaded from an external input device, etc. Furthermore, by controlling the drive of the gripper (electric motor) so that no click sensation is presented during normal leader-follower control, the operator's procedural operation is not hindered during normal operation, as compared to methods that present a click sensation using a mechanical structure.

[0140] 15 , the surgical tool replacement sequence will be described. When a provisional decision to replace the surgical tool is made (Yes in step S1506), the display control unit 314 displays a selection screen for finalizing the provisional decision (step S1507). The display control unit 314 places the selection screen in the second display area 1702 or the third display area 1703 on the screen of the display device 114.

[0141] 27 shows an example of the configuration of a selection screen 2700. The selection screen 2700 shown in the figure is a screen used when deciding to change the second surgical tool 122A attached to the second follower 122 (LeftArm) on the left hand side from forceps to micro scissors, and displays the target follower (LeftArm) and the surgical tool to be changed for the current surgical tool (Tweezers-L → Micro scissors), as well as including a "Decide" button 2701 for instructing the user to make a final decision on the surgical tool change and a "Cancel" button 2702 for instructing the user to cancel the surgical tool change.

[0142] When the operator places the mouse over the "Cancel" button 2702 in the selection screen 2700 and performs a decision operation such as closing the gripper of the second operation unit 112C (No in step S1508), the process returns to step S1501, waits until the clutch button 113C is next operated, and repeats the above process.

[0143] On the other hand, when the operator performs a confirmation operation such as closing the gripper of the second operation unit 112C while hovering the mouse over the "OK" button 2701 in the selection screen 2700 (Yes in step S1508), the drive control unit 312 moves the follower who will perform the surgical tool exchange to the surgical tool exchange area (step S1509). Note that when the operator performs the confirmation operation using the gripper in steps S1508 and S1509, tactile feedback (described above) such as a clicking sensation may be provided.

[0144] Then, the drive control unit 312 rotates the tool turret 124C so that the surgical tool to be replaced is in a position where it can be easily removed, and then performs the surgical tool replacement on the follower (step S1510), thereby terminating this process. The series of operations when replacing a surgical tool in the operation unit 120 has already been described with reference to FIGS. 6 to 11.

[0145] As explained above, the surgical support robot system 100 according to this embodiment enables the operator to input surgical tool replacement instructions and replace multiple surgical tools accurately in a short time, thereby shortening the surgical time.

[0146] Furthermore, in the surgical support robot system 100, the clutch button 113C is assigned the function of displaying the surgical tool selection UI, so that this function can be realized without adding a dedicated button to the foot switch unit 113, allowing the system to be constructed in a space-saving manner.

[0147] The main configuration and basic operation of the surgical support robot system 100 according to this embodiment are not limited to specific surgical tools, and the same surgical tool exchange sequence can be achieved even when the type of surgical tool used changes. Furthermore, even if the surgical tools are the same type, by introducing a mechanism for identifying each surgical tool, multiple surgical tools of the same type can be treated as individual surgical tools and exchanged between surgical tools of the same type. For example, if a surgical tool malfunctions due to wear or breakage during surgery, it can be replaced with another surgical tool of the same type. The mechanism for identifying each surgical tool can be achieved, for example, by attaching identification information such as a marker to the surgical tool and providing an identification information reader on the mounting portion of each follower.

[0148] F. Second Embodiment In this section F, a second embodiment of the surgical support robot system 100 and the operation of the system will be described.

[0149] In the above-described surgical support robot system 100, the function of displaying a surgical tool selection UI is assigned to the clutch button 113C of the foot switch unit 113, and the operator can start the surgical tool exchange sequence by operating the clutch button 113C. In contrast, in the second embodiment, it is assumed that the surgical support robot system 100 is further provided with a surgical tool exchange button.

[0150] The surgical tool change button is configured, for example, with a mechanical operator that operates momentarily or alternately. The following description assumes a surgical tool change button that operates momentarily. When an alternate-operation surgical tool change button is used, the state of the button may be determined by clicking it again, or the resumption of the procedure may be determined by the state of the button, or it may be automatically resumed. In either case, the operator can smoothly start the surgical tool change sequence simply by operating the surgical tool change button.

[0151] 14 is provided with a first surgical tool change button 113A and a second surgical tool change button 113B on the left and right near the front end, corresponding to the first follower 121 and the second follower 122, respectively. The operator can use his or her left and right feet to operate the first surgical tool change button 113A and the second surgical tool change button 113B to individually instruct each of the first follower 121 and the second follower 122 to change their surgical tools. The first surgical tool change button 113A and the second surgical tool change button 113B may be provided on the foot switch unit 113, or on the first reader 111 and the second reader 112, respectively. Of course, the surgical tool change buttons may be provided in locations other than the foot switch unit 113 or the first reader 111 and the second reader 112.

[0152] Figure 30 shows the processing operations of the surgery support robot system 100 according to the second embodiment in the form of a flowchart. This operation is mainly performed by the arithmetic processing unit 310 (see Figure 3) in the robot control device 130. The processing operations performed by the surgery support robot system 100 will be described below with reference to this flowchart.

[0153] First, the drive control unit 312 determines the states of the first surgical tool exchange button 113A and the second surgical tool exchange button 113B (step S3001). The first surgical tool exchange button 113A and the second surgical tool exchange button 113B arranged on the foot switch unit 113 correspond to the first follower 121 and the second follower 122, respectively.

[0154] If the surgical tool change button has not been pressed (No in step S3001), the leader-follower control continues as is. Also, if the pressing of the surgical tool change button is released (No in step S3001) and the corresponding follower has moved to the surgical tool change waiting area, the follower is immediately moved to its original position before the surgical tool change button was pressed (step S3002), the leader-follower control is resumed (step S3003), the process returns to step S3001, and the process waits until the surgical tool change button is next pressed, repeating the above process.

[0155] On the other hand, if the surgical tool change button is operated (Yes in step S3001), the drive control unit 312 temporarily suspends the leader-follower control of the follower (step S3004).

[0156] Of the first surgical tool change button 113A and the second surgical tool change button 113B, only the leader corresponding to the operated side operates as a pointing device during the surgical tool change sequence, while the follower corresponding to the non-operated side continues leader-follower control by the drive control unit 312. The drive control unit 312 may also drive each electric motor so that the operable direction of the reader used as a pointing device is restricted to one direction, forward and backward, for selecting a surgical tool, or to two directions, including up and down operation for changing the settings of the imaging magnification and the operation magnification. This allows the operator to more accurately move the mouse cursor on the surgical tool selection UI, enabling the operator to select a surgical tool more quickly and smoothly.

[0157] If the follower who operated the surgical tool exchange button is not in the surgical tool exchange waiting area, the drive control unit 312 moves the follower to the surgical tool exchange waiting area (see FIG. 7) (step S3006).

[0158] The image generation unit 313 also calculates, from among the selectable surgical tools, which surgical tools are likely to be replaced (step S3005). After the series of calculations is completed in the image generation unit 313, the display control unit 314 then displays, on the screen of the display device 114, a surgical tool selection UI generated by the screen generation unit 313 based on the calculation results (step S3007). In the surgical tool selection UI, as described above with reference to FIG. 17 , a list of surgical tools that can be replaced by the target follower is displayed in a pull-down menu format using the second display area and the third display area.

[0159] In step S3007, the drive control unit 312 controls the drive of the electric motors 111A and 112A so that the operation direction of the reader used as a pointing device is restricted to two directions, including one direction (forward and backward) for selecting a surgical tool and one direction (up and down) for changing the settings of the operation magnification and imaging magnification. This allows the operator to move the mouse cursor more accurately on the surgical tool selection UI, enabling the operator to select a surgical tool more quickly and smoothly.

[0160] Then, when the operator performs an operation such as closing the operation unit (gripper) of the leader corresponding to the follower who will be performing the surgical tool replacement while hovering the mouse over the menu item corresponding to the surgical tool to be replaced on the pull-down menu, it is tentatively decided to replace the surgical tool with the surgical tool corresponding to the menu item hovered over (Yes in step S3008). Therefore, the operator can select the surgical tool in a shorter time by hovering the mouse over the surgical tool displayed at the top of the pull-down menu, i.e., the surgical tool calculated to be most likely to be replaced, at the same time as the surgical tool selection UI is displayed on the screen of the display device 114.

[0161] Next, the display control unit 314 displays a selection screen for finalizing the provisional decision (step S3009). This selection screen is as described above with reference to Fig. 27, and displays the target follower, the current surgical tool, and the replacement surgical tool, and also has an "OK" button and a "Cancel" button.

[0162] In the next step S3010, the operator can select either the "OK" button or the "Cancel" button by rotating the wrist axis of the operation part of the leader corresponding to the follower who will be performing the surgical tool change.

[0163] In this case, the drive control unit 312 may be configured to drive and control the electric motors that operate the wrist axes of the first operation unit 111C and the second operation unit 112C to provide the operator with a clicking sensation and to return the wrist axis to the central angle after input (spring return). In this embodiment, the functions of providing a clicking sensation and performing the spring return operation of the wrist axis of the operation unit are realized by motor control. Therefore, as in the case of the gripper opening and closing operation (see FIGS. 24 to 26), the functions can be configured not to function during normal leader-follower control operation so as not to interfere with the operator's manipulation. Since the options can be changed by operating one axis of the wrist axis, the operator can make inputs in a shorter time.

[0164] When the operator selects the "OK" button on this selection screen to confirm the surgical tool change (Yes in step S3010), the drive control unit 312 moves the follower that will perform the surgical tool change from the surgical tool change standby area to the surgical tool change area (step S3011). The drive control unit 312 then rotates the tool turret 124C so that the surgical tool to be changed is in a position that makes it easy to remove, and then performs the surgical tool change on the follower (step S3012), thereby ending this process. The series of operations performed when changing a surgical tool in the operation unit 120 have already been described with reference to FIGS. 6 to 11.

[0165] On the other hand, if the surgical tool change is not tentatively decided (No in step S3008), or if the surgical tool change is not decided (No in step S3010), it is further checked whether the "Cancel" button has been selected (step S3013).

[0166] If the "Cancel" button is selected (Yes in step S3013), the follower that was moved to the surgical tool exchange waiting area in step S3006 is moved to its original position before the surgical tool exchange button was operated (step S3002), leader-follower control is resumed (step S3003), the process returns to step S3001, and the process waits until the surgical tool exchange button is next operated, repeating the above processing.If the "Cancel" button is not selected (No in step S3013), the process returns to step S3001, and the process waits until the surgical tool exchange button is next operated, repeating the above processing.

[0167] As explained above, according to the surgical support robot system 100 of the second embodiment, after the operator operates the surgical tool change button (in other words, in parallel with the display processing of the surgical tool selection UI), the follower starts moving to the surgical tool change waiting area near the surgical tool change unit 124, so the time required for the follower to move after the operator has decided on the replacement surgical tool is shortened, making it possible to change the surgical tool in a shorter time.

[0168] G. Third Embodiment In this section G, a third embodiment of the surgical support robot system 100 and the system operation thereof will be described.

[0169] 31 shows an example of the configuration of a surgical tool selection UI used in the third embodiment. Similar to the screen 1700 shown in FIG. 17 , the display screen 3100 of the display device 114 is divided into a first display area 3101 near the center of the screen, a second display area 3102 adjacent to the outside of the first display area 3101, and a third display area 3103 at the outermost periphery of the screen. Taking into consideration the surgical field and the tube path in the image of the patient's affected area captured by the camera 123A, the display of UI parts is prohibited in the first display area 3101 so that the UI parts do not overlap the surgical field and interfere with the operator's procedure. The second display area 3102 can display UI parts in part when the surgical tool is stopped, and the UI parts reside in the third display area 3103.

[0170] Additionally, the screen 3100 includes a surgical tool selection UI section 3111 as a UI part for selecting replacement surgical tools. Taking into consideration that surgical tools are often located from the bottom right to the center in images of affected areas, the surgical tool selection UI section 3111 is located in the upper left of the screen. The surgical tool selection UI section 3111 is a drum roll (or picker view) pull-down menu. A drum roll pull-down menu is a display format in which a list of options (in this case, a list of surgical tools replaceable with the corresponding follower) is arranged in the rotational direction around the outer periphery of a drum, and the option located in the center is selected depending on the rotational position of the drum. The selected option may be highlighted using various techniques, such as changing color, flashing, or displaying an animation. Additionally, menu items corresponding to currently attached surgical tools or non-replaceable surgical tools may be grayed out to indicate that they are unavailable for selection.

[0171] Near the bottom right and left ends of the third display area 3103, there are disposed, as other UI parts, a first surgical tool display UI section 3112 and a second surgical tool display section 3113, which display the surgical tools currently attached to the first follower 121 and the second follower 122. Near the center of the bottom end of the third display area 3113, there are disposed, as UI parts that are always displayed, an imaging magnification setting UI section 3114 and an operation magnification setting UI section 3115, which are used to set or change the imaging magnification of the imaging unit 123 and the operation magnification in leader-follower control, respectively.

[0172] Because the surgical tool selection UI unit 3111 is configured as a drum roll pull-down menu, even if the number of surgical tools to be replaced increases, the screen area occupied by the surgical tools can be reduced while still ensuring a clear overview of the surgical tools to be replaced. This does not interfere with the operator's ability to perform surgery while checking images of the patient's affected area, etc. Note that while only one drum roll pull-down menu is shown in FIG. 31 to simplify the illustration, multiple drum roll pull-down menus may be arranged side by side on the screen. However, it is preferable to arrange multiple drum roll pull-down menus so as not to interfere with the operator's procedure.

[0173] The use of a dial-type input means makes it easy to intuitively operate the drum roll pull-down menu. The operator can then select the desired surgical tool from the pull-down menu by rotating the wrist axis of the operation unit of the leader corresponding to the follower who will be changing the surgical tool (in other words, by operating the operator's own wrist joint). Since the surgical tool can be selected simply by rotating the operation unit, the surgical tool can be selected in a shorter time.

[0174] The operating unit is assumed to have a gripper consisting of a pair of grips that can be opened and closed, as shown in Figures 24 and 25. Figure 32 shows an operator gripping and rotating the gripper to rotate a drum roll. Figure 33 shows an operator closing the gripper when deciding on a target surgical tool that has been provisionally selected from a drum roll pull-down menu.

[0175] As explained above with reference to Fig. 26 , torque can be generated by controlling the drive of the electric motor that operates the gripper, and a haptic sensation can be presented to the fingers of the operator holding the gripper. When the operator holds the gripper and rotates it, torque corresponding to the direction of the rotation speed is generated at every certain phase angle, and the operator can be presented with an operating sensation simulating that of a dial-type input device.

[0176] For example, a dial-type input device that can be pressed has been proposed (see Patent Document 3). In contrast, in the surgical support robot system 100 according to the third embodiment, the operation of the operation unit of each leader is realized by motor control, so that, similar to the opening and closing operation of the gripper, torque is not generated by the rotation operation of the operation unit so as not to hinder the operator's procedural operation during normal leader-follower control operation.

[0177] In addition to the drum roll type, a turntable type can also be used as a form in which the desired surgical tool can be selected by rotating the operating unit. Figure 34 shows an example of the configuration of the display screen of the display device 114 equipped with a turntable type surgical tool selection UI.

[0178] The display screen 3400 shown in FIG. 34 is divided into a first display area 3401 near the center of the display device 114, a second display area 3402 adjacent to the outside of the first display area 3401, and a third display area 3403 at the outermost periphery of the screen. A turntable-style surgical tool selection UI 3411 is located in the upper left corner of the screen. Interchangeable surgical tools are assigned to predetermined phase angles of the turntable (every 60 degrees in the example shown in FIG. 34). An instrument aligned to a predetermined direction on the turntable (the 3 o'clock direction in the example shown in FIG. 34) is selected and displayed enlarged. The selected options may be highlighted using various techniques, such as color change, flashing, or animation. Other surgical tools that are not selected are displayed in a reduced size. Additionally, menu items corresponding to currently attached surgical tools or non-interchangeable surgical tools may be grayed out to indicate that they are unavailable for selection.

[0179] The turntable type can also be operated intuitively by using a dial-type input means (see Figures 32 and 33). By configuring the selected surgical tool to move by rotating the operating unit, similar to the drum roll type described above, it is possible to ensure a clear overview of the surgical tool being replaced while reducing the area occupied on the screen. Furthermore, when the operator holds and rotates the gripper, torque corresponding to the direction of the rotation speed is generated at certain phase angles, providing the operator with an operating feel similar to that of a dial-type input device.

[0180] Although only one turntable is shown in Fig. 34 for the sake of simplicity, multiple turntables may be arranged side by side on the screen. However, it is preferable to arrange multiple turntables so as not to interfere with the operator's procedure.

[0181] H. Fourth Embodiment In this section H, a fourth embodiment of the surgical support robot system 100 and the operation of the system will be described.

[0182] In the above-described surgical support robot system 100, the leader-follower control is temporarily stopped and the surgical tool change sequence is started using the clutch button 113C or the surgical tool change button of the foot switch unit 113. In contrast, in the fourth embodiment, the leader-follower control is temporarily stopped and the surgical tool change sequence is started by operating the follower to move. In the fourth embodiment, the operator can smoothly start the surgical tool change sequence by operating to move the follower to a predetermined safety area.

[0183] Figure 35 shows the processing operations of the surgery support robot system 100 according to the fourth embodiment in the form of a flowchart. This operation is mainly performed by the arithmetic processing unit 310 (see Figure 3) in the robot control device 130. The processing operations performed by the surgery support robot system 100 will be described below with reference to this flowchart.

[0184] First, the drive control unit 312 determines the regions of the first follower 121 and the second follower 122 (step S3501). Here, the drive control unit 312 determines whether either follower has moved to a region that instructs temporary suspension of leader-follower control and activation of a surgical tool replacement sequence. Specifically, the drive control unit 312 determines whether the follower has moved above the dashed-dotted line 2901 shown in FIG. 29. Because the operation of moving a follower to a region above the dashed-dotted line 2901 corresponds to the operation of safely retracting the follower from the affected area of ​​the patient, this determination criterion does not impede the operator's procedural operation.

[0185] 29 (No in step S3501), it can be determined that the operator is using the leader to operate the follower. In this case, therefore, the process returns to step S3501, and the leader-follower control continues.

[0186] On the other hand, if the operator uses the leader to move the follower above the dashed line 2901 shown in Fig. 29 (Yes in step S3501), it can be determined that the operator intends to replace the surgical tool attached to the follower. Therefore, in this case, the drive control unit 312 temporarily suspends the leader-follower control of that follower (step S3502).

[0187] Next, the image generation unit 313 calculates the surgical tool that is likely to be replaced from among the selectable surgical tools (step S3503). After the series of calculations is completed in the image generation unit 313, the display control unit 314 displays the surgical tool selection UI generated by the screen generation unit 313 based on the calculation results on the screen of the display device 114 (step S3505).

[0188] In parallel with the display processing of the surgical tool selection UI, the drive control unit 312 moves the follower to the surgical tool exchange standby area (see FIG. 7) (step S3504). By moving the follower to the surgical tool exchange standby area while the operator is selecting a surgical tool via the surgical tool selection UI on the screen, the time required for the follower to move after the operator has decided on the replacement surgical tool is shortened, and surgical tool exchange can be completed in a shorter time.

[0189] The surgical tool selection UI displayed on the screen in step S3505 may be any of the surgical tool selection UIs shown in Fig. 17, Fig. 31, and Fig. 34. The operator can tentatively determine the target surgical tool by operating the operation unit of the leader corresponding to the follower whose area was determined in step S3501 (by hovering the mouse over the operation unit, rotating the operation unit, and closing the gripper).

[0190] When the provisional decision to replace the surgical tool with the desired surgical tool is made (Yes in step S3506), the display control unit 314 displays a selection screen for finalizing the provisional decision (step S3507). This selection screen is as described above with reference to FIG. 27 and includes an "OK" button and a "Cancel" button.

[0191] When the operator selects the "OK" button on this selection screen to confirm the surgical tool change (Yes in step S3508), the drive control unit 312 moves the follower that will perform the surgical tool change from the surgical tool change standby area to the surgical tool change area (step S3509).The drive control unit 312 then rotates the tool turret 124C so that the surgical tool to be changed is in a position that makes it easy to remove, and then performs the surgical tool change on the follower (step S3510), thereby ending this process.The series of operations performed when changing a surgical tool in the operation unit 120 have already been described with reference to Figures 6 to 11.

[0192] If the replacement with the desired surgical tool is not tentatively decided (No in step S3506), or if the operator does not select the "OK" button on the selection screen (No in step S3508), it is further determined whether the operator selected the "Cancel" button (step S3511).

[0193] If the operator does not select the "Cancel" button (No in step S3512), the process returns to step S3501, waits until the follower is next moved upward, and repeats the above process.

[0194] Also, if the operator selects the "Cancel" button (Yes in step S3511), the follower that was moved to the surgical tool exchange waiting area in step S3504 above is moved to its original position before the surgical tool exchange button was operated (step S3512), leader-follower control is resumed (step S3513), the process returns to step S3501, and the process waits until the surgical tool exchange button is next operated, repeating the above process.

[0195] According to the fourth embodiment, the operator can temporarily suspend the leader-follower control and transition to the surgical tool exchange sequence by operating a specific follower without operating a button, which allows for smooth operation and improves safety because the leader-follower control is suspended after the follower has been moved to a safe location where the surgical tool has been removed from the affected area. Another advantage is that there is no need to provide a mechanical operator such as a surgical tool exchange button on the foot switch unit 113 or the leader.

[0196] The follower operation for temporarily suspending the leader-follower control and transitioning to the surgical tool replacement sequence is not limited to the operation of moving the follower upward. For example, the follower movement operation may be specified by a specific trajectory or speed, or may be specified by an operation on the gripper of the operation unit, such as double-clicking. In any case, it is preferable to specify the operation by the follower in a safe place where the surgical tool has been removed from the affected area.

[0197] Furthermore, some operations such as confirming the selection in step S3508 and canceling in step S3511 may be performed by voice input.

[0198] The present disclosure has been described in detail above with reference to specific embodiments. However, the present disclosure should not be construed as being limited to the above-described embodiments, and it is obvious that those skilled in the art can modify or substitute the embodiments without departing from the spirit of the present disclosure. Furthermore, the effects described in this specification are merely examples, and the effects brought about by the present disclosure are not limited thereto, and additional effects not described in this specification may exist.

[0199] This specification has mainly described an embodiment in which the present disclosure is applied to a surgical assistance robot system, but the surgical assistance robot system can be applied to various types of surgery, such as laparoscopic surgery, neurosurgery, and ophthalmic surgery.

[0200] Furthermore, the gist of the present disclosure is not limited to medical technology, but can also be applied to various types of work support robot systems that remotely control robot arms in a leader-follower manner to assist in difficult tasks in various industrial fields, such as high places, construction sites, factories where processing or assembly work is performed, mining sites, high places such as high-rise buildings, underwater or on the seabed, nuclear plants, and outer space.

[0201] In short, the present disclosure has been described in the form of examples, and the contents of the specification should not be interpreted as limiting. To determine the gist of the present disclosure, the claims should be taken into consideration.

[0202] The present disclosure may also be configured as follows.

[0203] (1) A control device for controlling a surgical support robot system including an operation unit including one or more followers to which surgical tools are interchangeably attached, a console unit including one or more readers operated by an operator, and a display device that displays on a screen an image of the affected area of ​​a patient undergoing surgery with the surgical tools, the control device controls the operation of the follower in accordance with the operation of the leader, and when a predetermined event is detected, releases the link between the leader and the follower, displays information about interchangeable surgical tools on the screen, and replaces the surgical tool attached to the follower.

[0204] (2) The control device according to (1) above, wherein the predetermined event is detected based on the operation of a predetermined operator disposed on the console unit or elsewhere.

[0205] (3) The control device described in (2) above, wherein the console unit includes a foot switch unit, detects the specified event based on the operation of a clutch button provided on the foot switch unit, displays information about surgical tools that are candidates for replacement to the corresponding follower, and performs the replacement of the surgical tool.

[0206] (4) The control device described in (2) above, wherein the console unit includes a foot switch unit, detects the specified event based on the operation of either the foot switch unit or a surgical tool change button provided on the leader and associated with each of the one or more followers, and displays information about surgical tools that are candidates for replacement to the follower corresponding to the leader whose surgical tool change button was operated, thereby replacing the surgical tool.

[0207] (5) The control device described in (2) above, which detects the specified event based on the movement of any of the one or more readers to a specified area, displays information about surgical tools that are candidates for replacement to the corresponding follower, and performs the replacement of the surgical tool.

[0208] (6) A control device according to any one of (1) to (5) above, which calculates the priority of each surgical tool that can be replaced with the surgical tool currently attached to the follower, and displays a list of surgical tools that are candidates for replacement based on the priority.

[0209] (6-1) The control device according to (6) above, which estimates surgical tools that are likely to be replaced based on a pattern of high frequency in the past, based on a stored order of replacing surgical tools.

[0210] (7) A control device described in any one of (1) to (6) above, wherein the screen of the display device is divided into a first display area that always displays an image of the patient's affected area during surgery, a second display area that allows UI parts to be displayed when the specified event is detected, and a third display area that allows UI parts to remain resident, and UI parts for displaying a list of surgical instruments that are candidates for replacement are placed in at least one of the second display area and the third display area.

[0211] (8) The control device according to (7) above, wherein a UI part is displayed that displays a list of surgical tools that are candidates for replacement in the form of a pull-down menu, a drum roll, or a turntable.

[0212] (9) The control device according to any one of (7) or (8), wherein when the predetermined event is detected, an operation on the screen is controlled based on an operation of the reader.

[0213] (10) A control device described in any one of (7) to (9) above, which provides multiple layers from the front side to the back side of the screen, manages the placement of UI parts for each layer, and superimposes UI parts placed in the front layer on UI parts placed in the back layer and displays them simultaneously on the screen.

[0214] (10-1) The control device described in (10) above, in which UI parts that are always displayed are arranged in a layer at the back, and UI parts that are displayed when the specified event is detected or when a specific operation is performed are arranged in a layer at the front.

[0215] (11) The control device according to any one of (7) to (10) above, wherein at least some UI parts are displayed semi-transparently.

[0216] (12) The control device according to (10), wherein the depth arrangement of UI parts is switched in response to an operation of moving the reader in a predetermined direction by a distance equal to or greater than a certain distance.

[0217] (13) The control device according to (9), wherein when controlling an action on the screen based on an operation of the reader, tactile feedback is presented using the reader.

[0218] (14) The control device described in (13) above, wherein the reader includes an operating unit consisting of a gripper that can be opened and closed, and the gripper is opened and closed in response to the gripper being closed to a certain extent or more, thereby providing tactile feedback.

[0219] (15) The control device according to any one of (1) to (14) above, wherein the leader whose linkage with the follower has been released is operated as a pointing device for operating a mouse cursor on the screen.

[0220] (16) The control device according to (15) above, wherein the operable directions of the reader while being used as a pointing device are restricted to only directions necessary for operating UI parts displayed on the screen.

[0221] (17) A control method for a surgical support robot system including an operation unit including one or more followers to which surgical tools are interchangeably attached, a console unit including one or more readers operated by an operator, a display device that displays on a screen an image of the affected area of ​​a patient undergoing surgery with the surgical tools, and a control device that controls the operation of the followers in accordance with the operation of the readers, the control method comprising the steps of: detecting a predetermined event that occurs on the surgical support robot system; releasing the link between the leader and the follower in response to the occurrence of the predetermined event and displaying information about interchangeable surgical tools on the screen; and replacing the surgical tool attached to the follower with a surgical tool selected based on the displayed information.

[0222] (18) A surgical support robot system comprising: an operation unit including one or more followers to which surgical tools are interchangeably attached; a console unit including one or more readers operated by an operator; a display device that displays on a screen an image of the affected area of ​​a patient undergoing surgery with the surgical tools; and a control device that controls the operation of the follower in accordance with the operation of the reader; wherein, when the control device detects a predetermined event, it releases the link between the leader and the follower, displays information about the interchangeable surgical tools on the screen, and replaces the surgical tools attached to the follower.

[0223] DESCRIPTION OF SYMBOLS 100...Surgery support robot system, 110...Console unit 111...First reader, 111A...Electric motor, 111B...Sensor unit 111C...First operation unit, 112...Second reader, 112A...Electric motor 112B...Sensor unit, 112C...Second operation unit 113...Foot switch unit, 113A...First surgical tool exchange button 113B...Second surgical tool exchange button, 113C...Clutch button 114...Display device, 120...Operating unit 121...First follower, 121A...First surgical tool, 121B...Electric motor 121C...Sensor unit, 122...Second follower, 122A...Second surgical tool 122B...Electric motor, 122C...Sensor unit, 123...Imaging unit 123A...Camera, 123B...Light source, 123C...Camera arm 124... surgical tool replacement unit, 124A... first replacement surgical tool, 124B... second replacement surgical tool, 124C... tool turret, 124D... electric motor, 124E... encoder, 125... base arm, 125A... electric motor, 125B... sensor unit, 130... robot control device, 131... CPU, 132... memory unit, 133... bus, 134... input / output unit, 140... external operation terminal, 141... input unit, 142... display unit, 150... operating table, 160... external network, 310... calculation processing unit, 311... input / output processing unit, 312... drive control unit, 313... image generation unit, 314... display control unit, 320... input / output interface, 400... housing, 401... top surface, 402... legs, 403... wheels, 411... support unit, 412... first link, 413... second link 421...first joint portion, 422...second joint portion, 423...third joint portion 501...first mounting portion, 502...second mounting portion 511...first robot arm, 512...second robot arm 1201...bottom portion, 1202...base portion, 1203...support portion 1204...chair, 1205...hand rest

Claims

1. A control device for controlling a surgical support robot system including an operation unit including one or more followers to which surgical tools are interchangeably attached, a console unit including one or more readers operated by an operator, and a display device that displays on a screen captured images of the affected area of ​​a patient during surgery with the surgical tools, the control device controls the operation of the followers in response to the operation of the leaders, and upon detecting a specified event, releases the link between the leaders and followers, displays information about interchangeable surgical tools on the screen, and replaces the surgical tools attached to the followers.

2. The control device according to claim 1, wherein the predetermined event is detected based on the operation of a predetermined operator disposed on the console unit or in another location.

3. The control device according to claim 2, wherein the console unit includes a foot switch section, detects the specified event based on the operation of a clutch button provided on the foot switch section, and displays information about surgical tools that are candidates for replacement to the corresponding follower, thereby carrying out the replacement of the surgical tool.

4. The control device according to claim 2, wherein the console unit includes a foot switch section, detects the specified event based on operation of either a surgical tool replacement button provided on the foot switch section or the leader and associated with each of the one or more followers, and displays information about surgical tools that are candidates for replacement to the follower corresponding to the leader whose surgical tool replacement button was operated, thereby performing a replacement of the surgical tool.

5. The control device according to claim 2, which detects the specified event based on the movement of any one of the one or more readers into a specified area, and displays information about surgical tools that are candidates for replacement to the corresponding follower, thereby carrying out the replacement of the surgical tool.

6. The control device according to claim 1, further comprising: a controller that calculates a priority order of each surgical tool that can be replaced with the surgical tool currently attached to the follower; and displays a list of surgical tools that are candidates for replacement based on the priority order.

7. A control device as described in claim 1, wherein the screen of the display device is divided into a first display area which always displays an image of the patient's affected area during surgery, a second display area which allows UI parts to be displayed when the specified event is detected, and a third display area which allows UI parts to remain resident, and wherein UI parts for displaying a list of surgical tools that are candidates for replacement are placed in at least one of the second display area and the third display area.

8. The control device according to claim 7, which displays a UI part that displays a list of surgical tools that are candidates for replacement in the form of a pull-down menu, a drum roll, or a turntable.

9. The control device according to claim 7, further comprising: a control unit for controlling an operation on said screen based on an operation of said reader when said predetermined event is detected.

10. The control device according to claim 7, which provides multiple layers from the front to the back of the screen, manages the placement of UI parts for each layer, and superimposes UI parts placed in a front layer on UI parts placed in a back layer and displays them simultaneously on the screen.

11. The control device according to claim 7, wherein at least a part of the UI parts is displayed semi-transparently.

12. The control device according to claim 10, further comprising: a control unit for controlling a user interface (UI) part to be displayed on a display screen; 13. The control device according to claim 9, further comprising: a display unit configured to display a display image of the user on the display screen; a display unit configured to display a display image of the user on the display screen; 14. The control device according to claim 13, wherein the reader includes an operating unit consisting of a gripper that can be opened and closed, and the gripper is opened and closed in response to the gripper being closed to a certain extent to provide tactile feedback.

15. The control device according to claim 1, wherein the leader, whose linkage with the follower has been released, is operated as a pointing device for operating a mouse cursor on the screen.

16. The control device according to claim 15, wherein the operable directions of the reader while being used as a pointing device are restricted to only directions required for operations on UI parts displayed on the screen.

17. A control method for a surgical support robot system equipped with an operation unit including one or more followers to which surgical tools are interchangeably attached, a console unit including one or more readers operated by an operator, a display device that displays on a screen an image of the affected area of ​​a patient undergoing surgery with the surgical tools, and a control device that controls the operation of the followers in response to the operation of the readers, the control method having the steps of: detecting a predetermined event that occurs on the surgical support robot system; releasing the link between the leader and the follower in response to the occurrence of the predetermined event, and displaying information about interchangeable surgical tools on the screen; and replacing the surgical tools attached to the followers with surgical tools selected based on the displayed information.

18. A surgical support robot system comprising: an operation unit including one or more followers to which surgical tools are interchangeably attached; a console unit including one or more readers operated by an operator; a display device that displays on a screen an image of the affected area of ​​a patient during surgery with the surgical tools; and a control device that controls the operation of the followers in accordance with the operation of the readers, wherein when the control device detects a predetermined event, it releases the link between the leader and the follower, displays information about the interchangeable surgical tools on the screen, and replaces the surgical tools attached to the followers.

Citation Information

Patent Citations

  • Surgical manipulator and its operation input apparatus

    JP2004129782A

  • Surgery system

    JP2021019949A

  • Surgical instrument and surgery support system having said surgical instrument

    WO2012073849A1