Surgical instrument storage system and surgical assisting robot system

Abstract

Provided is a surgical instrument storage system for storing a plurality of surgical instruments.　This surgical instrument storage system comprises a holding unit that holds surgical instruments and a detection unit that detects the surgical instruments held by the holding unit. Each surgical instrument has visual identification information. The holding unit holds the surgical instruments such that the visual identification information is included in a detection range of the detection unit. The detection unit includes a camera that captures an image of the surgical instruments held by the holding unit and a reading unit that reads the visual identification information on the basis of the image captured by the camera. The detection unit further includes a position estimation unit that estimates the position of each surgical instrument held by the holding unit in the up-down direction from the image captured by the camera.

Description

Surgical tool storage system and surgical support robot system

[0001] The technology disclosed in this specification (hereinafter referred to as "the present disclosure") relates to a surgical support robot that assists doctors in performing surgery.

[0002] Medical procedures are generally performed using surgical instruments such as forceps, tweezers, needle holders, scissors, electric scalpels, insufflation tubes, and energy treatment devices (hereinafter collectively referred to as "surgical instruments"), as well as imaging devices such as endoscopes. Recently, surgical support robot systems have been used that support surgeons with multiple arms supporting these surgical instruments and imaging devices. In the case of a master-slave (hereinafter referred to as a "leader-follower") surgical support robot system, a medical professional such as a doctor (hereinafter referred to as an "operator") can operate a control device (leader) to operate a robot arm (follower) and perform surgery while observing images of the surgical field captured by an imaging device.

[0003] Typically, multiple surgical instruments are used in a single surgery. Therefore, the task of changing the surgical instruments used by the surgical support robot occurs frequently during surgery. For example, a surgical device has been proposed that autonomously changes the surgical instruments to the desired surgical instruments by rotating a carousel having multiple grips that hold the surgical instruments (see Patent Document 1).

[0004] JP 2022-512316 A

[0005] An object of the present disclosure is to provide a surgical tool storage system that stores multiple surgical tools used in a surgical support robot system, and a surgical support robot system that performs treatment while exchanging multiple surgical tools.

[0006] The present disclosure has been made in consideration of the above-mentioned problems, and a first aspect thereof is a surgical tool storage system comprising: a holding section for holding a surgical tool; and a detection section for detecting the surgical tool held in the holding section, wherein the surgical tool has visual identification information, the holding section holds the surgical tool so that the visual identification information is included in the detection range of the detection section, and the detection section comprises a camera for capturing an image of the surgical tool held in the holding section, and a reading section for reading the visual identification information based on the image captured by the camera.

[0007] However, the term "system" used here refers to a logical collection of multiple devices (or functional modules that realize specific functions), regardless of whether each device or functional module is contained within a single housing. In other words, both a single device consisting of multiple parts or functional modules and a collection of multiple devices are considered "systems."

[0008] The surgical tool has an end effector at its tip and visual identification information is provided on the end face opposite the end effector. The holding unit is configured to hold the surgical tool with the end effector facing downward and the end face with the visual identification information facing upward. The reading unit then reads the visual identification information from the end face of the surgical tool captured in the image captured by the camera.

[0009] The detection unit may further estimate a horizontal displacement of the surgical tool relative to the holding unit based on visual identification information read by the reading unit from an end face of the surgical tool reflected in the image captured by the camera. The detection unit may further include a position estimation unit that estimates a vertical position of the surgical tool held by the holding unit from the image captured by the camera.

[0010] The holding unit further includes a mirror installed near the position where the surgical tool is held and directs light from the side of the surgical tool toward the camera. The camera is disposed so that the end face of the surgical tool and the mirror are included in its imaging area. The position estimation unit estimates the vertical position where the holding unit holds the surgical tool based on the results of edge detection of the top end face of the surgical tool from the image of the side of the surgical tool captured by the camera via the mirror. The position estimation unit may convert the horizontal displacement of the top end face of the surgical tool, which is edge-detected from the image captured by the camera, into vertical displacement of the surgical tool relative to the holding unit, depending on the tilt angle at which the mirror is attached.

[0011] A second aspect of the present disclosure is a surgical support robot system comprising an operation unit including one or more followers to which surgical tools are interchangeably attached, and a console unit including one or more readers operated by an operator, wherein the operation unit includes a surgical tool exchange unit that holds replacement surgical tools and a detection unit that detects the surgical tools held in the surgical tool exchange unit, the surgical tools having visual identification information, the surgical tool exchange unit holds the surgical tools so that the visual identification information is included in the detection range of the detection unit, and the detection unit includes a camera that captures an image of the surgical tool held in the surgical tool exchange unit and a reading unit that reads the visual identification information based on the image captured by the camera.

[0012] According to the present disclosure, it is possible to provide a surgical tool storage system that appropriately stores multiple surgical tools used in a surgical support robot system so that they can be automatically replaced, and a surgical support robot system that performs procedures while automatically replacing surgical tools using the surgical tool storage system.

[0013] It should be noted that the effects described in this specification are merely examples, and the effects brought about by the present disclosure are not limited to these. Furthermore, the present disclosure may also bring about additional effects in addition to the effects described above.

[0014] Further objects, features, and advantages of the present disclosure will become apparent from the following detailed description based on the embodiments and accompanying drawings.

[0015] FIG. 1 is a diagram showing the external configuration of the surgical support robot system 100. FIG. 2 is a diagram showing the functional configuration of the surgical support robot system 100. FIG. 3 is a diagram showing an example of the functional configuration of the robot control device 130. FIG. 4 is a diagram showing the external configuration of the operation unit 120 physically integrated with the robot control device 130. FIG. 5 is an enlarged view of the structure of the distal end of the follower base arm 125. FIG. 6 is an enlarged view of the detection unit 126 installed on the tool turret 124C. FIG. 7 is a view showing the top surface of the tool turret 124C. FIG. 8 is a view showing the top end surface of a surgical tool. FIG. 9 is a view showing how the camera 126A simultaneously captures images of the surgical tool 60X and mirror 61X held by the tool turret 124C. FIG. 10 is a partially enlarged perspective view showing an enlarged view of the surgical tool replacement unit 124. FIG. 11 is a partially enlarged perspective view showing an exploded view of the surgical tool replacement unit 124. FIG. 12 is a diagram showing the holding unit 1002 as viewed obliquely from below. FIG. 13 is an exploded perspective view of the holding unit 1002. FIG. 14 is a diagram showing the external configuration of the surgical tool 1004. FIG. 15 is a partially enlarged side view showing an enlarged portion of the surgical tool 1000 held by the holding unit 1002. FIG. 16 is a diagram for explaining the surgical tool replacement operation. FIG. 17 is a diagram for explaining the surgical tool replacement operation. FIG. 18 is a diagram for explaining the surgical tool replacement operation. FIG. 19 is a diagram showing a modified example of the tool turret (an example in which the mirror tilt angle α is smaller than 45 degrees). FIG. 20 is a diagram showing a modified example of the tool turret (an example in which the mirror tilt angle α is larger than 45 degrees). FIG. 21 is a diagram showing another modified example of the tool turret (a modified example in which the installation location of the mirror is changed). FIG. 22 is a diagram showing an example of the configuration of a surgical tool storage system 2200. FIG. 23 is a flowchart showing the procedure of the surgical tool detection operation executed by the detection unit 126. Fig. 24 is a diagram showing the operation procedure of the tool turret 124C when performing surgical tool detection by scanning in the surgical tool storage system 2200. Fig. 25 is a flowchart showing the processing procedure performed by the robot control device 130. Fig. 26 is a diagram showing an example of the configuration of a surgical tool management table.Fig. 27 is a diagram showing the external configuration of the console unit 110. Fig. 28 is a diagram showing the support section 2703 as viewed from the side of an operator sitting on a chair 2704.

[0016] Hereinafter, embodiments of the present disclosure will be described in the following order with reference to the drawings.

[0017] A. Overview B. Configuration of surgical support robot system B-1. Console unit B-2. Operation unit B-2-1. Imaging unit B-2-2. Follower base arm B-2-3. Tool exchange unit B-2-4. Detection unit B-3. Robot control device B-4. External network B-5. External operation terminal C. Functional configuration of robot control device D. Configuration of operation unit D-1. Follower base arm D-2. First follower, second follower, imaging unit D-2-1. Imaging unit D-2-2. First follower and second follower D-3. Tool exchange unit D-4. Detection unit E. Configuration of tool turret and surgical tool exchange operation E-1. Configuration of tool turret E-2. Tool exchange operation E-3. Summary F. Configuration and operation of surgical tool storage system F-1. Configuration of surgical tool storage system F-2. Detection operation F-3. Surgical tool detection using scanning method F-4. Processing on the robot control device side F-5. Summary G. Console unit configuration

[0018] A. Overview It is expected that multiple surgical instruments will be used in a single surgery. Therefore, the surgical robot system is equipped with an instrument exchange unit that holds multiple replacement surgical instruments. When a different surgical instrument is needed during the surgical procedure, the robot arm moves to the instrument exchange unit and the instrument is exchanged. The robot arm is then returned to its original location and the surgery is resumed.

[0019] If the surgical tools held in the surgical tool exchange unit cannot be accurately identified, automatic exchange with the appropriate surgical tool cannot be performed, and there is a risk of mistaking the wrong surgical tool. Furthermore, if the surgical tool is not properly held in the surgical tool exchange unit, the surgical tool exchange cannot be performed quickly and safely (for example, the robot arm may fail to grasp the surgical tool and drop it). Therefore, to safely perform automatic surgical tool exchange, it is necessary to properly manage the surgical tools held in the surgical tool exchange unit and prevent the surgical tools from being improperly attached.

[0020] Therefore, this disclosure proposes a surgical tool storage system that can accurately identify surgical tools held in a surgical tool exchange unit and determine the attachment state of the surgical tools. The surgical tool storage system according to the present disclosure can be incorporated into a surgical support robot system as a subsystem.

[0021] The surgical tool identification function of the surgical tool storage system according to the present disclosure utilizes visual identification information provided on the surgical tool. Specifically, the visual identification information is a two-dimensional code such as a QR (registered trademark) code. Of course, instead of a two-dimensional code, a barcode or other marker may also be used. This type of visual identification information can be used to identify the surgical tool by recognizing and processing an image captured by a camera.

[0022] For example, if the surgical tool exchange unit is composed of a rotating turret and holds surgical tools with their end effectors, such as forceps, facing downward, visual identification information is provided on the end face of the surgical tool opposite the end effector (i.e., the top end face when the end effector is facing downward).The surgical tool storage system can then recognize the visual identification information of the surgical tool from an image captured by a camera of the top end face of the surgical tool.If the camera is fixed so that the distance between the camera and the turret is constant, the camera's autofocus function is not necessary, and accurate recognition processing is possible even with a monocular camera.

[0023] Furthermore, in a mode in which the surgical tool is attached to the turret with the end effector facing downward, an example of an improperly attached surgical tool is when the surgical tool moves vertically relative to the turret (for example, when the surgical tool is floating above the turret). When the surgical tool is displaced vertically relative to the turret, the distance between the turret and the camera changes, which causes a problem of out-of-focus when capturing an image of the visual identification information on the upper end surface of the surgical tool. While using a monocular camera to recognize two-dimensional codes can simultaneously read the identification information and recognize the three-dimensional position of the surgical tool, the recognition accuracy in the vertical direction (or the optical axis direction of the camera) is relatively low.

[0024] Therefore, in the surgical tool storage system according to the present disclosure, a mirror that directs light from the side of the surgical tool attached to the turret toward the camera's imaging plane is installed in a location within the camera's imaging area along with the top end surface of the surgical tool. The camera captures an image of the side of the surgical tool via the mirror, and visual information regarding the vertical displacement of the surgical tool is added to the captured image, thereby improving the accuracy of estimating the vertical position of the surgical tool attached to the turret. Therefore, the surgical tool storage system according to the present disclosure can simultaneously read the visual identification information of the surgical tool and estimate the vertical position of the surgical tool with high accuracy using only the image captured by the monocular camera.

[0025] The mirror is installed on the turret or near the outer periphery of the turret, near the attachment point of the surgical tool, so that the mirror, together with the top surface of the surgical tool, fits within the imaging area of ​​the camera. When the mirror is attached at a 45-degree angle to the turret, vertical displacement of the surgical tool is reflected in the camera's imaging plane as an equal amount to the horizontal displacement of the top surface of the surgical tool, so that vertical displacement of the surgical tool can be accurately detected and focus deviation can be eliminated.

[0026] While using a monocular camera to recognize two-dimensional codes can read the identification information of surgical tools and recognize their three-dimensional position, the accuracy of recognition in the vertical direction (or the camera's optical axis direction) is relatively low. One reason for this is the lack of depth information provided by monocular cameras. While two-dimensional code recognition algorithms can calculate distance from changes in the size of two-dimensional codes in captured images, this method is easily affected by external disturbances such as camera distortion. While adding a camera to capture images of surgical tools attached to a turret from the side to obtain vertical information about the surgical tools is also an option, this poses challenges in terms of space and cost. In contrast, the surgical tool storage system disclosed herein uses a mirror to bring visual information regarding the vertical displacement of surgical tools into the monocular camera's field of view, thereby enabling a vertical displacement detection function for surgical tools to be realized at low cost and in a small space as a supplement to the two-dimensional code recognition function.

[0027] When the camera is fixed so that the distance between the camera and the turret is constant, if the mounting position of the surgical tool relative to the turret moves vertically, the distance between the camera and the 2D code attached to the top surface of the surgical tool changes. Therefore, unless the lens is manually focused, out-of-focus occurs, resulting in reduced accuracy in recognizing the 2D code. The out-of-focus problem is particularly serious in dark environments with a large aperture. Adding an autofocus function can solve this problem, but it complicates the camera mechanism and increases costs. In contrast, the surgical tool storage system disclosed herein uses a camera to capture the side of the surgical tool via a mirror, adding visual information regarding the vertical displacement of the surgical tool. This eliminates the out-of-focus problem when detecting the 2D code, and makes it possible to detect the vertical displacement of the surgical tool (floating from the turret) even without an autofocus function.

[0028] The surgical tool storage system according to the present disclosure can simultaneously read the two-dimensional code of the surgical tool and estimate the position of the surgical tool in real time with high accuracy using only the image captured by the monocular camera. If the surgical tool storage system determines that the surgical tool is improperly attached, it can correct the attachment state using, for example, the arm of a surgical support robot system, or sound an alarm to prompt a human to manually reattach the surgical tool.

[0029] In short, the surgical tool storage system according to the present disclosure can ensure the safety of the surgical tool replacement process and manage surgical tool usage information. Furthermore, the surgical tool storage system according to the present disclosure can reduce the effort required for reading two-dimensional codes and inspecting the attachment status of surgical tools when automatically replacing surgical tools on a turret, thereby shortening the overall time required for surgery.

[0030] Furthermore, each time a surgical tool is changed, the robot arm must move to the turret position, change the tool, and then return to the surgical site, which requires a movement operation of the robot arm. If the turret is located away from the robot arm, the movement distance of the robot arm becomes long, which takes time to change the surgical tool, lengthening the surgery time and ultimately increasing the physical burden on the patient. Therefore, it is preferable to configure a surgical support robot system so that the turret is located near the robot arm and the surgical tool held by the turret can be moved to an area where the tool can be changed by rotating the turret.

[0031] B. Configuration of the Surgical Assist Robot System Figure 1 shows the external configuration of a leader-follower type surgical assist robot system 100 to which the present disclosure is applied. Figure 2 also shows a schematic diagram of the functional configuration of the surgical assist robot system 100. In this section B, the functions of each component of the surgical assist robot system 100 will be described.

[0032] 1 and 2, the surgical robot system 100 includes a console unit 110, an operation unit 120, a robot control device 130, and an external operation terminal 140. The surgical robot system 100 can use an external network 160. The console unit 110 and the operation unit 120 correspond to the leader device and the follower device, respectively, in a leader-follower system. The robot control device 130 links the console unit 110 and the operation unit 120 to realize leader-follower control.

[0033] An operator such as a doctor or medical professional operates the console unit 110 to instruct the operation of the surgical support robot system 100. When an operation command or instruction is input by the operator, the console unit 110 transmits the operation command to the operation unit 120 via the robot control device 130.

[0034] On the other hand, the operation unit 120 is equipped with a robotic arm equipped with surgical tools required for surgery, is configured to operate in response to the operation of the console unit 110 by an operator, and is placed in the operating room near the operating table 150 on which the patient lies in order to perform surgery on the patient. The operation unit 120 operates the robotic arm (follower base arm 125) that supports the surgical tools and imaging device in accordance with operation commands received via the robot control device 130, and performs surgery on the patient.

[0035] B-1. Console Unit The console unit 110 is used when an operator remotely operates the operation unit 120 from a location away from the operating table 150 in an operating room (or outside the operating room), for example. As shown in FIG. 2, the console unit 110 is equipped with a first reader 111, a second reader 112, a foot switch unit 113, and a display device 114. It is assumed that the first reader 111 and the second reader 112 are operated by the operator's left and right hands, respectively, and that the foot switch unit 113 is operated by the operator's left or right foot or both feet. Although not shown, the console unit 110 is further equipped with a communication device capable of communicating various information with the robot control device 130.

[0036] The first reader 111 and the second reader 112 are each configured as a multi-joint robot arm. The first reader 111 and the second reader 112 are equipped with electric motors 111A and 112A for driving the respective joints of the robot arm and sensors 111B and 112B for detecting the state of the robot arm. The sensors 111B and 112B detect the torque reference value, joint angle, joint angular velocity, and other information for each joint as the state of the robot arm. The torque reference value substantially corresponds to the current value input to the electric motors 111A and 112A. The joint angle is obtained from an encoder attached to the output shaft of the electric motors 111A and 112A, and the joint angular velocity is obtained by differentiating the joint angle with respect to time. The electric motors 111A and 112A are driven by a drive signal from the robot control device 130 to perform a desired operation based on the position detection results detected by the sensors 111B and 112B, respectively. However, in this embodiment, the robot arms serving as the first reader 111 and the second reader 112 may include passive joints, and may be of a vertically multi-joint type or a horizontally multi-joint type.

[0037] The first reader 111 and the second reader 112 support a first operating unit 111C corresponding to right-handed operation and a second operating unit 112C corresponding to left-handed operation at the distal end of each robot arm. The first operating unit 111C and the second operating unit 112C can be moved vertically and horizontally, i.e., two-dimensionally, at each of the upper and lower height positions by an operator. However, since the technical scope of this disclosure is not limited to the degree of freedom configuration and structure of a specific robot arm, detailed description of the robot arm will be omitted in this specification. The first operating unit 111C and the second operating unit 112C each have a gripper consisting of a pair of grips (not shown) rotatably supported by a grip shaft and operable to open and close.

[0038] The foot switch unit 113 is configured so that an operator can use his or her feet to operate the operation unit 120 and to activate the functions of the operation unit 120. In the example shown in Fig. 2, the foot switch unit 113 includes a first surgical tool change button 113A, a second surgical tool change button 113B, and a clutch button 113C. However, the present disclosure is not limited to a specific configuration of the foot switch unit 113.

[0039] The display device 114 displays 2D or 3D images of the surgical field captured by the imaging unit 123 on the operation unit 120 side. Referring to FIG. 1 , the display device 114 is attached near the top of the console unit 110. The display device 114 may be an open-type 2D or 3D monitor with a flat screen or an immersive 2D or 3D viewer into which the operator peers. The display device 114 is installed at approximately the same height as the head of the operator seated in a chair. Therefore, the operator can operate the first reader 111 and the second reader 112 with their left and right hands and the foot switch unit 113 with their feet while observing the 2D or 3D image of the surgical field displayed on the display device 114. The display device 114 is not limited to being integrated with the console unit 110 as shown in FIGS. 1 and 2 , but may be a monitor display installed physically separate from the console unit 110. Alternatively, the display device 114 may be a head-mounted display (HMD) that is worn on the operator's head.

[0040] 2, the operation unit 120 includes a first follower 121, a second follower 122, an imaging unit 123, a surgical tool exchange unit 124, a follower base arm 125, and a detection unit 126. Although not shown, the operation unit 120 further includes a communication device capable of communicating various information with the robot control device 130.

[0041] The follower base arm 125 may be a single robot arm that commonly supports all of the first follower 121, the second follower 122, the imaging unit 123, the surgical tool exchange unit 124, and the detection unit 126, or may be composed of multiple robot arms that each support one of these. The degree of freedom configuration of the follower base arm 125 is not particularly limited. As a modified example, the imaging unit 123 may be supported by a robot arm independent of the follower base arm 125 (for example, a robot arm other than the follower base arm 125 that supports the first follower 121 and the second follower 122).

[0042] The first follower 121 is a robot arm that supports a first surgical tool 121A as an end effector at its distal end. It includes an electric motor 121B that drives each joint of the robot arm and a sensor unit 121C that detects the state of the robot arm. The sensor unit 121C detects the state of the robot arm, such as a torque reference value, joint angle, and joint angular velocity of each joint. The torque reference value substantially corresponds to the current value input to the electric motor 121B. The joint angle is obtained from an encoder attached to the output shaft of the electric motor 121B, and the joint angular velocity is obtained by differentiating the joint angle with respect to time. Similarly, the second follower 122 is a robot arm that supports a second surgical tool 122A as an end effector at its distal end. It includes an electric motor 122B that drives each joint of the robot arm and a sensor unit 122C. The robot arms that support the first follower 121 and the second follower 122 are not limited to a specific configuration or structure of degrees of freedom, and therefore a detailed description of the robot arms will be omitted in this specification.

[0043] The first follower 121 and the second follower 122 are movable up and down in conjunction with the movement of the first leader 111 and the second leader 112 by operation by the operator, and are also movable horizontally, i.e., two-dimensionally, at each height position. The movable areas of the first follower 121 and the second follower 122 include a working area in which surgery on the affected area is performed using the attached first surgical tool 121A and the second surgical tool 122A, respectively, as well as a surgical tool exchange area in which the attached first surgical tool 121A and the second surgical tool 122A are automatically exchanged for other surgical tools.

[0044] In response to the operation state of the operator on the console unit 110, the electric motors 121B and 122B are driven on the operation unit 120 side, and the first follower 121 and the second follower 122 control at least one of the position and posture of the first surgical tool 121A and the second surgical tool 122A, respectively. The first follower 121 and the second follower 122 preferably have six degrees of freedom for the position and posture of the first surgical tool 121A and the second surgical tool 122A and a degree of freedom for opening and closing the surgical tools, but may also be devices that can control at least one of the position and posture of the first surgical tool 121A and the second surgical tool 122A. Furthermore, if the first surgical tool 121A and the second surgical tool 122A are surgical tools such as tweezers or scissors, the first follower 121 and the second follower 122 may be devices that have a mechanism for driving the opening and closing of such surgical tools, and do not necessarily have an arm-like configuration.

[0045] In the surgical support robot system 100 shown in FIGS. 1 and 2 , the console unit 110 has two leaders, and the operation unit 120 has two followers. However, the number of followers may be three or more, or even just one. In the example shown in FIGS. 1 and 2 , leader-follower control is performed using one-to-one leader-follower pairs. Therefore, the number of leaders on the console unit 110 side and the number of followers on the operation unit 120 side are the same, but the number of leaders and the number of followers may be different. The number of followers corresponds to the number of surgical tools that can be used simultaneously in the surgical support robot system 100. In addition, in this embodiment, for simplicity of explanation, the first follower 121 and the second follower 122 have the same configuration, but of course, each follower may have a different configuration.

[0046] B-2-1. Imaging Unit The imaging unit 123 is a unit for capturing images of the surgical field, etc., and as shown in FIG. 2, includes a camera 123A, a light source 123B, and a camera arm 123C. The camera arm 123C is a robot arm equipped with an electric motor and an encoder (neither of which are shown) at its joints. It supports the camera 123A and the light source 123B and can move the camera 123A and the light source 123B to predetermined positions by driving the motor. The camera arm 123C supports the camera 123A so that the surgical field is within the field of view of the camera 123A and the light emitted by the light source 123B falls on the field of view of the camera 123A. The position and orientation of the camera arm 123C may be configured to be controlled by leader-follower control from the console unit 110 side.

[0047] B-2-2. Follower Base Arm The follower base arm 125 is a robot arm that supports all of the first follower 121, the second follower 122, the imaging unit 123, and the surgical tool exchange unit 124. The follower base arm 125 includes an electric motor 125A that drives each joint and a sensor unit 125B that detects the state of the follower base arm 125. The follower base arm 125 drives the electric motor 125A in response to an operation state by an operator using the console unit 110, and can move the first follower 121, the second follower 122, and the imaging unit 123, which are supported at the distal end, to predetermined positions. The sensor unit 125B detects the torque reference value, joint angle, joint angular velocity, etc. of each joint as the state of the follower base arm 125. The torque reference value substantially corresponds to the current value input to the electric motor 125A. The joint angle is obtained from an encoder attached to the output shaft of the electric motor 125A, and the joint angular velocity is obtained by differentiating the joint angle with respect to time.

[0048] B-2-3. Surgical Tool Exchange Unit The surgical tool exchange unit 124 includes a plurality of surgical tools (shown in FIG. 2 as a first exchange surgical tool 124A, a second exchange surgical tool 124B, ...) that are exchanged for the first surgical tool 121A and the second surgical tool 122A that are attached to the first follower 121 and the second follower 122, respectively, a tool turret 124C that holds the plurality of surgical tools, an electric motor 124D that rotates the tool turret 124C, and an encoder 124E that detects the rotational position of the electric motor 124D (or the tool turret 124C). When replacing the first surgical tool 121A attached to the first follower 121 or the second surgical tool 121B attached to the second follower 122 with any of the replacement surgical tools, the first replacement surgical tool 124A, the second replacement surgical tool 124B, ..., held by the tool turret 124C, the tool turret 124C is rotated in the horizontal direction using the electric motor 124D and the encoder 124E to move to an area where the surgical tools can be replaced with each of the first follower 121 and the second follower 122. In Fig. 2, for the sake of simplicity of the drawing, the tool turret 124C shows only two replacement surgical tools, the first replacement surgical tool 124A and the second replacement surgical tool 124B, but the tool turret 124C may be configured to be able to hold three or more (e.g., five) replacement surgical tools simultaneously.

[0049] In this embodiment, each surgical tool has a two-dimensional code representing identification information. The two-dimensional code is, for example, a QR (registered trademark) (Quick Response) code. For example, the QR (registered trademark) code may be marked on the surface of the surgical tool with a laser, or a sticker with the QR (registered trademark) code printed on it may be attached to the surgical tool. The tool turret 124C holds each of the first replacement surgical tool 124A and the second replacement surgical tool 124B so that they are included in the detection range of the detection unit 126, specifically so that the two-dimensional code faces upward (in other words, toward the camera 126A of the detection unit 126).

[0050] Furthermore, mirrors 124A-1, 124B-1, ... are installed near the attachment positions of the first exchange surgical tool 124A and the second exchange surgical tool 124B on the upper surface of the tool turret 124C or near the outer periphery of the tool turret 124C. Each of the mirrors 124A-1, 124B-1, ... reflects light reflected from a side surface (different from the surface on which the two-dimensional code is arranged) of the first exchange surgical tool 124A, the second exchange surgical tool 124B, ... upward (towards the camera 126A of the detection unit 126).

[0051] B-2-4. Detection Unit The detection unit 126 includes a camera 126A, a code reader 126B, and a position estimator 126C.

[0052] The camera 126A is mounted above the tool turret 124C, facing downward so that the upper end surface of the surgical tool and a mirror installed near the surgical tool (e.g., the first replacement surgical tool 124A and mirror 124A-1) are simultaneously included in the detection range (hereinafter also referred to as the "imaging area"). The camera 126A is basically a monocular camera, and the camera 126A is fixed so that its distance from the tool turret 124C is always constant. It is assumed that the camera 126A does not have an autofocus function. Of course, the camera 126A may be a stereo camera or a multi-lens camera, or may be equipped with at least one of a distance adjustment function between the camera 126A and the tool turret 124C and an autofocus function. However, omitting these functions can reduce costs.

[0053] In this embodiment, a two-dimensional code representing identification information is disposed on the top surface of each surgical tool. Code reader 126B reads the two-dimensional code contained in the image captured by camera 126A to obtain the identification information of the surgical tool. Code reader 126B further estimates the horizontal position of the surgical tool based on the coordinate position of the two-dimensional code in the captured image. Position estimator 126C detects the edge of the top surface of the surgical tool from the mirror reflection image contained in the image captured by camera 126A, and estimates the vertical displacement of the surgical tool based on the horizontal displacement of the top surface of the surgical tool observed in the image.

[0054] 2, the robot control device 130 includes a CPU (Central Processing Unit) 131, a memory unit 132, a bus 133, and an input / output (IO) unit 134 that converts and inputs / outputs various signals. The components of the CPU 131, the input / output unit 134, and the memory unit 132 are connected via the bus 133 so that signals can be transmitted to one another. The robot control device 130 may include a processing circuit such as a GPU (Graphics Processing Unit), a DSP (Digital Processing Unit), or an ASIC (Application Specific Integrated Circuit) instead of or in addition to the CPU 131. The memory unit 132 is a local memory of the CPU 131, such as a random access memory (RAM) or a read only memory (ROM), but may also include a large-capacity storage device such as a solid state drive (SSD) or a hard disc drive (HDD). The robot control device 130 may be configured using one or more general computing devices, such as personal computers (PCs).

[0055] The CPU 131 functions as an arithmetic processing unit, and reads and executes various programs stored in a memory unit 132 such as a ROM or a RAM, or an external storage device (not shown).

[0056] The RAM or ROM in the memory unit 132 stores programs and other data for executing control of the surgical assistance robot system 100. Of these, the ROM stores data such as programs and device constants used by the CPU 131 in a non-volatile manner, while the RAM temporarily stores programs used by the CPU 131 and variables that change sequentially during program execution.

[0057] The program causes the CPU 131, the RAM and ROM in the memory unit 132, and the input / output unit 134 to operate cooperatively, causing them to function as an arithmetic processing unit (described later), thereby executing each function of the robot control device 130. The program is written in, for example, a low-level language such as machine language or assembly language, or a high-level language such as C language.

[0058] The input / output unit 134 includes a communication device, a D / A converter, an A / D converter, a motor drive circuit, etc. (all of which are not shown). The input / output unit 134 also includes an interface that connects external devices, actuators, encoders, and other various sensors (not shown) to the robot control device 130. However, in Fig. 2, the components within the input / output unit 134 are not shown in order to simplify the drawing.

[0059] The communication device included in the input / output unit 134 is configured to be able to send and receive signals, etc. between the Internet and other communication devices in accordance with a specified protocol such as TCP / IP, EtherCAT (registered trademark) or EtherNet / IP (registered trademark) used as an industrial network.

[0060] The input / output unit 134 may be configured to connect the robot control device 130 to a storage device that stores various types of data or a drive device that is a reader / writer for recording media. The storage device may be realized, for example, by a magnetic storage device such as an HDD, a semiconductor storage device such as an SSD, an optical storage device, or a magneto-optical storage device. The input / output unit 134 may also be configured to have an interface such as a USB (Universal Serial Bus) or an HDMI (High Definition Multimedia Interface), and to be connected to various other peripheral devices such as a printer.

[0061] 1 and 2, the robot control device 130 is configured as a physically integrated unit (i.e., a single housing) in which the control devices for the console unit 110, the operation unit 120, and the imaging unit 123 are each configured as an integrated unit. As a modified example, the dedicated control devices for the console unit 110, the operation unit 120, and the imaging unit 123 may each be physically separated (i.e., each as a separate housing) to configure a unit similar to the robot control device 130, and the control devices may be connected to each other by wire or wirelessly, or a unit similar to the robot control device 130 may be configured as a collection of multiple control devices.

[0062] B-4. External Network The external network 160 is a wired or wireless transmission path for information transmitted from devices connected to the network. The network referred to here may include public network such as the Internet, telephone network, or satellite communication network, as well as various LANs (Local Area Networks) including Ethernet (registered trademark), and WANs (Wide Area Networks). The network may also include dedicated network such as an IP-VPN (Internet Protocol-Virtual Private Network).

[0063] The robot control device 130 is connected to an external network 160 by a communication device (described above) within the input / output unit 134. The robot control device 130 may share data regarding surgical information such as the surgical tools used and the operating status of the surgical support robot system 100 with multiple other surgical support robot systems and upper control devices (not shown) that are connected to each other via the external network 160. The upper control device may be, for example, a PC, various servers, or a combination of these, and monitors the operating status of each surgical support robot system and performs statistical data analysis.

[0064] B-5. External Operation Terminal The external operation terminal 140 (not shown in FIG. 1) is externally connected to the robot control device 130, and is configured to receive and display the operating status of the surgical support robot system 100, as well as to allow an assistant to input commands and set parameters. As shown in FIG. 2, the external operation terminal 140 includes an input unit 141 and a display unit 142.

[0065] The input unit 141 consists of, for example, one or a combination of two or more of a keyboard, mouse, button, switch, infrared or other remote control means using radio waves, and is configured to be operated by the operator and assistant to input commands to the surgical support robot system 100 (for example, instructions to attach the initial surgical tools when starting up the system) and to set parameters adjusted for each operator.

[0066] The display unit 142 receives and displays the operating status of the surgical support robot system 100 from the robot control device 130, allowing the operator and assistant to visually and intuitively recognize this information. The display unit 142 may be a display device such as a liquid crystal panel or a light, or may be a sound device that presents information by alert sound or voice guidance, and may be set to issue an alert when a system error occurs, for example. The display unit 142 may also be configured as an integrated unit with an input unit, like a tablet terminal, and may be equipped with an application for input and status display.

[0067] 1 and 2, the main components of the surgical support robot system 100 have been described, but each component may be configured using general-purpose parts or may be configured using hardware specialized for the function of each component. The configuration used for each component can be changed as appropriate depending on the technical level at the time of implementing the present disclosure.

[0068] C. Functional Configuration of the Robot Control Device In Section A above, it was mentioned that the CPU 131, the RAM and ROM in the memory unit 132, and the input / output unit 134 operate cooperatively by executing a program within the robot control device 130 to function as the arithmetic processing unit of the surgery support robot system 100. In this Section B, the specific functional configuration of the robot control device 130 will be described.

[0069] 3 shows an example of the functional configuration of the robot control device 130. In the example shown, the robot control device 130 includes a calculation processing unit 310 and an input / output interface 320. The calculation processing unit 310, as the calculation processing unit of the surgery support robot system 100, includes an input / output processing unit 311, a drive control unit 312, an image generation unit 313, and a display control unit 314.

[0070] The input / output processing unit 311 acquires signals output from the camera 123A, sensors, detection unit 126, etc., performs signal processing such as filtering, and outputs the signals to each functional unit. The input / output processing unit 311 also outputs the outputs from each functional unit to other system components such as actuators.

[0071] The drive control unit 312 controls the movement of the first follower 121 and the second follower 122 by leader-follower control based on operations performed by the operator on the first leader 111 and the second leader 112. This makes it possible to change the positions and postures of the first surgical tool 121A and the second surgical tool 122A attached to the first follower 121 and the second follower 122, respectively (when the first surgical tool 121A and the second surgical tool 122A are surgical tools that perform opening and closing operations, leader-follower control is also performed for the opening and closing operations). Furthermore, the drive control unit 312 controls the movement of the first leader 111 and the second leader 112 based on sensor information from the first follower 121 and the second follower 122, thereby presenting a haptic sensation to the operator. The drive control unit 312 may apply a bilateral control method so that the operator can remotely operate the operation unit 120 from the console unit 110 while recognizing the state of the operation unit 120, but details of leader-follower control will be omitted. The drive control unit 312 is configured to drive and control the camera arm 123C to move the imaging unit 123 and change the field of view of the camera 123A. The drive control unit 312 is also configured to control the drive of the operation unit 120 based on identification information and position information of the surgical tool (displacement of the horizontal position and vertical position of the surgical tool relative to the tool turret 124C) output from the detection unit 126.

[0072] The image generation unit 313 generates a surgical tool selection UI, which is a graphical user interface (GUI) for selecting a surgical tool, in accordance with input from the operator and the state of the surgical support robot system 100. The display control unit 314 superimposes the surgical tool selection UI generated by the image generation unit 313 on the surgical field image captured by the camera 123A and displays the superimposed image on the screen of the display device 114 of the console unit 110 or the display unit 142 of the external operation terminal 140. It is preferable that the display control unit 314 displays the GUI in a manner that does not interfere with viewing the surgical field image captured by the camera 123A. The display control unit 314 may also include functions for adjusting the imaging magnification of the camera 123A and the light source 123B on the screen.

[0073] Some or all of the functions for executing the processing of each of the functional modules 311 to 314 of the arithmetic processing unit 310 can be realized using a dedicated hardware circuit, an integrated circuit customized for a specific application such as an ASIC, or a programmable logic circuit such as an FPGA (Field Programmable Gate Array). Furthermore, there is no particular limit to the number of computers (such as PCs) that execute the programs related to the processing in the arithmetic processing unit 310. For example, multiple computers, including a higher-level control device, may cooperate with each other to execute the programs related to the processing in the arithmetic processing unit 310.

[0074] Furthermore, the storage device storing various programs for realizing the functions of the arithmetic processing unit 310 may be, but is not limited to, a memory incorporated in the robot controller 130. The programs may be recorded in a computer-readable storage device that is distributed separately from the robot controller 130 to provide the programs. Alternatively, the programs may be downloaded via a network such as the external network 160 and installed or updated in a computer that operates as the robot controller 130.

[0075] The surgical support robot system 100 has two operating modes, for example, a procedure mode and a surgical tool exchange mode. In the procedure mode, a first follower 121 and a second follower 122, each equipped with a surgical tool, are driven by the operation unit 120 in conjunction with the operation of the first leader 111 and the second leader 112 by an operator on the console unit 110, through leader-follower control, to perform surgery. In addition, in the surgical tool exchange mode, the leader-follower control is interrupted, and the surgical tool of at least one of the first follower 121 and the second follower 122 is exchanged through a surgical tool exchange sequence (described later). The operator can instruct a transition of the operating mode by performing a predetermined operation on the foot switch unit 113, the first leader 111, the second leader 112, or the like. The robot control device 130 performs processing to control leader-follower control in the procedure mode, transition of operation modes in response to operator operations, presentation of an instrument selection UI in the instrument exchange mode, and execution of an instrument exchange sequence for the selected instrument.

[0076] 4 shows the external structure of the robot control device 130 that is physically integrated with the operation unit 120. In the example shown in FIG. 4, a housing 400 of the robot control device 130 that houses the above-mentioned components 131 to 134 (not shown in FIG. 4) is connected at its top surface 401 to a support column 411 at the base of the follower base arm 125 of the operation unit 120. In addition, a plurality of (e.g., four) legs 402 with wheels 403 are provided at the bottom of the housing 400 of the robot control device 130 as a means of transportation. Therefore, an operator or an assistant can manually push the housing 400 of the robot control device 130 and the operation unit 120 to move them in and out of the operating room.

[0077] However, the housing 400 of the robot control device 130 and the operation unit 120 may be physically separated. In this case, the housing 400 of the robot control device 130 may be fixedly installed in a predetermined location inside or outside the operating room. Furthermore, the operation unit 120, which is separate from the robot control device 130, may be directly attached and fixed to the floor of the operating room, or may have wheeled legs (not shown) at the bottom ends of the support columns 411 as a means for moving. As yet another variation, the robot control device 130 may be physically integrated with the console unit 110 instead of the operation unit 120.

[0078] D. Configuration of the Operation Unit In this section C, the specific configuration of the operation unit 120 will be described in more detail.

[0079] 4 shows the external structure of the operation unit 120 that is physically integrated with the robot control device 130. In the example shown in FIG. 4, a support column 411 at the base of the follower base arm 125 of the operation unit 120 is fixed vertically to the top surface 401 of the housing 400 of the robot control device 130. In addition, the bottom of the housing 400 of the robot control device 130 is provided with a plurality of legs 402 (for example, four) with wheels 403 as a means of transportation. Therefore, an operator or an assistant can manually push the housing 400 of the robot control device 130 and the operation unit 120 to move them in and out of the operating room.

[0080] However, the housing 400 of the robot control device 130 and the operation unit 120 may be physically separated. In this case, the bottom of the operation unit 120 may be equipped with a plurality of (for example, four) legs 402 with wheels 403 as a means of transportation, so that the operation unit 120 can be freely moved by being pushed by hand by an operator or an assistant.

[0081] As explained in Section A above with reference to Figure 2, the operation unit 120 consists of a first follower 121, a second follower 122, an imaging unit 123, a surgical tool exchange unit 124, a detection unit 126, and a follower base arm 125 that supports these units together at their distal ends.

[0082] D-1. Follower Base Arm The follower base arm 125 is a single robot arm that commonly supports all of the first follower 121, the second follower 122, the imaging unit 123, the surgical tool replacement unit 124, and the detection unit 126. However, the follower base arm 125 may be configured with a plurality of robot arms that individually support the first follower 121, the second follower 122, the imaging unit 123, the surgical tool replacement unit 124, and the detection unit 126. In the following, the follower base arm 125 will be described as being a single robot arm, as shown in FIG.

[0083] Referring to FIG. 4 , the follower base arm 125 has a serial link structure in which, from the base, a support 411, a first link 412, and a second link 413 are connected. The support 411 is a columnar member and is connected at its base to the top surface 401 of the housing 400 of the robot control device 130. The support 411 may be configured to be extendable in the vertical direction or may have a fixed length. The first link 412 is connected at its base to the upper end of the support 411 via a first joint 421. The second link 413 is connected at its base to the distal end of the first link 412 via a second joint 422, and at its distal end, commonly supports the first follower 121, the second follower 122, the imaging unit 123, the surgical tool exchange unit 124, and the detection unit 126, which are major components of the operation unit 120, via a third joint 423.

[0084] The first joint 421, the second joint 422, and the third joint 423 have degrees of freedom of rotation about vertically extending yaw axes 421A, 422A, and 423A, respectively, but may also have degrees of freedom of rotation about other axes. In addition, one or more links are further connected to the distal end side of the second link 413, and the distal end of the most distal link is configured to support the first follower 121, the second follower 122, the imaging unit 123, the surgical tool replacement unit 124, and the detection unit 126.

[0085] The first joint 421, the second joint 422, and the third joint 423 are assumed to be active joints that are rotationally driven by electric motors, but some or all of the joints may be passive joints. An operator or an assistant may manually operate the rotation angles of the passive joints to adjust the positions and attitudes of the first follower 121, the second follower 122, and the imaging unit 123. The follower base arm 125 may include a passive joint equipped with a locking mechanism, and may be a vertical multi-joint type or a horizontal multi-joint type, for example.

[0086] 2, for simplicity, the follower base arm 125 is illustrated in an abstract manner as including an electric motor 125A and a sensor unit 125B that detects the state of the follower base arm 125. In reality, the electric motor 125A includes multiple motors that individually drive the joints 421 to 423. The sensor unit 125B includes encoders that detect the joint angles and are arranged in the joints 421 to 423. The follower base arm 125 is driven through the rotational movement of the electric motor 125A that follows the operating state of the operator at the console unit 110, and can move the first follower 121, the second follower 122, and the imaging unit 123, which are supported at the distal end, to predetermined positions.

[0087] D-2. First Follower, Second Follower, and Imaging Unit Figure 5 shows an enlarged view of the structure of the distal end of the follower base arm 125, illustrating in more detail the structure in which the follower base arm 125 (second link 413) supports each of the first follower 121, the second follower 122, the imaging unit 123, the surgical tool replacement unit 124, and the detection unit 126 on the distal end side.

[0088] D-2-1. Imaging Unit The imaging unit 123 is a unit for capturing images of the surgical field, etc., and as shown in FIG. 5, it includes a camera 123A that can capture moving or still images, a light source 123B that illuminates the field of view of the camera 123A, and a camera arm 123C.

[0089] The camera arm 123C is a robot arm equipped with electric motors and encoders (neither of which is shown) at its joints. The camera arm 123C supports the camera 123A and the light source 123B at its distal end and can move the camera 123A and the light source 123B to predetermined positions by driving the motor. Specifically, the camera arm 123C has a mechanism (zoom mechanism) that moves the camera 123A up and down along the yaw axis 423A. As can be seen from FIG. 5 , the camera arm 123C has a base (or root) connected to the distal end of the follower base arm 125 (strictly speaking, connected to the distal end of the second link 413 via the third joint 423), and supports the camera 123A and the light source 123B facing downward so that the surgical field (near the first surgical tool 121A and the second surgical tool 122A) is within the field of view of the camera 123A. In this embodiment, all of the joints of the camera arm 123C are active joints driven by electric motors, but some or all of the joints may be changed to passive joints as appropriate. An operator or an assistant may adjust the rotation angle of the passive joints by manual operation. However, since the present disclosure is not limited to a specific structure of the camera arm 123C, a detailed description of the structure of the camera arm 123C will be omitted.

[0090] D-2-2. First Follower and Second Follower The first follower 121, the second follower 122, the surgical tool exchange unit 124, and the detection unit 126 are supported by a follower support arm 521 branched off from the camera arm 123C. When the third joint 423 is driven, the camera arm 123C rotates around the yaw axis 423A, which allows the follower support arm 521 to rotate and move horizontally.

[0091] The first follower 121 includes a first mounting part 501 to which a first surgical tool 121A serving as an end effector is replaceably mounted, and a first robot arm 511 that supports at its distal end the first mounting part 501. As shown in Fig. 2, the first follower 121 includes an electric motor 121B that drives each joint of the first robot arm 511, and a sensor part 121C that detects the state of the first robot arm 511. The sensor part 121C includes an encoder that detects the joint angle.

[0092] Similarly, as shown in Fig. 5, the second follower 122 includes a second mounting part 502 to which a second surgical tool 122A is replaceably attached, and a second robot arm 512 that supports the second mounting part 502 at its distal end. Also, as shown in Fig. 2, the second follower 122 includes an electric motor 122B that drives each joint of the second robot arm 512, and a sensor part 122C that detects the state of the second robot arm 512. In the example shown in Fig. 5, for convenience, the first robot arm 511 and the second robot arm 512 have the same configuration, but of course the robot arm configurations may differ for each follower.

[0093] The first surgical tool 121A and the second surgical tool 122A are, for example, tweezers, a needle holder, scissors, an electric scalpel, or any other type of treatment tool. The first mounting unit 501 and the second mounting unit 502 are equipped with a mechanism for attaching and detaching the surgical tools, and are configured so that the first surgical tool 121A and the second surgical tool 122A can be replaced by driving an actuator, respectively. For example, Patent Document 2 discloses an example of a surgical tool attachment and detachment mechanism. When surgical tools that open and close, such as tweezers, a needle holder, or scissors, are attached as the first surgical tool 121A and the second surgical tool 122A, the first follower 121 and the second follower 122 can drive such surgical tools to open and close.

[0094] The first follower 121 and the second follower 122 are disposed on either side of the follower support arm 521. Specifically, the first robot arm 511 and the second robot arm 512 are attached at symmetrical positions across the follower support arm 521 via a root side drive unit 522 and a tip side drive unit 523. The root side drive unit 522 rotates the tip side drive unit 523 about a yaw axis 522A that extends vertically. The tip side drive unit 523 rotates about a horizontal pitch axis 523A that is perpendicular to the yaw axis 522A. The motors that drive the root side drive unit 522 and the tip side drive unit 523 are included in the electric motor 121A in FIG. 2 .

[0095] The base side drive unit 522 rotates the first robot arm 511 and the second robot arm 512 about a yaw axis 522A. The tip side drive unit 523 rotates the first robot arm 511 and the second robot arm 512 about a pitch axis 523A. The first robot arm 511 and the second robot arm 512 then displace the position and orientation of the first surgical tool 121A and the second surgical tool 122A, respectively, relative to the tip side drive unit 523. It is preferable that the first robot arm 511 and the second robot arm 512 each have six degrees of freedom for the position and orientation of the first surgical tool 121A and the second surgical tool 122A, respectively, and a degree of freedom for opening and closing the surgical tools, but they may also be devices that can control at least one of the position and orientation of the first surgical tool 121A and the second surgical tool 122A.

[0096] The surgical support robot system 100 employs a leader-follower control system. That is, in response to the operation of the first leader 111 and the second leader 112 by the operator on the console unit 110, the electric motors 121B, 122B, and 125A are driven on the operation unit 120 side to operate the first follower 121 and the second follower 122. With regard to the first follower 121, the base side drive unit 522 and the tip side drive unit 523 are each driven to rotate, and the first robot arm 511 is driven to control the position and posture of the end effector of the first surgical tool 121A held by the first robot arm 511. It should be understood that there is interlocking between the first leader 111 and the first follower 121, and between the second leader 112 and the second follower 122, and that the followers operate cooperatively when the leader is operated.

[0097] D-3. Surgical Tool Exchange Unit The surgical tool exchange unit 124 has the function of holding replacement surgical tools and supplying the replacement surgical tools to the first follower 121 and the second follower 122 at the time of exchange. As shown in Fig. 2, the surgical tool exchange unit 124 includes a plurality of surgical tools (including a first replacement surgical tool 124A and a second replacement surgical tool 124B) that are exchangeable with the first surgical tool 121A and the second surgical tool 122A that are attached to the first follower 121 and the second follower 122, respectively, a tool turret 124C that holds the plurality of surgical tools, an electric motor 124D that rotates the tool turret 124C, and an encoder 124E that detects the rotational position of the electric motor 124D (or the tool turret 124C).

[0098] The tool turret 124C is connected to the distal end of the follower support arm 521 via an electric motor 124D. The tool turret 124C is horizontally rotatable around the output shaft of the electric motor 124D and its rotational position can be adjusted. The tool turret 124C is installed within a range where the first follower 121 and the second follower 122 can move to replace surgical tools, but it is preferable to install the tool turret 124C near the first follower 121 and the second follower 122 to shorten the replacement time.

[0099] When the first surgical tool 121A attached to the first follower 121 is replaced with the first replacement surgical tool 124A held by the tool turret 124C, the electric motor 124D and the encoder 124E are used to position the rotational position of the tool turret 124C, and the first replacement surgical tool 124A is moved to a predetermined position (an area where the surgical tool is replaced with the first follower 121). The same applies when the second surgical tool 121B attached to the second follower 122 is replaced with the second replacement surgical tool 124B.

[0100] In this embodiment, a two-dimensional code representing identification information is attached to the upper end surface of each surgical tool (for example, a sticker with a two-dimensional code printed on it is attached). The two-dimensional code is, for example, a QR (registered trademark) code. The tool turret 124C holds each replacement surgical tool (first replacement surgical tool 124A, second replacement surgical tool 124B, ...) with the two-dimensional code facing upward (in other words, toward the camera 126A of the detection unit 126). Specifically, the tool turret 124C holds each of the first replacement surgical tool 124A and the second replacement surgical tool 124B with the end effector (forceps, tweezers, needle holder, scissors, electric scalpel, etc.) facing downward and the distal end side that couples with the robot arm facing upward.

[0101] 5, mirrors 124A-1, 124B-1, ... are installed on the tool turret 124C or near the outer periphery of the tool turret 124C near the mounting positions of the first exchange surgical tool 124A, the second exchange surgical tool 124B, .... The mirrors 124A-1, 124B-1, ... are attached at an inclination angle of 45 degrees with respect to the upper surface (or the horizontal plane) of the tool turret 124C, for example, and reflect light from the side surfaces of the first exchange surgical tool 124A and the second exchange surgical tool 124B upward (to the camera 126A of the detection unit 126).

[0102] D-4. Detection Unit The detection unit 126 has the function of detecting each surgical tool held in the surgical tool exchange unit 124. As shown in FIGS. 4 and 5, the detection unit 126 is supported at the distal end of the follower support arm 521. As shown in FIG. 2, the detection unit 126 includes a camera 126A, a code reader 126B, and a position estimation unit 126C. The camera 126A is basically a monocular camera, and is fixed so that the distance from the tool turret 124C is always constant. In this embodiment, it is assumed that the camera 126A does not have an autofocus function.

[0103] Fig. 6 shows an enlarged view of the detection unit 126 (or camera 126A) installed on the tool turret 124C. Fig. 7 shows the tool turret 124C as viewed from directly above. Fig. 8 shows the top surface of one surgical tool.

[0104] As shown in Figures 6 and 7, the tool turret 124C has a fan-shaped, plate-like shape and holds five replacement surgical tools 601, 602, ..., 605 at approximately equal intervals along an arc, with the end effectors (forceps, tweezers, needle holders, scissors, electric scalpels, etc.) facing downward. Also, as shown in Figures 7 and 8, a two-dimensional code representing identification information is attached to the upper end surface of each surgical tool 601, 602, ..., 605 (e.g., the two-dimensional code is laser-marked or a sticker with a printed two-dimensional code is attached). The two-dimensional code is, for example, a QR (registered trademark) code and is approximately 5 mm x 5 mm in size. Also, as shown in Figure 7, the tool turret 124C has mirrors 611, 612, ..., 615 installed near the holding positions of each surgical tool 601, 602, ..., 605. Each of the mirrors 611, 612, ..., 615 is attached at an inclination angle of 45 degrees with respect to the upper surface (or horizontal plane) of the tool turret 124C, and reflects light from the side of the corresponding surgical tool 601, 602, ..., 605 upward (toward the camera 126A of the detection unit 126). The bottom part of Figure 7 also shows a side view of one surgical tool 60X held by the tool turret 124C and a mirror 61X placed near the surgical tool 60X at an inclination angle of 45 degrees (where X is an integer from 1 to 5).

[0105] As shown in FIG. 6 , camera 126A is positioned almost directly above tool turret 124C so as to capture the upper surfaces of each surgical tool 601, 602, ..., 605 held by tool turret 124C. Specifically, camera 126A is installed facing downward above tool turret 124C in a position where the imaging area can simultaneously include the upper end surfaces of the surgical tools and mirrors installed near the surgical tools. By rotating motor 124D to position the rotational position of tool turret 124C, a desired combination of surgical tools and mirrors can be included in the imaging area directly below camera 126A. In the example shown in FIG. 7 , surgical tool 603 and the corresponding mirror 613, both located at the center of tool turret 124C, are included in the imaging area of ​​camera 126A. Note that the imaging area shown in FIG. 7 is a portion of camera 124A shown in FIG. 6 and can be acquired using, for example, a region of interest (ROI) method. Making the imaging area smaller than the field of view of camera 124A has the advantage of reducing the computational costs of code reader 126B, position estimator 126C, etc. It should also be understood that edge detection of the upper end surface of the surgical tool by position estimator 126C is effective only within the imaging area range shown in FIG.

[0106] The images of the upper end surface of the surgical tool and the mirror simultaneously captured by camera 126A are supplied to code reader 126B and position estimator 126C. Code reader 126B reads a two-dimensional code (e.g., a QR (registered trademark) code) from the image of the upper end surface of the surgical tool included in the image captured by camera 126A to obtain the identification information of the surgical tool. The two-dimensional code may be an encoded string of characters describing the identification information of the surgical tool, or a URI (Uniform Resource Indicator) describing the location of the identification information. The identification information may be information indicating the type of surgical tool or individual identification information that can identify each individual surgical tool. Code reader 126B further estimates the horizontal position of the surgical tool based on the coordinate position of the two-dimensional code in the image captured by camera 126A.

[0107] Furthermore, the position estimation unit 126C detects the horizontal position of the upper end surface of the surgical tool from the mirror reflection image included in the image captured by the camera 126A, and estimates the vertical position of the surgical tool based on the detection result. Specifically, the position estimation unit 126C estimates the vertical displacement of the surgical tool relative to the tool turret 124C.

[0108] 2, the code reading unit 126B and the position estimation unit 126C are shown as components inside the detection unit 126 (i.e., inside the operation unit 120) together with the camera 126A, but they may also be components located outside the operation unit 120. In this case, images of the surgical tool and mirror captured by the camera 126A are output to the outside of the operation unit 120 and supplied to the code reading unit 126B and the position estimation unit 126C. The code reading unit 126B and the position estimation unit 126C may also be components of the robot control device 130 (for example, programs executed by the CPU 131).

[0109] 9 shows a side view of the camera 126A simultaneously capturing images of the surgical tool 60X and mirror 61X held by the tool turret 124C. The camera 126A can simultaneously capture images of the surgical tool 60X and mirror 61X positioned directly below it by the rotational drive of the motor 124D. When the mirror 61X is attached at an inclination angle of 45 degrees to the tool turret 124C, the vertical displacement ΔZ of the surgical tool 60X is reflected as an amount equivalent to the horizontal displacement ΔX on the imaging surface of the camera 126A. Therefore, the actual displacement ΔZ of the upper end of the surgical tool can be calculated using the following formula based on the displacement ΔX of the upper end of the surgical tool reflected on the mirror 61X: ΔX = ΔZ

[0110] The left side of Fig. 9 shows a state in which the surgical tool 60X is properly attached to the tool turret 124C, while the right side of Fig. 9 shows a state in which the surgical tool 60X is improperly attached, floating by ΔZ from the normal vertical position. The upward displacement of the surgical tool 60X appears as an equivalent horizontal displacement ΔX on the image captured by the camera 126A.

[0111] If the upward displacement ΔZ of the surgical tool 60X is large based on the estimation result of the position estimation unit 126C, it can be determined that the surgical tool 60X is floating above the tool turret 124C and is unstable in attachment. Furthermore, by integrating the horizontal position of the surgical tool estimated by the code reading unit 126B and the vertical displacement of the surgical tool estimated by the position estimation unit 126C, it can be determined whether the attachment state of the surgical tool is good or bad.

[0112] While FIG. 9 shows an example in which the mirror 61X is mounted at a 45-degree inclination angle relative to the tool turret 124C, the inclination angle α at which the mirror 61X is mounted is not limited to 45 degrees. However, if the inclination angle α of the mirror 61X is not 45 degrees, the vertical displacement ΔZ of the surgical tool 60X will not be equivalent to the horizontal displacement ΔX on the imaging surface of the camera 126A (i.e., ΔX ≠ ΔZ). Therefore, the horizontal displacement ΔX calculated from the image captured by the camera 126A must be converted to the vertical displacement ΔZ based on the geometric relationship of the optical system. If the inclination angle α at which the mirror 61X is mounted is changed from 45 degrees, the displacement ΔX of the upper end of the surgical tool reflected on the mirror 61X will be smaller than the actual displacement ΔZ (i.e., ΔX < ΔZ). Therefore, changing the inclination angle α of the mirror 61X increases the range of the surgical tool 60X reflected on the mirror 61X, thereby expanding the detectable range of ΔZ. Changing the tilt angle α of the mirror 61X also has the effect of improving flexibility in mechanism design.

[0113] 19 shows a side view of the camera 126A simultaneously capturing images of the surgical tool 60X and mirror 61X held by the tool turret 124C when the tilt angle α of the mirror 61X relative to the tool turret 124C is less than 45 degrees. When the tilt angle α is less than 45 degrees, the actual displacement ΔZ of the upper end of the surgical tool can be calculated using the following formula based on the displacement ΔX of the upper end of the surgical tool reflected on the mirror 61X: ΔZ = ΔX / sin(2α)

[0114] 20 shows a side view of the camera 126A simultaneously capturing images of the surgical tool 60X and mirror 61X held by the tool turret 124C when the tilt angle α of the mirror 61X relative to the tool turret 124C is greater than 45 degrees. When the tilt angle α is greater than 45 degrees, the actual displacement ΔZ of the upper end of the surgical tool can be calculated using the following formula based on the displacement ΔX of the upper end of the surgical tool reflected on the mirror 61X: ΔZ = ΔX / cos(2α - π / 2)

[0115] Up to this point, we have described an example in which the mirror 61X is installed on the tool turret 124C, but the installation location of the mirror 61X can be changed as appropriate within the range in which it fits within the imaging area of ​​the camera 126A together with the surgical tool 60X held on the tool turret 124C.

[0116] FIG. 21 shows a modified example in which the mirror 61X is installed in a location separate from the tool turret 124C. In the example shown in FIG. 21, the mirror 61X is supported on a base 2101 (separate from the tool turret 124C) installed near the outer periphery of the tool turret 124C. The mirror 61X is positioned adjacent to the surgical tool 60X held by the tool turret 124C, allowing both the surgical tool 60X and the mirror 61X to be included in the imaging area and simultaneously imaged by the camera 126A directly above. While FIG. 21 shows an example in which the mirror 61X is installed at an inclination angle of 45 degrees, it may also be installed at an inclination angle α other than 45 degrees. The horizontal displacement ΔX calculated from the image captured by the camera 126A can be converted to a vertical displacement ΔZ based on the geometric relationship of the optical system (same as above).

[0117] During surgery, the tool turret 124C and the surgical tool 60X held by the tool turret 124C may become contaminated with blood. As shown in FIG. 21 , if the mirror 61X is installed in a location separate from the tool turret 124C, contamination of the mirror 61X can be avoided, allowing the camera 126A to capture a good image of the surgical tool. Separating the mirror 61X from the tool turret 124C also facilitates maintenance such as component replacement and cleaning. Separating the mirror 61X from the tool turret 124C also contributes to the compactness of the tool turret 124C and improved flexibility in mechanical design. The method for installing the base 2101 for supporting the mirror 61X near the outer periphery of the tool turret 124C is not particularly limited.

[0118] E. Tool Turret Configuration and Tool Exchange Operation This section E provides a detailed description of a specific configuration example of the tool turret 124C, a major component of the tool exchange unit 124, and the tool exchange operation. The tool turret 124C holds multiple tools and is connected to the distal end of the follower support arm 521 via an electric motor 124D. When exchanging a tool, the tool turret 124C is positioned using the electric motor 124D and the encoder 124E so that the desired tool to be exchanged is in a predetermined position.

[0119] E-1. Tool Turret Configuration Fig. 10 shows a partially enlarged perspective view of the vicinity of the surgical tool exchange unit 124 of the operation unit 120. Fig. 11 shows a partially enlarged perspective view of the surgical tool exchange unit 124 in an exploded state. However, for the sake of simplicity of the drawings and ease of explanation, the illustration and explanation of the mirror that reflects the side surface of the exchange surgical tool held in the tool turret 124C have been omitted as appropriate.

[0120] The tool turret 124C has a detachable part 1001 and a holding unit 1002. In the description here, the detachable part 1001 and the holding unit 1002 are described as separate components, but the detachable part 1001 and the holding unit 1002 may be formed integrally.

[0121] 11 , the detachable part 1001 has a coupling part 1001a that couples with the output shaft 1011 of the electric motor 124D. A hole into which the output shaft 1011 is inserted is formed in the coupling part 1001a. By inserting the output shaft 1011 into the coupling part 1001a, the detachable part 1001 is attached to the distal end of the follower support arm 521 via the electric motor 124D, and rotates together with the output shaft 1011 around the rotation axis 1011A. The detachable part 1001 also has a second coupling part 1001b to which the holding unit 1002 is attached.

[0122] As shown in FIG. 10 , the holding unit 1002 is a member that holds multiple surgical tools 1004. As shown in FIG. 11 , the holding unit 1002 has a fan-shaped, plate-like shape with an arc-shaped outer edge 1002a in a plan view. As can be seen from FIG. 10 , the arc-shaped outer edge 1002a overlaps with a portion of a circle centered on the rotation axis 1011A of the output shaft 1011. On the outer edge 1002a of the holding unit 1002, multiple holding portions 1003 (five in the example shown in FIGS. 10 and 11 ) that each hold a surgical tool 1004 are formed so as to be arranged at approximately equal intervals on the arc. In other words, the multiple holding portions 1003 are formed on an arc centered on the output shaft 611, and the holding unit 1002 holds multiple surgical tools 1004 along the arc. Then, by positioning the rotational position of the output shaft 1011 by driving the electric motor 124D, any one of the desired surgical tools 1004 held in the holding unit 1002 can be moved to a predetermined position (for tool replacement).

[0123] Fig. 12 shows the holding unit 1002 as viewed obliquely from below. Fig. 13 shows the holding unit 1002 in a disassembled state as viewed obliquely. As shown in Fig. 13, the holding unit 1002 is made up of a first plate portion 1201 and a second plate portion 1202. By stacking the first plate portion 1201 and the second plate portion 1202 in the thickness direction, a single plate-shaped holding unit 1002 as shown in Fig. 10 is assembled. The outer edge 1002a of the above-mentioned holding unit 1002 is the outer edge 1201a of the first plate portion 1201.

[0124] The first plate portion 1201 is fan-shaped. Grooves 1203 are formed on the outer edge 1201a of the first plate portion 1201 at positions corresponding to the plurality of holding portions 1003 of the holding unit 1002. Therefore, the plurality of grooves 1203 are formed on the outer edge 1201a of the first plate portion 1201 so as to be arranged at approximately equal intervals on an arc. Each groove 1203 has a width that allows a surgical tool to be inserted.

[0125] The second plate portion 1202 has a base portion 1211, which is attached to the second coupling portion 1001b of the detachable portion 1001. The base portion 1211 is provided with leaf springs 1212 and elastic portions 1213 at positions corresponding to the plurality of holding portions 1003 of the holding unit 1002. The base portion 1211 is provided in an area that does not overlap with the grooves 1203 formed in the first plate portion 1201 in a plan view.

[0126] 13 , a rectangular groove 1211a is formed on the upper surface of the base 1211 (the surface that joins with the first plate 1201). By overlapping the first plate 1201 and the second plate 1202 in the thickness direction, the upper part of the groove 1211a is blocked by the first plate 1201, forming a hole into which the second coupling portion 1001b of the detachable portion 1001 is inserted. The depth of the groove 1211a is approximately the same as the thickness of the second coupling portion 1001b. Therefore, by inserting the second coupling portion 1001b into the groove 1211a formed in the base 1211, the holding unit 1002 is attached to the detachable portion 1001. The hole into which the second coupling portion 1001b is inserted may be formed by a groove formed in the first plate portion 1201 instead of the second plate portion 1202, or may be formed by grooves formed in both the first plate portion 1201 and the second plate portion 1202. When the detachable portion 1001 and the holding unit 1002 are integrated, when the detachable portion 1001 is rotated around the rotation axis 1011A by the electric motor 124D, the holding unit 1002 and the detachable portion 1001 rotate horizontally as one unit.

[0127] As can be seen in Figure 13, the leaf springs 1212 extend radially outward from the base 1211 of the fan-shaped member. With the first and second leaf portions 1201 and 1202 stacked, the leaf springs 1212 are positioned on both sides of a groove 1203 formed in the first leaf portion 1201. When each leaf spring 1212 is deformed to expand outward in the width direction of the groove 1203, it generates a restoring force that returns it to the inside in the width direction due to its elasticity. A widened portion 1214 is formed at the tip of each leaf spring 1212.

[0128] The elastic portion 1213 is made of a folded or zigzag spring and is formed at the innermost portion between adjacent leaf springs 1212. When the elastic portion 1213 is compressed toward the base portion 1211 (i.e., toward the radially inward direction of the fan shape), it generates a restoring force due to its elasticity in a direction away from the base portion 1211 (i.e., toward the radially outward direction of the fan shape).

[0129] Fig. 14 shows the external configuration of the surgical tool 1004. The surgical tool 1004 is used by being attached to the first follower 121 and the second follower 122, and is configured to be held by the holding portion 1003 of the tool turret 124C. The first surgical tool 121A, the second surgical tool 122A, the first replacement surgical tool 124A, and the second replacement surgical tool 124B all have substantially the same configuration as the surgical tool 1004 shown in Fig. 14, at least in the vicinity of the base portion 1401, excluding the end effector portion.

[0130] The surgical tool 1004 has a base portion 1401, a shaft 1402, and an end effector 1403. The base portion 1401 is a portion that is detachably attached to the first robot arm 511 (or the second robot arm 512), has a hollow cylindrical structure, and supports the shaft 1402 at its distal end. The base portion 1401 is formed with a pair of parallel surfaces 1401a that are parallel to each other along the longitudinal direction, and is formed with a pair of protrusions 1401b that protrude from the cylindrical surface on both sides of the parallel surfaces 1401a.

[0131] The shaft 1402 supports an end effector 1403 at its tip. The base portion 1401 receives power for driving the end effector 1403 from the first robot arm 511. The shaft 1402 has a hollow cylindrical structure that is longer and thinner than the base portion 1401, and a power transmission mechanism (not shown) such as a wire is inserted through the shaft 1402 to transmit the driving force of the end effector 1403 from the base portion 1401 to the end effector 1403. The end effector 1403 is made up of various medical treatment tools such as forceps, tweezers, a needle holder, scissors, and an electric scalpel. Patent Document 2 discloses an example of the internal structure of a surgical tool.

[0132] The surgical tool 1004 is held in the holding unit 1002 with the end effector 1403 facing downward and the end face of the base portion 1401 facing upward, as shown in FIG. 14 ( FIG. 10 shows each holding portion 1003 of the tool turret 124C holding a surgical tool with the end face on the base portion 1401 facing upward). A two-dimensional code (e.g., a QR (registered trademark) code) serving as visual identification information for the surgical tool 1004 is provided on the upper end face of the base portion 1401 (see FIGS. 7 and 8 ). Therefore, when the surgical tool 1004 is held in the holding unit 1002, the camera 126A of the detection unit 126 installed above the surgical tool exchange unit 124 can capture an image of the two-dimensional code on the upper end face of the surgical tool 1004. The code reading unit 126B can then read the two-dimensional code from the image captured by the camera 126A.

[0133] 15 shows an enlarged view of the upper end portion of the surgical tool 1004 held by the holding unit 1002. In the example shown in FIG. 15, the surgical tool 1004 is inserted into one of the holding portions 1003 of the holding unit 1002. The width W1 between the parallel surfaces 1401a formed on the base portion 1401 of the surgical tool 1004 is narrower than the width W2 of the groove 1203 formed in the first plate portion 1201. Therefore, it is possible to insert the portion of the surgical tool 1004 where the parallel surfaces 1401a are formed into the holding portion 1003.

[0134] The outer diameter W3 of the pair of protrusions 1401b formed on the base portion 1004 is larger than the width W2 of the groove 1203. Therefore, when the surgical tool 1004 is inserted into the holding portion 1003, the protrusions 1401b are caught on the first plate portion 1201, thereby restricting the vertical movement of the surgical tool 1004 relative to the holding portion 1003. This prevents the surgical tool 1004 from falling off the holding portion 1003.

[0135] The distance W4 between the widened portions 1214 formed at the tips of adjacent leaf springs 1212 (not shown in FIG. 15 ) is narrower than the width W1 between the parallel surfaces 1401 a of the surgical tool 1004. By pushing the surgical tool 1004 so that the parallel surfaces 1401 a abut between the adjacent widened portions 1214, the surgical tool 1004 can be inserted into the holding portion 1003 while widening the gap between the adjacent widened portions 1214. When the surgical tool 1004 passes between the adjacent widened portions 1214 and is inserted into the holding portion 1003, the leaf spring 1212 returns to its original shape. Because the distance W4 between the adjacent widened portions 1214 is narrower than the width W1 between the parallel surfaces 1401 a of the surgical tool 1004, the surgical tool 1004 can be locked between the adjacent leaf springs 1212, preventing the surgical tool 1004 from falling off from the front side of the paper in FIG. 15 .

[0136] 12 and 13 , it can be seen that the surgical tool 1004 inserted into the holding portion 1003 (in other words, inserted between adjacent leaf springs 1212) abuts against the innermost elastic portion 1213. When the surgical tool 1004 is further pushed in, the elastic portion 1213 is compressed toward the base 1211 (i.e., toward the radially inward direction of the fan shape), generating a restoring force. A force acts on the surgical tool 1004, moving it away from the base 1211 (i.e., toward the radially outward direction of the fan shape), and the surgical tool 1004 is sandwiched between the elastic portion 1213 and the widening portion 1214. This determines the position of the surgical tool 1004 in the depth direction of the holding portion 1003 (the depth direction of the paper in FIG. 15 ). The leaf springs 1212 and the elastic portion 1213 position the surgical tool 1004 inserted into the groove 1203. Furthermore, the groove 1203 , the leaf spring 1212 , and the elastic portion 1213 cooperate to realize the function of the holding portion 1003 to hold the surgical tool 1004 .

[0137] 15 shows the surgical tool 1004 properly attached to the holding portion 1003, but in reality, an improper attachment state may occur. For example, an improper attachment state may occur in which the position where the surgical tool 1004 is attached to the holding portion 1003 is shifted in at least one of the vertical and horizontal directions due to vertical positional deviation when the surgical tool 1004 is inserted into one of the holding portions 1003 of the holding unit 1002, a decrease in holding force due to wear of at least one of the widened portion 1214 on the holding unit 1002 side or the convex portion 1401b on the surgical tool 1004 side, or other causes.

[0138] E-2. Surgical Tool Replacement Operation In this section E-2, the surgical tool replacement operation in the operation unit 120 will be explained with reference to Figures 16 to 18. However, for the sake of simplicity of the drawings and ease of explanation, illustrations and explanations of the mirrors installed on the tool turret 124C have been omitted as appropriate. The surgical tool replacement operation is initiated during treatment, for example, in response to instructions from the console unit 110.

[0139] FIG. 16 shows a plan view of the first follower 121, the second follower 122, and the tool turret 124C. However, FIG. 16 also shows the rotational position of the tool turret 124C when a surgical procedure is being performed in the operation unit 120. The first robot arm 511 and the second robot arm 512 are positioned so that the end effectors 1403 of the first surgical tool 121A and the second surgical tool 122A, respectively, are positioned within a surgical field 1601. The tool turret 124C is positioned away from the range of movement of the first robot arm 511 and the second robot arm 512 during a surgical procedure. Therefore, during a surgical procedure, the movement of the first robot arm 511 and the second robot arm 512 is not interfered with by the tool turret 124C. In FIG. 16, the approximate range of movement of the first robot arm 511 during a surgical procedure is indicated by reference numeral 1602.

[0140] Fig. 17 shows a plan view of the first follower 121, the second follower 122, and the tool turret 124C when a surgical tool is replaced by the first follower 121. Fig. 18 shows a perspective view of the first follower 121, the second follower 122, and the tool turret 124C when a surgical tool is replaced by the first follower 121.

[0141] 17 , when the first surgical tool 121A attached to the first robot arm 511 is replaced with the surgical tool 1004 held by the tool turret 124C, the base side drive unit 522 and the tip side drive unit 523 are actuated to move the first robot arm 511 to the opposite side of the surgical field 1601 and bring it closer to the tool turret 124C. In addition, the tool turret 124C is rotated horizontally about the rotation axis 1011A by driving the electric motor 124D, and moved into the range of motion of the first robot arm 511.

[0142] First, the tool turret 124C is rotated horizontally to move the holder 1003 that is not holding a surgical tool (i.e., is empty) to a position where it overlaps with the first mounting portion 501 of the first robot arm 511. At this time, the first mounting portion 501 of the first robot arm 511 is moved to a position where it does not interfere with the tool turret 124C.

[0143] Next, the first robot arm 511 holds the first surgical tool 121A attached to the first mounting portion 501 in the holding portion 1003 (described above) that does not hold a surgical tool, and removes the first surgical tool 121A from the first mounting portion 501.

[0144] Next, the tool turret 124C is further rotated to move the holder 1003 holding the surgical tool 1004 to be replaced to a position where it overlaps with the first mounting portion 501 of the first robot arm 511. Then, the first robot arm 511 mounts the surgical tool 1004 on the first mounting portion 501 and removes it from the tool turret 124C.

[0145] Once the surgical tool has been replaced in the first mounting unit 501 in this manner, the proximal drive unit 522 and distal drive unit 523 are actuated to return the first robot arm 511 to its original position within the surgical field 1601 (see FIG. 16 ), and the treatment is resumed using the newly replaced surgical tool. In addition, the tool turret 124C is rotated horizontally to return it to the position where it was retracted from the range of movement of the first robot arm 511.

[0146] E-3. Summary Finally, the features of the operation unit 120 to which the above-described surgical tool exchange unit 124 is applied will be summarized.

[0147] The tool turret 124C, like the first follower 121 and the second follower 122, is supported by a follower support arm 521 branching off from the distal end of the follower base arm 125. Therefore, the tool turret 124C is disposed in close proximity to the first follower 121 and the second follower 122. In this case, for example, when replacing a surgical tool on the first follower 121 side, the surgical tool can be replaced simply by rotating the tool turret 124C and moving the first robot arm 511 a short distance, thereby shortening the time required for the surgical tool replacement and also shortening the surgery time. Since the range of movement of the first follower 121 and the second follower 122 during surgical tool replacement is reduced, there is also the effect of reducing the risk of interference with surrounding objects such as the patient.

[0148] 16 to 18, when changing surgical tools, the relative positional relationship between the follower support arm 521 supporting the first follower 121 and the second follower 122 and the surgical field 1601 does not change. Therefore, while changing a surgical tool for either the first follower 121 or the second follower 122, the treatment can be continued using the other follower. As a result, the time that the surgery is interrupted for tool changes is shortened, and the surgery time is also shortened.

[0149] 16 to 18, by rotating the tool turret 124C in the horizontal direction using the electric motor 124D and the encoder 124E, the surgical tool to be replaced held by the tool turret 124C can be moved to the surgical tool replacement area of ​​the first follower 121 and the second follower 122. Since surgical tools for multiple followers can be replaced using only one surgical tool replacement unit, it is possible to reduce the size of the device and the manufacturing costs.

[0150] 11, the tool turret 124C is composed of a detachable part 1001 and a holding unit 1002, and the holding unit 1002 is configured to be detachable from the output shaft 1011 of the electric motor 124D at the detachable part 1001. Therefore, replacement surgical tools can be replaced in units of the holding unit 1002 depending on the type and process of surgery.

[0151] Sterilization can be performed by removing the detachable part 1001 and the holding unit 1002. For example, if the detachable part 1001 and the holding unit 1002 are made of polypropylene, polymethylpentene, or polytetrafluoroethylene, sterilization can be performed by autoclave, and in particular, if they are made of polytetrafluoroethylene, polysulfone, or polymethylpentene, dry sterilization can be performed.

[0152] 12 and 13, the holding unit 1002 has a structure that holds each surgical tool 1004 using a simple mechanical structure such as a leaf spring 1212 and an elastic part 1213, and does not include any electrical components, making sterilization easy and reducing manufacturing costs. The detachable part 1001 and holding unit 1002 can be replaced with new ones when they are used up (or their useful life has expired) to ensure cleanliness.

[0153] 10, if a drape 1012 is placed over the operation unit 120 with the detachable attachment / detachment part 1001 and the holding unit 1002 removed, a clean area of ​​the operation unit 120 can be secured. The drape 1012 only needs to have an opening large enough to allow the output shaft 1011 of the electric motor 124D to pass through.

[0154] F. Configuration and Operation of the Surgical Tool Storage System This section G describes the configuration and processing operations of the surgical tool storage system according to the present disclosure. The surgical tool storage system is configured using components of the surgical tool exchange unit 124 and the detection unit 126 of the surgical support robot system 100. In other words, the surgical tool storage system according to the present disclosure can be used as a subsystem incorporated into the surgical support robot system 100.

[0155] F-1. Configuration of the Surgical Tool Storage System Fig. 22 shows an example of the configuration of a surgical tool storage system 2200. This surgical tool storage system 2200 is configured using the surgical tool exchange unit 124 and detection unit 126 in the operation unit 120 of the surgical support robot system 100. Furthermore, the robot control device 130 may be configured to control the operation of the first follower 121 and the second follower 122 (i.e., the drive of the first robot arm 511 and the second robot arm 512) based on the detection results of the detection unit 126.

[0156] As shown in Figures 6 and 7, the surgical tool exchange unit 124 has a fan-shaped, plate-like tool turret 124C that holds five replacement surgical tools at approximately equal intervals along an arc. The tool turret 124C holds the surgical tools facing downward, and a mirror is attached at a 45-degree inclination angle near the holding position of each surgical tool. Furthermore, as shown in Figure 8, each surgical tool has a two-dimensional code on the end face opposite the end effector.

[0157] 6, the camera 126A of the detection unit 126 is disposed almost directly above the tool turret 124C. As shown in Fig. 9, the camera 126A can simultaneously capture images of the upper end surface of the surgical tool and a mirror installed near the surgical tool in its imaging area.

[0158] The image captured by camera 126A is supplied to both code reading unit 126B and position estimation unit 126C. However, half of the captured image that captures the top end surface of the surgical tool may be supplied to code reading unit 126B, and the remaining half that captures the reflected image from the mirror may be supplied to position estimation unit 126C.

[0159] The code reader 126B reads a two-dimensional code (such as a QR (registered trademark) code) from the image of the top surface of the surgical tool included in the image captured by the camera 126A to obtain the identification information of the surgical tool. The two-dimensional code may be an encoded string of characters describing the identification information of the surgical tool, or may be a URI describing the location of the identification information. The identification information may be information indicating the type of surgical tool, or individual identification information that can identify each individual surgical tool. The code reader 126B further estimates the horizontal position of the surgical tool based on the coordinate position of the two-dimensional code in the image captured by the camera 126A.

[0160] Meanwhile, the position estimation unit 126C detects the edge of the top surface of the surgical tool from the mirror's reflected image included in the image captured by the camera 126A, and estimates the vertical displacement of the surgical tool based on the horizontal displacement of the top surface of the surgical tool in the image. As explained with reference to FIG. 9 , when the mirror is attached at a 45-degree tilt angle with respect to the tool turret 124C, the vertical displacement ΔZ of the surgical tool is reflected as being equivalent to the horizontal displacement ΔX on the imaging surface of the camera 126A (i.e., ΔX = ΔZ). As explained with reference to FIGS. 19 and 20 , the position estimation unit 126C can convert the horizontal displacement ΔX obtained from the image captured by the camera 126A into the vertical displacement ΔZ of the surgical tool according to the mirror's tilt angle α.

[0161] The camera 126A is attached to the tip of the follower support arm 521. When the distance between the camera 126A and the tool turret 124C is fixed to be constant, the autofocus function of the camera 126A is not necessary, and the code reading unit 126B and the position estimation unit 126C can perform accurate recognition processing even from an image captured by the monocular camera.

[0162] The surgical tool exchange unit 124 is attached with the tool turret 124C so that the center of the arc coincides with the output shaft 1011 of the electric motor 124D. The surgical tool exchange unit 124 can horizontally rotate and position the tool turret 124C using the electric motor 124D and the encoder 124E. Therefore, the surgical tool exchange unit 124 can horizontally rotate the tool turret 124C in stages so that the five surgical tools held by the tool turret 124C sequentially enter the imaging area of ​​the camera 126A.

[0163] F-2. Detection Operation Fig. 23 shows the procedure of the operation of detecting a surgical tool executed by the detection unit 126 in the form of a flowchart.

[0164] First, a camera calibration process is performed on the image captured by the camera 126A (step S2301). Camera calibration is a process of correcting the image captured by the camera to remove the effects of disturbances such as camera distortion by calculating internal parameters such as the lens focal length, external parameters representing the camera's position and orientation, and lens distortion coefficients. Camera calibration can be performed using a parameter file prepared in advance. The camera calibration process may be performed within the camera 126A or by any unit external to the camera 126A. An open source library may be used for camera calibration. Note that the camera calibration process only needs to be performed once and does not need to be repeated for each detection process. Camera calibration is a well-known technique, so a detailed description will be omitted here.

[0165] Then, the surgical tool changing unit 124 rotates the tool turret 124C horizontally to set the surgical tool to be detected and the corresponding mirror in the imaging area of ​​the camera 126A, and then the camera 126A captures an image of this imaging area (step S2302).

[0166] The code reading unit 126B reads the two-dimensional code from the image of the upper end surface of the surgical tool included in the captured image after the camera calibration process, and obtains the identification information of the surgical tool (step S2311). If the two-dimensional code is a QR (registered trademark) code, the code reading process can be implemented using an open source library.

[0167] Next, the code reader 126B estimates the position information of the surgical tool based on the coordinate position of the two-dimensional code read from the image captured by the camera 126A (step S2312). The position information of the surgical tool here refers to the horizontal position (x-y coordinates) at which the two-dimensional code is detected in the image, but may also include the vertical position (z-coordinate) of the two-dimensional code (or the upper end surface of the surgical tool).

[0168] Then, the code reading unit 126B transmits the identification information of the surgical tool acquired in step S2311 and the position information of the surgical tool acquired in step S2312 to the robot control device 130 (step S2313).

[0169] The detection results by the code reader 126B are also displayed on the screen (step S2314). The screen display process is basically performed on a monitor (not shown) on the detection unit 126 side. The detection results here refer to position information of the surgical tool. For example, the detection results may be superimposed on the image captured by the camera 126A. Furthermore, the identification information of the surgical tool read from the two-dimensional code may also be displayed.

[0170] Furthermore, the position estimation unit 126C detects the edge of the upper end surface of the surgical tool from the reflected image of the mirror included in the captured image after the camera calibration process (step S2321).

[0171] For example, in the two-dimensional code reading process using an open source library performed in step S2311, three-dimensional position information of the two-dimensional code (or the upper end surface of the surgical tool) can be obtained, but the detection accuracy in the vertical direction is low. Therefore, in step S2321, the position estimation unit 126C detects the edges of the upper end surface of the surgical tool with high accuracy from the reflected image of the mirror based on a predetermined edge detection algorithm, and estimates the vertical displacement, thereby substituting or interpolating the low-accuracy vertical position information obtained in the two-dimensional code reading process.

[0172] In step S2321, the position estimation unit 126C uses an edge detection algorithm developed by appropriately combining a series of image processes for the characteristics of the image of the side of the surgical tool to extract with higher accuracy the edge of the upper end surface of the surgical tool from the mirror reflection image included in the captured image. Specifically, the edge detection process in step S2321 is realized by sequentially executing an edge enhancement process (substep S2321-1), an edge feature enhancement process (substep S2321-2), a straight line detection process (substep S2321-3), and an outlier removal process (substep S2321-4).

[0173] In the edge enhancement process in substep S2321-1, vertical edges are enhanced and other edges are diluted for the mirror's reflected image included in the captured image after the camera calibration process. Because the tracking target is only the y-direction edge (the edge at the top of the surgical tool) of the image of the surgical tool on the mirror, the y-direction edge is enhanced and other edge features are diluted as noise. This vertical edge enhancement process can be realized, for example, using a Sobel x filter. The Sobel filter is a filter that extracts contours from an image using first-order derivatives, and the Sobel x filter can enhance vertical edges by calculating the gradient of the image in the x direction.

[0174] In the edge feature enhancement process in substep S2321-2, a different filtering algorithm is applied to the image subjected to vertical edge enhancement in substep S2321-1, thereby performing binarization processing to further enhance the edge features and thin the edges. The edge feature enhancement process facilitates edge detection and makes it easier to identify their positions. This edge feature enhancement process can be implemented using, for example, a Canny filter. The Canny filter, along with the Sobel filter, is known as an edge detection algorithm. The main features of the Canny filter include reduced missed or false detection of contours, detection of a single contour at each point, and true edge detection. The Canny filter roughly performs the following processes in this order: image smoothing using a Gaussian filter, differentiation of the smoothed image, calculation of the gradient magnitude and direction from the differentiation result, non-maximum suppression (NMS) processing, and hysteresis threshold processing.

[0175] After substep S2321-2 is completed, the edges are still separate. Therefore, in substep S2321-3, the line detection process is performed on the image whose edge features have been further enhanced in substep S2321-2, excluding contours that deviate from the edges and integrating the separate edges into continuous lines. This line detection process can be realized, for example, using Hough transform processing. The Hough transform is an algorithm that effectively detects the contours of parametric basic shapes (lines, circles, etc.) in an image by replacing them with a voting problem in parameter space.

[0176] In the outlier removal process in sub-step S2321-4, the coordinates of points on the straight line detected in sub-step S2321-3 are detected, and points that deviate from the straight line, ie, noise, are removed using a numerical filtering algorithm.

[0177] At least some of the sub-steps S2321-1, . . . , S2321-4 included in step S2321 can be implemented using an open source library.

[0178] Next, the position estimation unit 126C calculates the average value of the x-coordinates of each point remaining after substep S2321-4 to calculate the horizontal displacement ΔX of the edge corresponding to the upper end surface of the surgical tool, and further converts this into a vertical displacement ΔZ of the surgical tool according to the tilt angle α of the mirror (step S2322). When the mirror is attached at a tilt angle of 45 degrees to the tool turret 124C, the vertical displacement ΔZ of the surgical tool is equivalent to the horizontal displacement ΔX on the imaging plane of the camera 126A.

[0179] Then, the position estimation unit 126C transmits the vertical displacement ΔZ of the surgical tool calculated in step S2322 to the robot control device 130 (step S2323).

[0180] The result of the surgical tool position estimation by the position estimation unit 126C is displayed on the screen (step S2324). The screen display process is basically performed on a monitor (not shown) on the detection unit 126 side. For example, the estimated edge of the upper end surface may be superimposed on the image captured by the camera 126A. The identification information of the surgical tool read by the code reading unit 126B may also be displayed on the screen.

[0181] Until the termination condition for the surgical tool detection process is met (No in step S2303), the process returns to step S2302, and the detection unit 126 repeatedly executes the surgical tool detection process. On the other hand, when the termination condition for the surgical tool detection process is met (Yes in step S2303), the entire process ends. The termination condition is arbitrary. For example, the termination condition for the surgical tool detection process may be the end of surgery by the surgical support robot system 100, or an end instruction from the user via the console unit 110.

[0182] 23, the code reading unit 126B transmits the code reading result to the robot control device 130 in step S2313, while the position estimation unit 126C transmits the position estimation result to the robot control device 130 in step S2323. As shown in FIG. 23, the code reading unit 126B and the position estimation unit 126C may each perform the transmission process individually, or the processing results of the code reading unit 126B and the position estimation unit 126C may be transmitted together to the robot control device 130.

[0183] 23 is performed on the holder of the tool turret 124C to which the surgical tool removed from the follower is returned, for example, when the surgical tool of the first follower 121 or the second follower 122 is replaced. Of course, the surgical tool detection process may be performed at any timing on any holder on the tool turret 124C. In accordance with instructions from the console unit 110 (or the user) or the robot control device 130, the surgical tool detection process may be performed at any timing on a designated holder on the tool turret 124C.

[0184] F-3. Surgical Tool Detection Using Scanning Method In the example shown in FIG. 7, the imaging area of ​​camera 126A is set to accommodate one surgical tool (i.e., only the top surface of the surgical tool and the nearby mirror are within the area). Therefore, detection unit 126 can only detect one surgical tool per detection operation. Therefore, the positioning operation of tool turret 124C by surgical tool exchange unit 124 and the detection operation of detection unit 126 may be linked to detect all surgical tools held by tool turret 124C using a scanning method.

[0185] Figure 24 shows the operating procedure of the tool turret 124C when performing surgical tool detection using a scanning method in the surgical tool storage system 2200. Figure 24 shows the tool turret 124C viewed from above in each of steps (a) to (d), and it can be seen that the tool turret 124C rotates stepwise for each step. Figure 24 also shows the imaging area of ​​the camera 126A in each of steps (a) to (d). Because the camera 126A is fixed to the tip of the follower support arm 521, the position of the imaging area does not move even during scanning. The tool turret 124C has five holding sections for holding surgical tools, and turret numbers 1 to 5 are assigned in order from right to left on the page.

[0186] When starting surgical tool detection by the scanning method, first, as shown in FIG. 24(a), the tool turret 124C is set to a position where it has been rotated counterclockwise to the maximum extent possible on the paper.

[0187] 24B, the tool turret 124C is rotated clockwise on the page, and the holders of the tool turret 124C are aligned with the imaging area of ​​the camera 126A in the order of No. 1 to No. 5. The surgical tool exchange unit 124 may store in advance the rotational positions at which each of the holders No. 1 to No. 5 is aligned with the imaging area, and control the rotational position of the tool turret 124C, or the rotational position may be controlled based on the recognition result of the image captured by the camera 126A so that the holders enter the imaging area, or the rotational position of the tool turret 124C may be controlled using both the pre-stored rotational position information and image recognition.

[0188] 23 every time each holding part of the tool turret 124C is aligned with the imaging area of ​​the camera 126A, the detection unit 126 activates the surgical tool detection operation shown in Fig. 23, performs a two-dimensional code reading process by the code reading unit 126B for the surgical tool held in the corresponding holding part, and performs a process of estimating the holding position of the surgical tool by the position estimation unit 126C, and transmits the code reading results and position estimation results to the robot control device 130. Note that the detection unit 126 may perform a transmission process to the robot control device 130 every time it completes the surgical tool detection operation for one holding part, or may perform a transmission process to the robot control device 130 by collecting the detection results for all holding parts after completing the scanning operation.

[0189] When the surgical tool detection process has been performed in this manner for all of the holders No. 1 to No. 5, the tool turret 124C reaches a position where it has rotated to the maximum extent clockwise as shown in Fig. 24(c) . The completion of the scanning operation may be recognized when the surgical tool exchange unit 124 reaches the maximum rotation position of the tool turret 124C clockwise as shown in Fig. 24(c) , or the completion of the scanning operation may be recognized when it is recognized from the image captured by the camera 126A that the fifth (or last) holder has been reached.

[0190] Then, when the scanning surgical tool detection is completed, the surgical tool exchange unit 124 returns the tool turret 124C to the default position as shown in Figure 24(d). The default position is, for example, a position where the third (or central) holding portion of the tool turret 124C is aligned with the imaging area of ​​the camera 126A. Of course, the default position of the tool turret 124C may be a rotational position other than the center.

[0191] During surgery (i.e., while the operation unit 120 is operating), the detection unit 126 can constantly monitor the attachment status of surgical tools passing within the imaging area (or field of view) of the camera 126A (see, for example, Figure 6).

[0192] The operation unit 120 may be configured to perform the above-described scanning tool detection operation periodically or irregularly. For example, the scanning tool detection operation may be initiated whenever necessary, such as when the tool of the first follower 121 or the second follower 122 is replaced, or when the entire tool turret 124C is replaced. Furthermore, the operation unit 120 may be configured to initiate the scanning tool detection operation in accordance with an instruction from the console unit 110 (or a user) or the robot control device 130.

[0193] F-4. Processing on the Robot Control Device Side As described above, the surgical tool storage system 2200 transmits the code reading results and position estimation results obtained by the surgical tool detection operation shown in Fig. 23 to the robot control device 130. Based on the information supplied from the surgical tool storage system 2200, the robot control device 130 can manage each surgical tool attached to the tool turret 124C and inspect the attachment status.

[0194] 25 shows in the form of a flowchart the processing procedure that the robot control device 130 performs based on information supplied from the surgical tool storage system 2200. This processing is realized, for example, by the CPU 131 of the robot control device 130 executing a predetermined surgical tool management application (hereinafter referred to as the "surgical tool management app").

[0195] When information arrives from the surgical tool storage system 2200 (Yes in step S2501), the detection unit 126 performs a receiving process (step S2502). For simplicity's sake, it is assumed that the detection unit 126 receives both the result of reading the two-dimensional code by the code reader 126B and the position information of the surgical tool. The position information of the surgical tool includes the horizontal position (x- and y-coordinates) at which the code reader 126B detected the two-dimensional code on the top surface of the surgical tool, and the vertical position (z-coordinate) of the top surface of the surgical tool as determined by the position estimation unit 126C.

[0196] The surgical tool management app matches the surgical tool identification information identified from the received code reading result with the identification information of the surgical tool that should be stored in the tool turret 124C in the management information (step S2503). If the surgical tool identification information actually read from the tool turret 124C does not match the surgical tool identification information in the management information (No in step S2504), the surgical tool management app cancels the surgical tool replacement operation (step S2507) and terminates this process. In addition, the surgical tool management app may issue an alert via a screen display or audio guidance in step S2507 to prompt the user to check the surgical tool on the tool turret 124C. This is because there is a possibility that the wrong surgical tool may be attached to the first follower 121 or the second follower 122 due to a surgical tool mix-up, or that the wrong surgical tool may be attached to the first follower 121 or the second follower 122 during a subsequent surgical tool replacement.

[0197] The surgical tool management app may use, for example, a surgical tool management table as shown in Fig. 26 to manage each surgical tool currently in use in the surgical support robot system 100. The surgical tools currently in use in the surgical support robot system 100 are the surgical tools attached to the first follower 121 or the second follower 122 and the five surgical tools stored in the tool turret 124C. In the example shown in Fig. 26, the surgical tool management table provides a record (row) for each surgical tool installation location (turret number), and each record includes the "surgical tool identification information," "surgical tool type," "number of uses," "usage period," etc., for the corresponding surgical tool.

[0198] The surgical tool management app rewrites the surgical tool management table (e.g., swaps the values ​​of each field between the row of the first follower 121 or second follower 122 and the row of the replaced turret number) each time a surgical tool is replaced on the operation unit 120 side. At that time, it updates the values ​​of the "number of uses," "usage period," etc. fields. The surgical tool management app can also match the code read results received from the detection unit 126 with the surgical tool identification information stored in the row of the corresponding turret number in the surgical tool management table. The surgical tool management app can also manage the useful life of the corresponding surgical tool based on the "number of uses" and "usage period" values ​​for each row in the surgical tool management table. When the surgical tool management app detects a surgical tool that has reached its useful life, it may issue an alert via a screen display or audio guidance, or prompt the user to replace the surgical tool.

[0199] 26 is an example of the configuration of the surgical tool management table, and is not limited to this. The surgical tool management app may use a surgical tool management table having a different data structure to manage the surgical tools being used in the surgical support robot system 100, or may manage the surgical tools without using any surgical tool management table.

[0200] Referring again to FIG. 25, the processing performed by the robot control device 130 will be further described.

[0201] If the surgical tool management application confirms that the surgical tool identification information actually read from the tool turret 124C matches the surgical tool identification information in the management information (Yes in step S2504), it checks the attachment status of the surgical tool to the tool turret 124C based on the received surgical tool position information (x, y, z coordinates) (step S2505). In this embodiment, it should be fully understood that the detection unit 126 captures the side of the surgical tool using the reflected image of the mirror, and therefore, the position estimation error of the surgical tool in the vertical direction in particular can be significantly improved.

[0202] For example, the x, y, and z coordinates of a surgical tool that is properly attached to the tool turret 124C are acquired in advance and set as reference x, y, and z coordinate values. The surgical tool management app then compares the x, y, and z coordinate values ​​of the surgical tool received (in other words, detected) from the detection unit 126 to determine whether the surgical tool is properly attached. If the detected x, y, and z coordinate values ​​are close to the reference x, y, and z coordinate values, the surgical tool management app determines that the attachment state is good (Yes in step S2506). In this case, the surgical tool management app returns to step S2501 and waits until information is again transmitted from the detection unit 126 due to, for example, the next surgical tool change. Alternatively, the surgical tool management app may normally terminate this process instead of returning to step S2501.

[0203] On the other hand, if the detected x, y, and z coordinate values ​​deviate significantly from the reference x, y, and z coordinate values, or if the z coordinate value deviates significantly from the reference z coordinate value and it is determined that the surgical tool is floating above the tool turret 124C (No in step S2506), the surgical tool management app stops the surgical tool replacement operation (step S2507) and ends this processing. Also, in step S2507, the surgical tool management app may issue an alert by displaying on the screen or by audio guidance to prompt the user to check the attachment status of the surgical tool on the tool turret 124C.

[0204] F-5. Summary Finally, the features of the surgical tool storage system 2200 applied to the surgery support robot system 100 will be summarized.

[0205] The surgical tool storage system 2200 is configured to project visual information of the side of the surgical tool held by the tool turret 124C into the field of view of the camera 126A using a mirror positioned in the imaging area of ​​the camera 126A. Therefore, the surgical tool storage system 2200 can read the code on the upper end surface of the surgical tool and estimate the three-dimensional position of the surgical tool from an image captured by a monocular camera simultaneously using two viewing angles (upward and horizontal). The surgical tool storage system 2200 is low-cost and simple in structure, and can accurately estimate the vertical position of the surgical tool and detect whether it is floating from the tool turret 124C. Furthermore, the surgical tool storage system 2200 can avoid the problem of focus shift when the surgical tool is displaced vertically.

[0206] The surgical tool storage system 2200 is suitably incorporated into the surgical support robot system 100, and can smoothly read the codes of the surgical tools held in the tool turret 124C and detect improper attachment states without disrupting the flow of surgical tool replacement in the surgical tool replacement unit 124. Therefore, the surgical tool storage system 2200 can meet the needs for surgical tool management and ensuring safety when replacing surgical tools.

[0207] G. Configuration of the Console Unit As explained in Section B-1 above, the console unit 110 includes a first reader 111, a second reader 112, a foot switch unit 113, and a display device 114, and the operator operates the console unit 110 to instruct the surgery support robot system 100 to perform certain operations. In this Section G, the structure of the console unit 110 will be explained in detail.

[0208] 27 shows the external configuration of the console unit 110. The console unit 110 is a structure that is approximately L-shaped when viewed from the side, and has a bottom portion 2701 at its lowest end that is U-shaped when viewed from above, with a base portion 2702 connected to the center of the bottom portion 2701 in an approximately vertical direction.

[0209] An O-shaped or ring-shaped support portion 2703 is attached near the middle of the base portion 2702. The support portion 2703 is at approximately the same height as the elbows of an operator sitting on a chair 2704. However, the height of the chair 2704 may be adjusted so that the support portion 2703 is at approximately the same height as the elbows of the operator.

[0210] A display device 114 is attached to the upper end of the base unit 2702. In the example shown in Fig. 27 , the display device 114 is an immersive 2D and 3D viewer into which the operator peers. Therefore, the operator can operate the first reader 111 and the second reader 112 with their left and right hands, and the foot switch unit 113 with their feet, while peering into the screen of the display device 114 and observing a 2D or 3D image of the surgical field. Although the foot switch 113 is not shown in Fig. 27 , it should be understood that the foot switch 113 is installed, for example, inside the U-shaped base unit 2702.

[0211] 28 shows the support unit 2703 as seen from the side of an operator sitting on a chair 2704. A first reader 111 and a second reader 112 are arranged near the base of the support unit 2703. The first reader 111 and the second reader 112 are readers that the operator operates using the left and right hands, respectively, and have the same structure and are symmetrical. The operator can use the front edge of the ring-shaped support unit 2703 as a hand rest or wrist rest 2705, and can rest both arms or elbows on the support unit 2703 and use the left and right hands to operate the first operating unit 111C at the tip of the first reader 111 and the second operating unit 112C at the tip of the second reader 112, respectively.

[0212] 28 , the first reader 111 and the second reader 112 are each configured as an articulated robot arm, and support a first operating unit 111C corresponding to right-handed operation and a second operating unit 112C corresponding to left-handed operation at the distal end of each robot arm. However, since the technical scope of the present disclosure is not limited to the structure of a specific robot arm, detailed description of the robot arm will be omitted in this specification. Also, in this embodiment, for the sake of simplicity, the first reader 111 and the second reader 112 have the same configuration, but of course, each reader may have a different configuration.

[0213] The first reader 111 and the second reader 112 support, at their distal ends, a first operating unit 111C for right-handed operation and a second operating unit 112C for left-handed operation. As shown in Fig. 2, the first reader 111 and the second reader 112 are equipped with sensor units 111B and 112B, respectively, that detect the state of the robot arm. The first reader 111 and the second reader 112 are configured to compensate for their own weight through mechanical or motor control, so that they will not fall even if the operator lets go of them.

[0214] There is interlocking between the first leader 111 and the first follower 121, and between the second leader 112 and the second follower 122. That is, leader-follower control is possible, and in accordance with the operation state of the first leader 111 and the second leader 112 by the operator on the console unit 110, the electric motors 121B and 122B are driven on the operation unit 120 side, and the positions and attitudes of the first surgical tool 121A and the second surgical tool 122A of the first follower 121 and the second follower 122, respectively, are controlled.

[0215] The electric motors 111A and 112A are driven by control signals from the robot control device 130 so as to perform a target operation based on the position detection results converted into electric signals. However, in this embodiment, the robot arms serving as the first reader 111 and the second reader 112 may include passive joints, and may be of a vertically multi-joint type or a horizontally multi-joint type.

[0216] The first operation unit 111C and the second operation unit 112C are, for example, composed of a pair of grips rotatably supported by a grip shaft and operable to open and close, and can open and close the grippers by driving the electric motors 111A and 112A, respectively. Therefore, the first operation unit 111C and the second operation unit 112C can present haptic sensations to the operator's right and left hands by driving the electric motors 111A and 112A based on control signals input from the robot control device 130. Presenting haptic sensations here means transmitting force information to the operator's right and left hands so that the operator can approximate the forces acting on the first follower 121 and the second follower 122, respectively, and the results of their operation inputs.

[0217] The foot switch unit 113 is configured so that the operator can use their feet to operate the operation unit 120 and activate the functions of the operation unit 120. As shown in FIG. 2, the foot switch unit 113 includes a first surgical tool change button 113A, a second surgical tool change button 113B, and a clutch button 113C. While the presence of the foot switch unit 113 is difficult to see in FIG. 27, it should be understood that it is located inside the U-shape of the U-shaped bottom portion 2701. Each button 113A to 113C is located on the top surface of the foot switch unit 113, so that an operator sitting in a chair 2704 can press the buttons with their feet. FIG. 27 shows the external configuration of the top surface of the foot switch unit 113 so that the arrangement of each button 113A to 113C can be seen.

[0218] The first surgical tool change button 113A is operated to execute the surgical tool change function of the first follower 121. Specifically, by operating the first surgical tool change button 113A, the link between the first leader 111 and the first follower 121 is released, and the operation of the first follower 121 is switched to an operation for executing the surgical tool change function. Similarly, the second surgical tool change button 113B is operated to execute the surgical tool change function of the second follower 122, and the link between the second leader 112 and the second follower 122 is released, and the operation of the second follower 122 is switched to an operation for executing the surgical tool change (described above).

[0219] The clutch button 113C is operated to execute the clutch function. Specifically, the clutch button 113C is used to collectively interrupt the interlocking between the first leader 111 and the first follower 121 and the interlocking between the second leader 112 and the second follower 122, and to stop the operation of the first surgical tool 121A and the second surgical tool 122A on the operation unit 120 side. While the clutch button 113C is being operated, the first follower 121 and the second follower 122 will not operate even if the first leader 111 and the second reader 112 are operated.

[0220] For example, when the operator operates the first leader 111 and the second leader 112 and either the first leader 111 or the second leader 112 reaches the vicinity of the end of the movable range, the operator can temporarily disconnect the interlock between the first leader 111 and the first follower 121, and between the second leader 112 and the second follower 122, by operating the clutch button 113C, and can return the first leader 111 and the second leader 112 to the vicinity of the center of the movable range. Furthermore, when the operator stops operating the clutch button 113C, the leader-follower control is resumed, and the operator can resume operating the first operation unit 111C of the first leader 111 and the second operation unit 112C of the second reader 112 near the center of the movable range.

[0221] The configuration of the foot switch unit 113 is not limited to the example shown in FIG. 2 , and may further include other switches, and the switch may be in the form of a button or another mechanical operator such as a joystick. For example, the foot switch unit 113 may further include a switch for switching on and off the supply of high-frequency current to the surgical tool, a camera switch for switching the operation of the camera arm 123C, and the like. For example, when the camera switch is operated, the camera arm 123C may be driven by a follower operation by the operator, so that at least one of the position and posture of the camera 123A can be changed. Furthermore, the first surgical tool exchange button 113A and the second surgical tool exchange button 113B may be located on the first reader 111 and the second reader 112, respectively, rather than on the foot switch unit 113.

[0222] The present disclosure has been described in detail above with reference to specific embodiments. However, the present disclosure should not be construed as being limited to the above-described embodiments, and it is obvious that those skilled in the art can modify or substitute the embodiments without departing from the spirit of the present disclosure. Furthermore, the effects described in this specification are merely examples, and the effects brought about by the present disclosure are not limited thereto, and additional effects not described in this specification may exist.

[0223] This specification has mainly described an embodiment in which the present disclosure is applied to a surgical assistance robot system, but the surgical assistance robot system can be applied to various types of surgery, such as laparoscopic surgery, neurosurgery, and ophthalmic surgery.

[0224] Furthermore, the gist of the present disclosure is not limited to medical technology. For example, the present disclosure can be similarly applied to various types of work support robot systems that remotely control robot arms in a leader-follower manner to assist in difficult work in various industrial fields, such as high altitudes, construction sites, factories where processing or assembly work is performed, mining sites, high altitudes such as in high-rise buildings, underwater or on the seabed, nuclear plants, and outer space, and can appropriately manage multiple tools used by the robot and perform automatic tool change.

[0225] In short, the present disclosure has been described in the form of examples, and the contents of the specification should not be interpreted as limiting. To determine the gist of the present disclosure, the claims should be taken into consideration.

[0226] The present disclosure may also be configured as follows.

[0227] (1) A surgical tool storage system comprising: a holding unit that holds a surgical tool; and a detection unit that detects the surgical tool held in the holding unit, wherein the surgical tool has visual identification information, the holding unit holds the surgical tool so that the visual identification information is included in the detection range of the detection unit, and the detection unit has a camera that captures an image of the surgical tool held in the holding unit and a reading unit that reads the visual identification information based on the image captured by the camera.

[0228] (1-1) The surgical tool storage system according to (1) above, wherein the camera is a monocular camera.

[0229] (1-2) The surgical tool storage system according to (1) above, wherein the camera is fixed at a constant distance from the holder.

[0230] (1-3) The surgical tool storage system according to (1) above, wherein the camera does not have an autofocus function.

[0231] (2) A surgical tool storage system as described in (1) above, wherein the surgical tool has an end effector at its tip and visual identification information is provided on the end face opposite the end effector, the holding unit holds the surgical tool with the end effector facing downward and the end face with the visual identification information facing upward, and the reading unit reads the visual identification information from the end face of the surgical tool reflected in the image captured by the camera.

[0232] (3) The surgical tool storage system described in (2) above, wherein the detection unit further estimates the horizontal displacement of the surgical tool relative to the holding unit based on visual identification information read by the reading unit from the end face of the surgical tool captured in the image captured by the camera.

[0233] (4) The surgical tool storage system described in any one of (2) or (3) above, wherein the detection unit further includes a position estimation unit that estimates the vertical position of the surgical tool held in the holding unit from an image captured by the camera.

[0234] (5) A surgical tool storage system as described in (4) above, wherein the holding unit further includes a mirror installed near the position where the surgical tool is held and directs light from the side of the surgical tool toward the camera, the camera is arranged so that the end face of the surgical tool and the mirror are included in the imaging area, and the position estimation unit estimates the vertical position where the holding unit holds the surgical tool based on the result of edge detection of the upper end face of the surgical tool from the image of the side of the surgical tool captured by the camera via the mirror.

[0235] (6) The surgical tool storage system described in (5) above, wherein the position estimation unit converts the horizontal displacement of the upper end surface of the surgical tool, whose edge is detected from the image captured by the camera, into the vertical displacement of the surgical tool relative to the holding unit, depending on the tilt angle at which the mirror is attached.

[0236] (7) The position estimation unit performs the following processes in sequence on the image captured by the camera: edge enhancement processing to enhance the edge of the top end of the surgical tool in the y direction of the image of the surgical tool on the mirror; edge feature enhancement processing to binarize the image and further enhance the edge features to thin the edges; line detection processing to remove contours that deviate from the edges by line detection and combine them into continuous lines; and outlier removal processing to detect the coordinates of points on the line and remove points that deviate from the line as noise, thereby detecting the edge of the top end surface of the surgical tool from the image captured by the camera. A surgical tool storage system described in any one of (5) or (6) above.

[0237] (7-1) The surgical tool storage system according to (7), wherein the position estimation unit performs the edge enhancement process using a Sobel x filter.

[0238] (7-2) The surgical tool storage system according to (7), wherein the position estimation unit performs the edge enhancement process using a Canny filter.

[0239] (7-3) The surgical tool storage system according to (7), wherein the position estimation unit performs the line detection process using a Hough transform.

[0240] (7-4) The surgical tool storage system according to (7), wherein the position estimation unit performs the edge enhancement process using a numerical filter.

[0241] (8) The surgical tool storage system according to any one of (5) to (7), wherein the state of attachment of the surgical tool by the holder is judged based on the estimation result by the position estimation unit.

[0242] (9) A surgical instrument storage system described in any one of (5) to (8) above, further comprising a turret having a plurality of holding portions formed thereon, wherein the mirror is positioned near each of the plurality of holding portions formed on the turret, or is positioned on the outer periphery of the turret in a location included in the imaging area of ​​the camera.

[0243] (10) A surgical support robot system comprising an operation unit including one or more followers to which surgical tools are interchangeably attached, and a console unit including one or more readers operated by an operator, wherein the operation unit includes a surgical tool exchange unit that holds replacement surgical tools and a detection unit that detects the surgical tools held in the surgical tool exchange unit, the surgical tools having visual identification information, the surgical tool exchange unit holds the surgical tools so that the visual identification information is included in the detection range of the detection unit, and the detection unit includes a camera that captures an image of the surgical tool held in the surgical tool exchange unit and a reading unit that reads the visual identification information based on the image captured by the camera.

[0244] (11) The surgical support robot system according to (10), wherein the detection unit further includes a position estimation unit that estimates the vertical position of the surgical tool held in the surgical tool exchange unit from an image captured by the camera.

[0245] (12) The surgical support robot system described in (11) above, wherein the surgical tool exchange unit further includes a mirror installed near the position where the surgical tool is held and directs light from the side of the surgical tool toward the camera, the camera is arranged so that the end face of the surgical tool and the mirror are included in the imaging area, and the position estimation unit estimates the vertical position where the holding unit holds the surgical tool based on the result of edge detection of the upper end face of the surgical tool from the image of the side of the surgical tool captured by the camera via the mirror.

[0246] DESCRIPTION OF SYMBOLS 100...Surgery support robot system, 110...Console unit 111...First reader, 111A...Electric motor, 111B...Sensor unit 111C...First operation unit, 112...Second reader, 112A...Electric motor 112B...Sensor unit, 112C...Second operation unit 113...Foot switch unit, 113A...First surgical tool exchange button 113B...Second surgical tool exchange button, 113C...Clutch button 114...Display device, 120...Operating unit 121...First follower, 121A...First surgical tool, 121B...Electric motor 121C...Sensor unit, 122...Second follower, 122A...Second surgical tool 122B...Electric motor, 122C...Sensor unit, 123...Imaging unit 123A...Camera, 123B...Light source, 123C...Camera arm 124... surgical tool replacement unit, 124A... first replacement surgical tool 124A-1... mirror, 124B... second replacement surgical tool 124B-1... mirror, 124C... tool turret 124D... electric motor, 124E... encoder 125... follower base arm, 125A... electric motor, 125B... sensor section 126... detection unit, 126A... camera, 126B... code reading section 126C... position estimation section 130... robot control device, 131... CPU, 132... memory section 133... bus, 134... input / output section 140... external operation terminal, 141... input section, 142... display section 150... operating table, 160... external network 310... calculation processing section, 311... input / output processing section, 312... drive control section 313... image generation section, 314... display control section 320... input / output interface DESCRIPTION OF SYMBOLS 400...housing, 401...top surface, 402...legs, 403...wheels 411...support section, 412...first link, 413...second link 421...first joint section, 422...second joint section, 423...third joint section 501...first attachment section, 502...second attachment section 511...first robot arm, 512...second robot arm 521...follower support arm, 522...base side drive section 523...tip side drive section 601-605...surgical tool (replacement), 611-615...mirror 1001...detachable section, 1001a...coupling section, 1001b...second coupling section 1002...holding unit, 1003...holding section, 1004...surgical tool 1011...output shaft (of electric motor 124D), 1012...drapeDESCRIPTION OF SYMBOLS 1201...first plate portion, 1202...second plate portion, 1203...groove, 1211...base portion, 1212...plate spring, 1213...elastic portion, 1214...widening portion, 1401...root portion, 1401a...parallel surface, 1401b...convex portion, 1402...shaft, 1403...end effector, 2101...base, 2200...surgical tool storage system, 2701...bottom portion, 2702...base portion, 2703...support portion

Claims

1. A surgical tool storage system comprising: a holding section for holding surgical tools; and a detection section for detecting the surgical tools held in the holding section; wherein the surgical tools have visual identification information, the holding section holds the surgical tools so that the visual identification information is within the detection range of the detection section, and the detection section comprises a camera for capturing an image of the surgical tools held in the holding section, and a reading section for reading the visual identification information based on the image captured by the camera.

2. A surgical tool storage system as described in claim 1, wherein the surgical tool has an end effector at its tip and visual identification information is provided on the end face opposite the end effector, the holding unit holds the surgical tool with the end effector facing downwards and the end face with the visual identification information facing upwards, and the reading unit reads the visual identification information from the end face of the surgical tool captured in the image captured by the camera.

3. The surgical tool storage system of claim 2, wherein the detection unit further estimates the horizontal displacement of the surgical tool relative to the holding unit based on visual identification information read by the reading unit from the end face of the surgical tool captured in the image captured by the camera.

4. A surgical tool storage system as described in claim 2, wherein the detection unit further comprises a position estimation unit that estimates the vertical position of the surgical tool held in the holding unit from an image captured by the camera.

5. A surgical tool storage system as described in claim 4, wherein the holding unit further comprises a mirror installed near the position where the surgical tool is held and directing light from the side of the surgical tool toward the camera, the camera is arranged so that the end face of the surgical tool and the mirror are included in the imaging area, and the position estimation unit estimates the vertical position at which the holding unit holds the surgical tool based on the result of edge detection of the top end face of the surgical tool from the image of the side of the surgical tool captured by the camera via the mirror.

6. A surgical tool storage system as described in claim 5, wherein the position estimation unit converts the horizontal displacement of the upper end surface of the surgical tool, edge-detected from the image captured by the camera, into a vertical displacement of the surgical tool relative to the holding unit, depending on the inclination angle at which the mirror is attached.

7. The surgical tool storage system of claim 5, wherein the position estimation unit performs the following processes in sequence on the image captured by the camera: edge enhancement processing for enhancing the edge of the top end of the surgical tool in the y direction of the image of the surgical tool on the mirror; edge feature enhancement processing for binarizing the image to further enhance the edge features and thin the edges; line detection processing for removing contours that deviate from the edges by line detection and combining them into a continuous line; and outlier removal processing for detecting the coordinates of points on the line and removing points that deviate from the line as noise, thereby detecting the edges of the top surface of the surgical tool from the image captured by the camera.

8. A surgical tool storage system as described in claim 5, further comprising: a determination as to whether the state of attachment of the surgical tool by said holder is good or bad based on the result of estimation by said position estimation unit.

9. The surgical tool storage system of claim 5, further comprising a turret on which a plurality of the holding portions are formed, and the mirror is positioned near each of the plurality of holding portions formed on the turret, or is positioned on the outer periphery of the turret at a location included in the imaging area of ​​the camera.

10. A surgical support robot system comprising an operation unit including one or more followers on which surgical tools are interchangeably attached, and a console unit including one or more readers operated by an operator, wherein the operation unit includes a surgical tool exchange unit that holds replacement surgical tools and a detection unit that detects the surgical tools held in the surgical tool exchange unit, the surgical tools having visual identification information, the surgical tool exchange unit holds the surgical tools such that the visual identification information is within the detection range of the detection unit, and the detection unit comprises a camera that captures an image of the surgical tools held in the surgical tool exchange unit, and a reading unit that reads the visual identification information based on the image captured by the camera.

11. The surgical support robot system according to claim 10, wherein the detection unit further comprises a position estimation unit that estimates the vertical position of the surgical tool held in the surgical tool exchange unit from an image captured by the camera.

12. The surgical support robot system of claim 11, wherein the surgical tool exchange unit further comprises a mirror installed near the position where the surgical tool is held and directing light from the side of the surgical tool to the camera, the camera is disposed so as to include an end face of the surgical tool and the mirror in its imaging area, and the position estimation unit estimates the vertical position at which the holding unit holds the surgical tool based on the result of edge detection of the top end face of the surgical tool from the image of the side of the surgical tool captured by the camera via the mirror.

Images