Surgery system, display method, and program

Abstract

To provide a surgery system that enables an operator to easily view a sign indicating a surgical instrument located outside a visual field without largely moving a line-of-sight.SOLUTION: In a surgical system 100 (surgery system), when at least one surgical instrument among a plurality of surgical instruments is located outside a visual field of an endoscope 6, an image processing device 8b displays a sign MK1 indicating the surgical instrument located outside the visual field in an inside region of a graphical user interface G not including an edge vicinity of a screen of a display part in response to a predetermined input through a camera pedal 22c.SELECTED DRAWING: Figure 13

Description

Technical Field

[0001] This invention relates to a surgical system, a display method, and a program, and particularly to a surgical system, a display method, and a program for superimposing a graphical user interface on an image captured by an endoscope and displaying it on a display device.

Background Art

[0002] Conventionally, a robotic surgical system that superimposes a graphical user interface on an image captured by an endoscope is known (for example, see Patent Document 1).

[0003] In the robotic surgical system of Patent Document 1, the endoscope captures an image of a surgical site for display in the display area of a monitor. Also, in this robotic surgical system, the current position of a tool with respect to the image (field of view) captured by the endoscope is determined. And when the determined current position of the tool is outside the image (field of view) displayed on the monitor, the current position of the tool is displayed as a symbol in a boundary area outside the display area of the monitor screen.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in robotic surgical systems like the one described in Patent Document 1, the operator focuses their gaze on the area near the center of the monitor's display area during surgery. Therefore, even if the current position of a tool (surgical instrument) outside the field of view is displayed as a symbol (marker) in the boundary area outside the display area of ​​the monitor screen, as in Patent Document 1, there is a problem that the operator may not notice it. Furthermore, if the operator does notice it, there is another problem that they need to move their gaze significantly.

[0006] This invention was made to solve the above-mentioned problems, and one of its objectives is to provide a surgical system and display method that allows the operator to easily see markers indicating surgical instruments located outside the field of view without having to move their eyes significantly. [Means for solving the problem]

[0007] A surgical system according to the first aspect of this invention comprises an endoscope, a plurality of manipulators for supporting a plurality of surgical instruments, a first display device for displaying endoscopic images captured by the endoscope, a remote control device including two operating handles for operating two of the plurality of surgical instruments, a control device configured to display a graphical user interface superimposed on the endoscopic image captured by the endoscope on the display unit of the first display device, and an input device for the operator to input predetermined information. When at least one of the plurality of surgical instruments is located outside the field of view of the endoscope, the control device is configured to display a marker indicating the surgical instrument located outside the field of view in the inner area of ​​the graphical user interface, excluding the vicinity of the edge of the display unit's screen, in response to a predetermined input from the input device.

[0008] In the surgical system according to the first aspect of this invention, as described above, when at least one of the multiple surgical instruments is located outside the field of view of the endoscope, the control device, in response to a predetermined input from the input device, displays a marker indicating the surgical instrument located outside the field of view in the inner area of ​​the graphical user interface, excluding the vicinity of the edge of the display screen. As a result, the marker indicating the surgical instrument located outside the field of view is displayed relatively towards the center of the display screen rather than near the edges. Consequently, the operator can easily see the marker indicating the surgical instrument located outside the field of view without moving their gaze significantly. Furthermore, since the marker indicating the surgical instrument located outside the field of view is displayed in response to a predetermined input from the input device, the marker can be kept hidden if no predetermined input is made to the input device. This prevents the marker from being displayed unnecessarily in a position easily visible to the operator.

[0009] A second aspect of the present invention is a display method performed by a control device provided in a surgical system, comprising the steps of: acquiring an endoscopic image captured by an endoscope; determining whether a plurality of surgical instruments are located outside the field of view of the endoscope; determining whether a predetermined input has been made by an input device that performs a predetermined input; and, if at least one of the plurality of surgical instruments is located outside the field of view of the endoscope and a predetermined input has been made by the input device, displaying a graphical user interface, including a marker indicating the surgical instrument located outside the field of view, in an inner region of the display unit's screen that does not include the vicinity of the edge, superimposed on the endoscopic image captured by the endoscope.

[0010] In the second aspect of the present invention, as described above, when at least one of the multiple surgical instruments is located outside the field of view of the endoscope, and a predetermined input is made by the input device, a graphical user interface including a marker indicating the surgical instrument located outside the field of view is displayed on the display unit overlaid on the endoscopic image captured by the endoscope in an inner region of the display unit screen that does not include the vicinity of the edge. As a result, the marker indicating the surgical instrument located outside the field of view is displayed relatively towards the center of the display unit screen rather than near the edge. Consequently, a display method is provided that allows the operator to easily see the marker indicating the surgical instrument located outside the field of view without having to move their gaze significantly. Furthermore, since the marker indicating the surgical instrument located outside the field of view is displayed in response to a predetermined input from the input device, the marker can be kept hidden if no predetermined input is made to the input device. This provides a display method that can prevent the marker from being displayed unnecessarily in a position easily visible to the operator.

[0011] A program according to a third aspect of this invention is a program that causes a computer to display information about surgical instruments, and causes the computer to perform the following steps: acquire an endoscopic image captured by an endoscope; determine whether a plurality of surgical instruments are located outside the field of view of the endoscope; determine whether a predetermined input has been made by an input device that performs a predetermined input; and, if at least one of the plurality of surgical instruments is located outside the field of view of the endoscope and a predetermined input has been made by the input device, display a graphical user interface including a marker indicating the surgical instrument located outside the field of view in an inner region of the display unit screen that does not include the vicinity of the edge of the display unit screen, overlaid on the endoscopic image captured by the endoscope.

[0012] In the program according to the third aspect of this invention, as described above, if at least one of the multiple surgical instruments is located outside the field of view of the endoscope, and a predetermined input is made by the input device, a graphical user interface including a marker indicating the surgical instrument located outside the field of view is displayed on the display unit overlaid on the endoscopic image captured by the endoscope in an inner region of the display unit screen that does not include the vicinity of the edge. As a result, the marker indicating the surgical instrument located outside the field of view is displayed relatively towards the center of the display unit screen rather than near the edge. As a result, a program is provided that allows the operator to easily see the marker indicating the surgical instrument located outside the field of view without moving their gaze significantly. Furthermore, since the marker indicating the surgical instrument located outside the field of view is displayed in response to a predetermined input from the input device, the marker can be hidden if no predetermined input is made to the input device. This makes it possible to provide a program that can prevent the marker from being displayed unnecessarily in a position easily visible to the operator. [Effects of the Invention]

[0013] According to the present invention, the operator can easily see the markings indicating surgical instruments located outside the field of view without having to move their gaze significantly. [Brief explanation of the drawing]

[0014] [Figure 1] This is a diagram showing the configuration of a surgical system according to the first embodiment. [Figure 2] This figure shows the configuration of a medical manipulator according to the first embodiment. [Figure 3] This figure shows the configuration of the operating handle according to the first embodiment. [Figure 4] This figure shows the configuration of the foot pedal according to the first embodiment. [Figure 5] This figure shows the configuration of the arm of a medical manipulator according to the first embodiment. [Figure 6] This is a diagram of forceps. [Figure 7]It is a perspective view showing the configuration of an operation unit of a medical manipulator according to the first embodiment. [Figure 8] It is a diagram showing an endoscope. [Figure 9] It is a diagram showing a pivot position teaching tool. [Figure 10] It is a diagram for explaining the translational movement of the arm. [Figure 11] It is a diagram for explaining the rotational movement of the arm. [Figure 12] It is a block diagram showing the configuration of a control unit of a medical manipulator according to the first embodiment. [Figure 13] It is a diagram showing an image captured by an endoscope and a graphical user interface. [Figure 14] It is a diagram showing an area of a graphical user interface. [Figure 15] It is a diagram for explaining the display of a clutch area. [Figure 16] It is a diagram for explaining the display of a medical instrument status information area. [Figure 17] It is a diagram for explaining the display of a left pop-up area. [Figure 18] It is a diagram for explaining the display of a right pop-up area. [Figure 19] It is a diagram for explaining the display of a touch panel of a remote operation device. [Figure 20] It is a diagram for explaining a label according to the first embodiment. [Figure 21] It is a diagram for explaining a label (operation target) according to the first embodiment. [Figure 22] It is a diagram for explaining a label (non-operation target) according to the first embodiment. [Figure 23] It is a diagram for explaining a label arranged closer to the center side according to the first embodiment. [Figure 24] It is a flowchart for explaining a display method according to the first embodiment. [Figure 25] It is a diagram (1) for explaining a label according to the second embodiment. [Figure 26] Figure (2) illustrates the signage according to the second embodiment. [Figure 27] Figure (3) illustrates the signage according to the second embodiment. [Figure 28] This is a diagram illustrating the signage according to the first modified example. [Figure 29] This is a diagram illustrating the labeling using a second modified example. [Modes for carrying out the invention]

[0015] The following describes embodiments of the present invention based on the drawings.

[0016] [First Embodiment] Referring to Figures 1 to 24, the configuration of the surgical system 100 according to the first embodiment will be described. The surgical system 100 comprises a medical manipulator 1, which is a patient P-side device, and a remote control device 2, which is an operator-side device for operating the medical manipulator 1. The medical manipulator 1 is equipped with a medical trolley 3 and is configured to be movable. The remote control device 2 is positioned at a distance from the medical manipulator 1, and the medical manipulator 1 is configured to be remotely controlled by the remote control device 2. The operator (such as a doctor) inputs a command to the remote control device 2 to cause the medical manipulator 1 to perform a desired action. The remote control device 2 transmits the input command to the medical manipulator 1. The medical manipulator 1 operates based on the received command. The medical manipulator 1 is also positioned in an operating room, which is a sterile field. Note that the surgical system 100 is an example of a "surgical system" in the claims.

[0017] The remote control device 2 is located, for example, inside or outside the operating room. The remote control device 2 includes an operating handle 21, a foot pedal 22, a touch panel 23, a monitor 24, a support arm 25, and a support bar 26. The operating handle 21 constitutes an operating handle for the operator (such as a doctor) to input commands. The monitor 24 is an example of the "first display device" in the claims.

[0018] The operating handle 21 is configured to operate the medical device 4. The operating handle 21 also receives input for the amount of manipulation applied to the medical device 4. The operating handle 21 includes an operating handle 21L located on the left side as viewed from the operator (such as a doctor) and operated by the operator's left hand, and an operating handle 21R located on the right side and operated by the operator's right hand.

[0019] Furthermore, as shown in Figure 3, the operating handle 21 includes a link section 21a, a link section 21b, a link section 21c, and a link section 21d operated by the operator (such as a doctor). The link section 21a rotates around the A4 axis. When the link section 21a rotates around the A4 axis, the arm section 61, which will be described later, rotates around the JT4 axis. The link section 21b rotates around the A5 axis relative to the link section 21a. When the link section 21b rotates around the A5 axis, the arm section 61, which will be described later, rotates around the JT5 axis. The link section 21c rotates around the A6 axis relative to the link section 21b. When the link section 21c rotates around the A6 axis, the arm section 61 rotates around the JT6 axis. The link section 21d rotates around the A7 axis relative to the link section 21c. As the link portion 21d rotates around the A7 axis, the arm portion 61 rotates around the JT7 axis. The medical instrument 4 is an example of a "surgical instrument" as defined in the claims.

[0020] Furthermore, the operating handle 21 changes (scales) the amount of movement of the arm 60 (medical instrument 4) in relation to the amount of operation received by the operating handle 21. For example, if the multiplier for the amount of movement is set to 1 / 2, the medical instrument 4 is controlled to move a distance that is half the distance of movement of the operating handle 21. This allows for precise and detailed surgical procedures to be performed. The arm 60 is an example of the "manipulator" in the claims.

[0021] As shown in Figure 4, the foot pedals 22 are provided in multiple configurations to perform functions related to the medical device 4. The multiple foot pedals 22 are arranged on a base 28. The foot pedals 22 include a switching pedal 22a, a clutch pedal 22b, a camera pedal 22c, an incision pedal 22d, and a coagulation pedal 22e. The switching pedal 22a, clutch pedal 22b, camera pedal 22c, incision pedal 22d, and coagulation pedal 22e are operated by the operator's feet. The incision pedal 22d includes an incision pedal 22dR for the right arm 60 and an incision pedal 22dL for the left arm 60. The coagulation pedal 22e includes a coagulation pedal 22eR for the right arm 60 and a coagulation pedal 22eL for the left arm 60. The camera pedal 22c is an example of an "input device" as defined in the claims.

[0022] The selector pedal 22a is configured to switch the arm 60 that is operated by the operating handle 21. In the first embodiment, the clutch pedal 22b is configured to perform a clutch operation that temporarily disconnects the operating connection between the arm 60 and the operating handle 21. While the clutch pedal 22b is pressed down by the operator, the operation by the operating handle 21 is no longer transmitted to the arm 60.

[0023] The camera pedal 22c is operated to cause the two operating handles 21 to move the endoscope 6. Specifically, the camera pedal 22c is provided for inputting commands to move the endoscope 6. Specifically, when the camera pedal 22c is pressed down by the operator, a command to move the endoscope 6 is input. Furthermore, when a command enabling the movement of the endoscope 6 is input by the camera pedal 22c (i.e., while the camera pedal 22c is pressed down by the operator), the endoscope 6 is configured to move by operating both the operating handle 21R and the operating handle 21L.

[0024] While the incision pedal 22d (coagulation pedal 22e) is pressed down by the operator, an electrosurgical device (not shown) is activated.

[0025] As shown in Figure 1, the monitor 24 is a scope-type display device for displaying endoscopic images (see Figure 13) captured by the endoscope 6. The support arm 25 supports the monitor 24 so that its height is at the same height as the operator's (such as a doctor's) face. The touch panel 23 is located on the support bar 26. The medical manipulator 1 can be operated by the remote control device 2 by detecting the operator's head using a sensor (not shown) located near the monitor 24. The operator operates the operating handle 21 and foot pedal 22 while viewing the affected area on the monitor 24. This inputs a command to the remote control device 2. The command input to the remote control device 2 is transmitted to the medical manipulator 1.

[0026] The medical trolley 3 is equipped with a control unit 31 that controls the operation of the medical manipulator 1, and a storage unit 32 that stores programs for controlling the operation of the medical manipulator 1. Based on commands input to the remote control device 2, the control unit 31 of the medical trolley 3 controls the operation of the medical manipulator 1.

[0027] Furthermore, the medical trolley 3 is equipped with an input device 33. The input device 33 is configured to accept commands to move or change the posture of the positioner 40, arm base 50, and multiple arms 60, primarily for preparing for surgery before the procedure.

[0028] The medical manipulator 1 shown in Figures 1 and 2 is positioned in an operating room. The medical manipulator 1 comprises a medical trolley 3, a positioner 40, an arm base 50, and multiple arms 60. The arm base 50 is attached to the tip of the positioner 40. The arm base 50 has a relatively long rod shape (elongated shape). The base of each of the multiple arms 60 is attached to the arm base 50. The multiple arms 60 are configured to be able to assume a folded position (storage position). The arm base 50 and the multiple arms 60 are covered with a sterile drape (not shown) during use. The arms 60 also support medical instruments 4.

[0029] The positioner 40 is composed of, for example, a 7-axis articulated robot. The positioner 40 is also positioned on a medical trolley 3. The positioner 40 moves the arm base 50. Specifically, the positioner 40 is configured to move the position of the arm base 50 in three dimensions.

[0030] The positioner 40 also includes a base portion 41 and a plurality of link portions 42 connected to the base portion 41. The plurality of link portions 42 are connected to each other by joint portions 43.

[0031] As shown in Figure 1, a medical instrument 4 is attached to the tip of each of the multiple arms 60. The medical instrument 4 includes, for example, interchangeable instruments and an endoscope 6 (see Figure 8) for imaging the surgical site.

[0032] Furthermore, the surgical system 100 is equipped with a monitor cart 8, as shown in Figure 1. The monitor cart 8 is equipped with a display unit 8a. The display unit 8a is provided separately from the monitor 24 of the remote control device 2. The monitor cart 8 is also equipped with an image processing device 8b. The display unit 8a of the monitor cart 8 displays the same image as the image displayed on the monitor 24 of the remote control device 2. In other words, the image displayed on the monitor 24 and viewed by the surgeon can be viewed by workers (nurses, assistants, etc.) around the medical manipulator 1 and patient P via the display unit 8a of the monitor cart 8. Note that the display unit 8a is an example of the "second display device" in the claims. The image processing device 8b is an example of the "control device" in the claims.

[0033] As shown in Figure 5, the instrument is equipped with a driven unit 4a, which is driven by a servo motor M2 located on the holder 71 of the arm 60. A forceps 4b is also provided at the tip of the instrument.

[0034] Furthermore, as shown in Figure 6, the instrument includes a first support 4e that rotatably supports the proximal ends of the end effector members 104a and 104b at the tip side around the JT11 axis, a second support 4f that rotatably supports the proximal end of the first support 4e at the tip side around the JT10 axis, and a shaft 4c connected to the proximal end of the second support 4f. The driven unit 4a, the shaft 4c, the second support 4f, the first support 4e, and the forceps 4b are arranged along the Z direction. The JT11 axis is perpendicular to the direction in which the shaft 4c extends (Z direction). The JT10 axis is spaced apart from the JT11 axis in the direction in which the shaft 4c extends and is perpendicular to the direction in which the shaft 4c extends and to the JT11 axis.

[0035] The first support 4e has a forceps 4b attached to it so as to rotate around the axis of the JT11 axis. The second support 4f supports the first support 4e so as to rotate around the JT10 axis. In other words, the first support 4e is attached to the second support 4f so as to rotate around the axis of the JT10 axis. The tip side (Z1 direction side) of the first support 4e has a U shape. The tool center point (TCP1, clevis) is set at the center of the U-shaped tip side of the first support 4e on the axis of the JT11 axis.

[0036] Furthermore, the medical instrument 4 (forceps 4b) is equipped with a JT9 axis as the rotation axis of the shaft 4c (the axis along the direction in which the shaft 4c extends) and a JT12 axis as the opening and closing axis of the forceps 4b. Multiple servo motors (for example, four) are provided on the holder 71 of the arm 60, and the rotating body of the driven unit 4a is driven by these multiple servo motors M2. This drives the medical instrument 4 around the J9 to J12 axes.

[0037] Furthermore, as shown in Figure 8, TCP2 of the endoscope 6 is set at the tip of the endoscope 6.

[0038] Next, we will describe the configuration of arm 60 in detail.

[0039] As shown in Figure 5, the arm 60 includes an arm section 61 (base section 62, link section 63, joint section 64) and a translational movement mechanism section 70 provided at the tip of the arm section 61. The arm 60 is configured to move its tip in three dimensions relative to its root side (arm base 50). The arm section 61 is composed of a 7-axis articulated robot arm. Multiple arms 60 have similar configurations.

[0040] As shown in Figure 5, the arm 60 is equipped with JT1 to JT7 axes as rotation axes and JT8 axis as a linear motion axis. The JT1 to JT7 axes correspond to the rotation axes of the joint 64 of the arm 61. The JT7 axis corresponds to the base end link portion 72 of the translational movement mechanism 70. The JT8 axis corresponds to the axis that moves the tip end link portion 73 of the translational movement mechanism 70 relative to the base end link portion 72 along the Z direction. That is, the servo motor M1 shown in Figure 12 is provided to correspond to the JT1 to JT7 axes of the arm 60. The servo motor M3 is provided to correspond to the JT8 axis.

[0041] The translational movement mechanism 70 is provided at the tip of the arm 61 and the medical instrument 4 is attached to it. The translational movement mechanism 70 translates the medical instrument 4 in the direction of insertion into the patient P. The translational movement mechanism 70 is configured to translate the medical instrument 4 relative to the arm 61. Specifically, the translational movement mechanism 70 is provided with a holder 71 for holding the medical instrument 4. A servo motor M2 (see Figure 12) is housed in the holder 71.

[0042] As shown in Figure 7, the medical manipulator 1 is attached to the arm 60 and includes an operating unit 80 for operating the arm 60. The operating unit 80 includes an enable switch 81 and a joystick 82 and a switch unit 83. The enable switch 81 permits or dispermits the movement of the arm 60 by the joystick 82 and the switch unit 83. The enable switch 81 is also activated when an operator (nurse, assistant, etc.) grasps and presses the operating unit 80, thereby permitting the movement of the medical instrument 4 by the arm 60. The joystick 82 and the switch unit 83 are used to operate the arm 60. The enable switch 81 is an example of the "input device" and "switch" in the claims. The joystick 82 and the switch unit 83 are examples of the "operating members" in the claims.

[0043] Furthermore, the switch unit 83 includes a switch unit 83a that moves the medical device 4 toward the direction of insertion into the patient P along the longitudinal direction of the medical device 4, and a switch unit 83b that moves the medical device 4 toward the opposite direction from the direction of insertion into the patient P. Both the switch unit 83a and the switch unit 83b are composed of push-button switches.

[0044] Furthermore, as shown in Figure 7, the operating unit 80 includes a pivot button 85 that teaches the pivot position PP, which serves as the fulcrum (see Figure 11) for the movement of the medical instrument 4 attached to the arm 60. The pivot button 85 is located on the surface 80b of the operating unit 80, adjacent to the enable switch 81. When the tip of the endoscope 6 (see Figure 8) or the pivot position teaching device 7 (Figure 9) is moved to a position corresponding to the insertion position of the trocar T inserted into the patient P's body surface S, the pivot position PP is taught when the pivot button 85 is pressed and stored in the memory unit 32. Note that in teaching the pivot position PP, the pivot position PP is set as a single point (coordinate), and teaching the pivot position PP does not set the direction of the medical instrument 4.

[0045] As shown in Figure 1, an endoscope 6 is attached to one of the multiple arms 60 (for example, arm 60c), and other medical instruments 4 are attached to the remaining arms 60 (for example, arms 60a, 60b, and 60d). Specifically, during surgery, an endoscope 6 is attached to one of the four arms 60, and other medical instruments 4 (such as forceps 4b) are attached to the other three arms 60. The pivot position PP is then taught to the arm 60 to which the endoscope 6 is attached, with the endoscope 6 attached. The pivot position PP is also taught to the arms 60 to which the other medical instruments 4 are attached, with the pivot position teaching device 7 attached. The endoscope 6 is attached to one of the two centrally located arms 60 (arms 60b and 60c) of the four arms 60 which are arranged adjacent to each other. In other words, the pivot position PP is set individually for each of the multiple arms 60.

[0046] Furthermore, as shown in Figure 7, an adjustment button 86 for optimizing the position of the arm 60 is provided on the surface 80b of the operating unit 80. After teaching the pivot position PP for the arm 60 to which the endoscope 6 is attached, pressing the adjustment button 86 optimizes the position of the other arm 60 (arm base 50).

[0047] Furthermore, as shown in Figure 7, the control unit 80 includes a mode switching button 84 that switches between a mode for translating (see Figure 10) and a mode for rotating (see Figure 11) the medical device 4 attached to the arm 60. A mode indicator 84a is also provided near the mode switching button 84. The mode indicator 84a displays the switched mode. Specifically, the current mode (translation mode or rotation mode) is displayed by whether the mode indicator 84a lights up (rotation mode) or turns off (translation mode).

[0048] Furthermore, the mode indicator 84a also functions as a pivot position indicator, showing that the pivot position PP has been taught.

[0049] As shown in Figure 10, in the mode in which the arm 60 is translated, the arm 60 is moved so that the tip 4d of the medical instrument 4 moves on the XY plane. Also, as shown in Figure 11, in the mode in which the arm 60 is rotated, when the pivot position PP is not taught, the arm 60 rotates around the forceps 4b, and when the pivot position PP is taught, the arm 60 is moved so that the medical instrument 4 rotates with the pivot position PP as the fulcrum. Note that the medical instrument 4 is rotated while its shaft 4c is inserted into the trocar T.

[0050] Furthermore, as shown in Figure 12, the arm 60 is equipped with multiple servo motors M1, an encoder E1, and a reduction gear (not shown) corresponding to the multiple joints 64 of the arm section 61. The encoder E1 is configured to detect the rotation angle of the servo motor M1. The reduction gear is configured to reduce the rotation of the servo motor M1 and increase the torque.

[0051] Furthermore, as shown in Figure 12, the translational movement mechanism 70 is equipped with a servo motor M2 for rotating a rotating body provided on the driven unit 4a of the medical instrument 4, a servo motor M3 for translating the medical instrument 4, encoders E2 and E3, and a reduction gear (not shown). Encoders E2 and E3 are configured to detect the rotation angles of servo motors M2 and M3, respectively. The reduction gear is configured to reduce the rotation of servo motors M2 and M3 to increase torque.

[0052] Furthermore, the positioner 40 is equipped with multiple servo motors M4, an encoder E4, and a reduction gear (not shown) to correspond to the multiple joints 43 of the positioner 40. The encoder E4 is configured to detect the rotation angle of the servo motors M4. The reduction gear is configured to reduce the rotation of the servo motors M4 and increase the torque.

[0053] The medical cart 3 is equipped with a servo motor M5, an encoder E5, a reduction gear (not shown), and a brake, each of which drives one of the multiple front wheels (not shown). The reduction gear is configured to reduce the rotation of the servo motor M5 and increase the torque. The throttle section 34a of the medical cart 3 is equipped with a potentiometer P1 (see Figure 1), and the servo motors M5 of the front wheels are driven based on the rotation angle detected by the potentiometer P1 in response to the twisting of the throttle section 34a. The rear wheels (not shown) of the medical cart 3 are of the twin-wheel type, and the rear wheels are steered based on the left and right rotation of the operating handle 34. The operating handle 34 of the medical cart 3 is equipped with a potentiometer P2 (see Figure 2), and the rear wheels of the medical cart 3 are equipped with a servo motor M6, an encoder E6, and a reduction gear (not shown). The reduction gear is configured to reduce the rotation of the servo motor M6 and increase the torque. The servo motor M6 is driven based on the rotation angle detected by the potentiometer P2 in response to the left and right rotation of the control handle 34. In other words, the steering of the rear wheels caused by the left and right rotation of the control handle 34 is power-assisted by the servo motor M6.

[0054] Furthermore, the medical trolley 3 moves in the forward and backward direction when its front wheels are driven. Also, when the operating handle 34 of the medical trolley 3 is rotated, the rear wheels are steered, causing the medical trolley 3 to rotate in the left and right direction.

[0055] The control unit 31 of the medical trolley 3 includes an arm control unit 31a that controls the movement of a plurality of arms 60 based on a command, and a positioner control unit 31b that controls the movement of the positioner 40 and the driving of the front wheels (not shown) of the medical trolley 3 based on a command. The arm control unit 31a is electrically connected to a servo control unit C1 for controlling a servo motor M1 for driving the arms 60. The servo control unit C1 is also electrically connected to an encoder E1 for detecting the rotation angle of the servo motor M1.

[0056] Furthermore, a servo control unit C2 for controlling a servo motor M2 for driving the medical device 4 is electrically connected to the arm control unit 31a. An encoder E2 for detecting the rotation angle of the servo motor M2 is electrically connected to the servo control unit C2. Furthermore, a servo control unit C3 for controlling a servo motor M3 for translating the translational movement mechanism 70 is electrically connected to the arm control unit 31a. An encoder E3 for detecting the rotation angle of the servo motor M3 is electrically connected to the servo control unit C3.

[0057] The motion command input to the remote control device 2 is then input to the arm control unit 31a. The arm control unit 31a generates a position command based on the input motion command and the rotation angle detected by the encoder E1 (E2, E3), and outputs the position command to the servo control unit C1 (C2, C3). The servo control unit C1 (C2, C3) generates a torque command based on the position command input from the arm control unit 31a and the rotation angle detected by the encoder E1 (E2, E3), and outputs the torque command to the servo motor M1 (M2, M3). As a result, the arm 60 is moved in accordance with the motion command input to the remote control device 2.

[0058] Furthermore, as shown in Figure 12, the control unit 31 (arm control unit 31a) is configured to operate the arm 60 based on input signals from the joystick 82 of the operation unit 80. Specifically, the arm control unit 31a generates a position command based on the input signal (operation command) received from the joystick 82 and the rotation angle detected by the encoder E1, and outputs the position command to the servo control unit C1. The servo control unit C1 generates a torque command based on the position command received from the arm control unit 31a and the rotation angle detected by the encoder E1, and outputs the torque command to the servo motor M1. As a result, the arm 60 is moved in accordance with the operation command input to the joystick 82.

[0059] The control unit 31 (arm control unit 31a) is configured to operate the arm 60 based on input signals from the switch unit 83 of the operation unit 80. Specifically, the arm control unit 31a generates a position command based on the input signal (operation command) received from the switch unit 83 and the rotation angle detected by the encoder E1 or E3, and outputs the position command to the servo control unit C1 or C3. The servo control unit C1 or C3 generates a torque command based on the position command received from the arm control unit 31a and the rotation angle detected by the encoder E1 or E3, and outputs the torque command to the servo motor M1 or M3. As a result, the arm 60 is moved in accordance with the operation command input to the switch unit 83.

[0060] Furthermore, as shown in Figure 12, a servo control unit C4 for controlling the servo motor M4 that moves the positioner 40 is electrically connected to the positioner control unit 31b. An encoder E4 for detecting the rotation angle of the servo motor M4 is electrically connected to the servo control unit C4. In addition, a servo control unit C5 for controlling the servo motor M5 that drives the front wheels (not shown) of the medical trolley 3 is electrically connected to the positioner control unit 31b. An encoder E5 for detecting the rotation angle of the servo motor M5 is electrically connected to the servo control unit C5.

[0061] Furthermore, operation commands related to setting the preparation position are input from the input device 33 to the positioner control unit 31b. The positioner control unit 31b generates a position command based on the operation command input from the input device 33 and the rotation angle detected by the encoder E4, and outputs the position command to the servo control unit C4. The servo control unit C4 generates a torque command based on the position command input from the positioner control unit 31b and the rotation angle detected by the encoder E4, and outputs the torque command to the servo motor M4. As a result, the positioner 40 is moved in accordance with the operation command input to the input device 33. Similarly, based on the operation command from the input device 33, the positioner control unit 31b moves the medical trolley 3.

[0062] The surgical system 100 includes an image processing device 8b. The image processing device 8b performs processing based on a predetermined program. The image processing device 8b is composed of a computer. The image processing device 8b includes a processing unit such as a CPU that executes the program, and a storage unit such as memory where the program is stored. The image processing device 8b is also configured to generate a graphical user interface G (see Figure 14) and to overlay the graphical user interface G onto the endoscopic image (see Figure 13) captured by the endoscope 6 and display it on the monitor 24 of the remote control device 2. The image processing device 8b is also configured to overlay the graphical user interface G onto the endoscopic image (see Figure 13) captured by the endoscope 6 and display it on the display unit 8a. The image processing device 8b is also configured to acquire images from the endoscope 6.

[0063] As shown in Figure 14, the graphical user interface G includes a clutch area G1. As shown in Figure 15, the clutch area G1 displays the state of the clutch pedal 22b. Figure 15(a) shows the state when the clutch pedal 22b is not pressed (OFF state). Figure 15(b) shows the state when the operator has their foot on the clutch pedal 22b (hover state). Figure 15(c) shows the state when the clutch pedal 22b is pressed (ON state).

[0064] As shown in Figure 14, in the first embodiment, the graphical user interface G includes a camera area G2 that displays information about the endoscope 6. The camera area G2 is displayed in the area near the lower edge ed (see Figure 13) of the screen gr of the monitor 24.

[0065] As shown in Figure 14, the graphical user interface G includes a hand area G3. The hand area G3 displays information about each medical instrument 4, each arm 60, and the status of the coagulation pedal 22e and the incision pedal 22d. The hand area G3 includes hand area G3a, which shows information about the medical instrument 4 and arm 60a (number "4" in hand area G3) operated by the left-hand operating handle 21L; hand area G3b, which shows information about the replacement medical instrument 4 and arm 60b (number "3" in hand area G3); and hand area G3c, which shows information about the medical instrument 4 and arm 60d (number "1" in hand area G3) operated by the left-hand operating handle 21L. The hand area G3 is displayed in the area near the lower edge ed of the screen gr of the monitor 24. The clutch area G1 is also displayed in the area near the lower edge ed of the screen gr. Specifically, as shown in Figure 14, the clutch area G1, camera area G2, and hand area G3 are displayed between the lower edge ed of the screen gr and a position above the edge ed by a length of 1 / 10 (L11) of the vertical length of the screen gr.

[0066] Furthermore, information regarding arm 60 includes the arm number (such as "1" or "2") and an arrow icon displayed when it is set as the replacement target for arm 60 to which a replacement medical device 4 is attached; information regarding medical device 4 includes the name of medical device 4; and information regarding the status of the coagulation pedal 22e and incision pedal 22d includes the operating status of the clutch pedal 22b, the operating status of the incision pedal 22d, and the operating status of the coagulation pedal 22e.

[0067] Furthermore, as shown in Figure 13, the hand areas of the arm 60 to be operated (hand area G3a and hand area G3c) are displayed in dark gray, with a white oval outlined within them, and the arm 60 numbers (1 and 4) displayed within the white oval outline. The hand area of ​​the arm 60 not to be operated (hand area G3b) is displayed in light gray, and the arm 60 number (3) is displayed in an even lighter gray.

[0068] As shown in Figure 14, the graphical user interface G includes a pop-up area, the medical device status information area G4. The medical device status information area G4 displays the current number of uses / maximum number of uses (see Figure 16) of the medical device 4 attached to each arm 60 as a pop-up. When the current number of uses equals the maximum number of uses, the current number of uses is displayed in red. In addition, if an error occurs in the medical device 4 attached to each arm 60, error information is displayed as a pop-up. If no medical device 4 is attached to the arm 60, nothing is displayed in the medical device status information area G4. The medical device status information area G4 is displayed on the monitor 24 in an adjacent area above the clutch area G1, camera area G2, and hand area G3.

[0069] As shown in Figure 14, the graphical user interface G includes a level indicator area G5. The level indicator area G5 displays the angle information of the endoscope 6. The level indicator area G5 is also displayed only while the camera pedal 22c is pressed down. That is, when the image processing device 8b receives a command to enable movement of the endoscope 6, it displays the level indicator LV of the endoscope 6 in the level indicator area G5. Note that the level indicator area G5 is an example of the "center area" in the claims.

[0070] As shown in Figure 14, the graphical user interface G includes a left pop-up area G6. When the foot is on the foot pedal 22 in a hover state, the icons shown in Figures 17(a) to (c) are displayed in the left pop-up area G6. Figure 17(a) shows the icons displayed when the foot is on the coagulation pedal 22eL and the incision pedal 22dL. Figure 17(b) shows the icons displayed when the foot is on the clutch pedal 22b. Figure 17(c) shows the icons displayed when the foot is on the camera pedal 22c. The left pop-up area G6 is displayed on the left side of the monitor 24.

[0071] As shown in Figure 14, the graphical user interface G includes a right pop-up area G7. The right pop-up area G7 displays icons (Figure 18) when the feet are placed on the coagulation pedal 22eR and the incision pedal 22dR. The right pop-up area G7 is displayed on the right side of the monitor 24.

[0072] Furthermore, as shown in Figure 14, the graphical user interface G includes a first area G8 that displays the range of motion of the arm 60 and the range of motion of the arm 60 that can be operated by the operating handle 21. The graphical user interface G also includes a second area G9 that displays the direction of operation of the operating handle 21 necessary to return the operating handle 21 to the operating range and / or (in the first embodiment, "and") return the arm 60 to the operating range.

[0073] Furthermore, there are two arms 60 that can be operated by the operating handle 21. For example, the left arm 60L (e.g., arm 60a, see Figure 1) for supporting the medical instrument 4 is operated by the operating handle 21L. The right arm 60R (e.g., arm 60d, see Figure 1) for supporting the medical instrument 4 is operated by the operating handle 21R. The first area G8 (first area G8L, first area G8R) and the second area G9 (second area G9L, second area G9R) are provided separately for the left arm 60L and the right arm 60R, respectively.

[0074] Furthermore, as shown in Figure 14, the graphical user interface G includes error notification areas G15 (G15a, G15b). Error notification area G15a is displayed as a pop-up to show warning or error information when a warning or error occurs. Error notification area G15b is displayed as a pop-up to show details of the notes related to the warning or error displayed in error notification area G15a.

[0075] The markers MK1, shown in Figure 13, and MK2, shown in Figure 25, which indicate medical instruments 4 located outside the field of view of the endoscope 6, are configured to be switchable between being displayed and not displayed. Specifically, as shown in Figure 19, when the touch panel 23 of the remote control device 2 is operated, the "Display Out-of-Field Forceps" button is displayed. When "On" is selected for "Display Out-of-Field Forceps," if a predetermined input operation is performed on the camera pedal 22c by the operator, markers MK1 and MK2 will be displayed if medical instruments 4 are located outside the field of view of the endoscope 6. Conversely, when "Off" is selected for "Display Out-of-Field Forceps," markers MK1 and MK2 will not be displayed. This allows the display setting of "Display Out-of-Field Forceps" to be easily switched according to the operator's skill level and needs.

[0076] As shown in Figure 14, the graphical user interface G includes a status area G10. The status area G10 displays information such as the remaining battery level of the medical manipulator 1, the brightness / contrast of the monitor 24, lap time, and elapsed time of surgery. Specifically, as shown in Figure 14, the status area G10 is displayed between the upper edge eu of the screen gr and a position below the edge eu by a length of 1 / 10 (L11) of the vertical length of the screen gr.

[0077] In the first embodiment, as shown in Figures 13 and 14, the image processing device 8b, in response to a predetermined input from the camera pedal 22c or enable switch 81, displays a marker MK1 indicating the medical instrument 4 located outside the field of view in the inner area of ​​the graphical user interface G, excluding the vicinity of the edge of the display screen, when at least one of the multiple medical instruments 4 is located outside the field of view of the endoscope 6. The image processing device 8b is also configured to display the marker MK1 in the area G11 near the outer edge of the level indicator area G5 of the graphical user interface G. In other words, the marker MK1 (graphical user interface G) is displayed in the area G11 near the outer edge of the level indicator area G5, which includes the central part CN1 but does not include the vicinity of the edge e of the screen gr of the monitor 24. The area G11 near the outer edge is the area located outside and near the level indicator LV, which will be described later.

[0078] Furthermore, in the first embodiment, the image processing device 8b is configured to display the marker MK1 on the graphical user interface G in response to a predetermined input which is an operation on the camera pedal 22c different from a camera movement operation performed on the camera pedal 22c to move the endoscope 6. For example, the image processing device 8b is configured to display the marker MK1 on the graphical user interface G in response to a predetermined input which is an operation on the camera pedal 22c different from a camera movement operation, in which the camera pedal 22c is operated twice in a row (double-click operation) within a predetermined time.

[0079] Furthermore, if the image processing device 8b receives a predetermined input that is an operation on the camera pedal 22c other than camera movement operation, it enables operation of the medical device 4 by operating the operating handles 21L and 21R. In other words, if a predetermined input that is an operation on the camera pedal 22c other than camera movement operation is performed, the endoscope 6 cannot be moved using the operating handles 21L and 21R.

[0080] Furthermore, in the first embodiment, the image processing device 8b is configured to display the indicator MK1 on the graphical user interface G in response to the operation of an enable switch 81 attached to the arm 60 as a predetermined input. In other words, when an operator such as an assistant or nurse presses the enable switch 81 to move the arm 60, the indicator MK1 is displayed on the graphical user interface G.

[0081] The indicator MK1 is configured to be switchable between a displayed state and a hidden state. Specifically, the image processing device 8b is configured to display the indicator MK1 on the graphical user interface G only when a predetermined input is received by the camera pedal 22c or the enable switch 81.

[0082] The image processing device 8b is configured to determine whether the multiple medical instruments 4, each supported by one of the multiple arms 60, are located outside the field of view of the endoscope 6, based on the imaging range information of the endoscope 6 and the position information of the tip of each of the multiple medical instruments 4. In other words, the image processing device 8b acquires the position of the medical instruments 4 based on the posture and position of the arms 60. The image processing device 8b also acquires the imaging direction of the endoscope 6 based on the posture and position of the arms 60. Furthermore, the image processing device 8b acquires the field of view (field of view) of the endoscope 6 based on the zoom state of the endoscope 6. The image processing device 8b acquires the field of view (field of view) of the endoscope 6 using a value set as a mechanical mechanism (lens, etc.) of the endoscope 6. Then, the image processing device 8b acquires the coordinates of the tip of the medical instrument 4 relative to the field of view of the endoscope 6 from the field of view of the endoscope 6, the posture and position of the endoscope 6, and the position information of the arms 60. As a result, the image processing device 8b determines whether the medical instruments 4 are located outside the field of view of the endoscope 6.

[0083] Furthermore, in the examples shown in Figures 13 and 14, among arms 60a, 60b, 60c, and 60d, arm 60a (number "4" in hand area G3a), arm 60b (number "3" in hand area G3b), and arm 60d (number "1" in hand area G3c) have medical instruments 4 (medical instruments 4 other than the endoscope 6, such as forceps 4b). In addition, the endoscope 6 is attached to arm 60c (number "2" in camera area G2). Arm 60a (number "4" in hand area G3a) and arm 60d (number "1" in hand area G3) are in an operable state (active) by the operating handle 21, and hand areas G3a and G3c are displayed in dark gray. Arm 60b is in an inactive state (not operable by the operating handle 21), and hand area G3b is displayed in light gray.

[0084] In the example shown in Figure 13, the medical instrument 4 supported by arm 60a (number "4" in hand area G3a) and arm 60d (number "1" in hand area G3c) is located within the field of view of the endoscope 6. On the other hand, the medical instrument 4 supported by arm 60b (number "3" in hand area G3b) is located outside the field of view of the endoscope 6.

[0085] In the first embodiment, as shown in Figure 13, the image processing device 8b is configured to display a marker MK1 in the direction in which the medical instrument 4 outside the field of view is located within the area G11 near the outer edge of the level indicator area G5. In other words, the graphical user interface G displays the marker MK1 indicating the medical instrument 4 outside the field of view in the area G11 near the outer edge, and in the area G12 (see Figure 20) which corresponds to the direction in which the medical instrument 4 outside the field of view is located relative to the center CN2 of the level indicator area G5. In Figure 13, the marker MK1 (the character "3") is displayed.

[0086] Furthermore, as shown in Figure 20, the area G12 where the marker MK1 indicating the medical device 4 outside the field of view is displayed is one of the areas G12 obtained by dividing the area near the outer edge G11 radially from the central CN2 of the level area G5. In the first embodiment, the area near the outer edge G11 is divided into eight radial sections from the central CN2. In Figure 13, the marker MK1 (the letter "3") is displayed, for example, in the upper left region G12d (see Figure 20). Note that Figure 20 shows all patterns in which the medical device 4 is located outside the field of view.

[0087] Furthermore, as shown in Figure 20, the image processing device 8b is configured to display a marker MK1 indicating the direction corresponding to the area where the medical device 4 is located, which is one of eight divisions of the out-of-field area according to the shape of the rectangular display unit, where the vertical length and horizontal length are not equal.

[0088] Furthermore, as shown in Figure 20, the display unit has a horizontal length greater than its vertical length. The area outside the field of view is divided into eight regions according to the shape of the display unit, such that the angles of the upper region G12c and the lower region G12g are greater than the angles of the right region G12a, the left region G12e, the upper right region G12b, the upper left region G12d, the lower right region G12h, and the lower left region G12f.

[0089] Furthermore, as shown in Figure 20, the aspect ratio (width-to-height ratio) of the outer edge of the area G11 near the outer edge is the same as that of the display area. For example, the aspect ratio of the outer edge of the area G11 near the outer edge and the aspect ratio of the display area are horizontal length:vertical length = 16:9.

[0090] Furthermore, the image processing device 8b is configured to divide the area outside the field of view into eight sections using lines that divide the area outside the field of view into eight sections, such that the length of the line segments in each section, which divides the upper, lower, right, and left edges of the display unit, is greater for the upper and lower sections than for the right, left, upper right, upper left, lower right, and lower left sections.

[0091] The image processing device 8b is configured to divide the area outside the field of view into eight sections using a straight line connecting a point 1 / n from the left of the top edge of the display unit and a point 1 / n from the right of the bottom edge, a straight line connecting a point 1 / n from the right of the top edge of the display unit and a point 1 / n from the left of the bottom edge, a straight line connecting a point 1 / n from the top of the left edge of the display unit and a point 1 / n from the bottom of the right edge, and a straight line connecting a point 1 / n from the bottom of the left edge and a point 1 / n from the top of the right edge. n is greater than 3 and less than 8. For example, the image processing device 8b is configured to divide the area outside the field of view into eight sections using a straight line connecting a point one-quarter from the left of the top edge of the display unit and a point one-quarter from the right of the bottom edge, a straight line connecting a point one-quarter from the right of the top edge of the top edge and a point one-quarter from the left of the bottom edge, a straight line connecting a point one-quarter from the top of the left edge of the display unit and a point one-quarter from the bottom of the right edge, and a straight line connecting a point one-quarter from the bottom of the left edge and a point one-quarter from the top of the right edge. In the example shown in Figure 20, L1 is one-quarter of the length of the top and bottom edges, and L2 is one-quarter of the length of the right and left edges.

[0092] For example, in the case of a screen with a width-to-height ratio of 16:9, the angle θ1 between the upper region G12c and the lower region G12g is approximately 41.6 degrees, the angle θ2 between the right region G12a and the left region G12e is 31.4 degrees, and the angle between the upper right region G12b, the upper left region G12d, the lower right region G12h, and the lower left region G12f is approximately 32.7 degrees.

[0093] Furthermore, the image processing device 8b is configured to display the marker MK1 at a predetermined position within each of the eight divided regions of the outer edge vicinity area G11.

[0094] Furthermore, as shown in Figure 14, when the image processing device 8b receives a command to enable movement of the endoscope 6, it displays the level indicator LV of the endoscope 6 in the level indicator area G5. In other words, while camera movement operations are being performed on the camera pedal 22c, the image processing device 8b displays the level indicator of the endoscope 6 in the level indicator area G5 of the graphical user interface G. The image processing device 8b then displays the graphical user interface G on the monitor 24 for displaying the marker MK1 in the area G11 near the outer edge outside the level indicator area G5. The level indicator LV represents the tilt of the endoscope 6's field of view relative to the patient P.

[0095] Furthermore, as shown in Figure 14, the level indicator area G5 is a rectangular region (a horizontally elongated rectangle) that includes the upper edge G5a, left edge G5b, right edge G5c, and lower edge G5d. By displaying the indicator MK1 in the area G11 near the outer edge of the level indicator area G5, which is of a predetermined size and includes the central part of the screen gr, the operator's visibility can be improved. The upper edge G5a is positioned between a location 1 / 10 (L11) above the central part CN1 of the screen gr of the monitor 24 by a length of 1 / 10 (L11) of the vertical length of the screen gr, and a location 1 / 10 (L11) below the upper edge eu of the screen gr by a length of 1 / 10 (L11) of the vertical length of the screen gr. Preferably, the upper edge portion G5a is positioned between a position 1 / 8 the length of the vertical length of the screen gr from the central portion CN1 of the screen gr of the monitor 24 and a position 1 / 8 the length of the vertical length of the screen gr from the upper edge eu of the screen gr. More preferably, the upper edge portion G5a is positioned between a position 1 / 6 the length of the vertical length of the screen gr from the central portion CN1 of the screen gr of the monitor 24 and a position 1 / 6 the length of the vertical length of the screen gr from the upper edge eu of the screen gr.

[0096] Furthermore, the upper part G5a is located below the status area G10, which displays information such as the remaining battery level of the medical manipulator 1.

[0097] Furthermore, the left-hand portion G5b is positioned between a position 1 / 10 (L12) to the left of the central portion CN1 of the screen gr of the monitor 24, and a position 1 / 10 (L12) to the right of the left edge el of the screen gr. Preferably, the left-hand portion G5b is positioned between a position 1 / 8 of the horizontal length of the screen gr to the left of the central portion CN1 of the screen gr of the monitor 24, and a position 1 / 8 of the horizontal length of the screen gr to the right of the left edge el of the screen gr. More preferably, the left-hand portion G5b is positioned between a position 1 / 6 of the horizontal length of the screen gr to the left of the central portion CN1 of the screen gr of the monitor 24, and a position 1 / 6 of the horizontal length of the screen gr to the right of the left edge el of the screen gr.

[0098] Furthermore, the right-hand portion G5c is positioned between a position to the right of the central portion CN1 of the screen gr of the monitor 24 by a length of 1 / 10 (L12) of the horizontal length of the screen gr, and a position to the left of the right edge er of the screen gr by a length of 1 / 10 (L12) of the horizontal length of the screen gr. Preferably, the right-hand portion G5c is positioned between a position to the right of the central portion CN1 of the screen gr of the monitor 24 by a length of 1 / 8 of the horizontal length of the screen gr, and a position to the left of the right edge er of the screen gr by a length of 1 / 8 of the horizontal length of the screen gr. More preferably, the right-hand portion G5c is positioned between a position to the right of the central portion CN1 of the screen gr of the monitor 24 by a length of 1 / 6 of the horizontal length of the screen gr, and a position to the left of the right edge er of the screen gr by a length of 1 / 6 of the horizontal length of the screen gr.

[0099] Furthermore, the lower edge portion G5d is positioned between a position 1 / 10 (L11) below the central portion CN1 of the screen gr of the monitor 24 by a length of 1 / 10 (L11) of the vertical length of the screen gr and a position 1 / 10 (L11) above the lower edge ed of the screen gr by a length of 1 / 10 of the vertical length of the screen gr. Preferably, the lower edge portion G5d is positioned between a position 1 / 8 of the vertical length of the screen gr of the monitor 24 by a length of 1 / 8 of the vertical length of the screen gr and a position 1 / 8 of the vertical length of the screen gr above the lower edge ed of the screen gr. More preferably, the lower edge portion G5d is positioned between a position 1 / 6 of the vertical length of the screen gr of the monitor 24 by a length of 1 / 6 of the vertical length of the screen gr and a position 1 / 6 of the vertical length of the screen gr above the lower edge ed of the screen gr.

[0100] The lower edge G5d of the level indicator area G5 is positioned above the hand area G3a, hand area G3b, hand area G3c, and camera area G2. Furthermore, the lower edge G5d of the level indicator area G5 is positioned above the medical device status information area G4.

[0101] Note that the central part CN1 of the screen gr of monitor 24 and the central part CN2 of the level indicator area G5 are in approximately the same position.

[0102] Furthermore, the marker MK1 indicating the medical instrument 4 located outside the field of view is positioned between the level area G5 and at least one (all of the first embodiment) of the medical instrument status information area G4, left pop-up area G6, right pop-up area G7, and status area G10. The medical instrument status information area G4 indicates the status of the medical instrument 4. The left pop-up area G6 is displayed when the foot pedal 22 is operated (when the foot is on the foot pedal 22 in a hover state). The right pop-up area G7 is displayed when the coagulation pedal 22eR and the incision pedal 22dR are operated (when the operator's foot is on them). The status area G10 indicates the status of the surgical system 100.

[0103] Furthermore, in the first embodiment, as shown in Figure 13, the label MK1 includes a number that identifies the arm 60 supporting the medical instrument 4 located outside the field of view. In the example shown in Figure 13, the number "3" is displayed as MK1, indicating that the medical instrument 4 supported by arm 60b (number "3" in hand area G3) is located outside the field of view of the endoscope 6.

[0104] Furthermore, as shown in Figures 21 and 22, the indicator MK1 is displayed in different ways depending on whether the medical device 4, which is located outside the field of view, is the target of operation or not. Specifically, as shown in Figure 21, when the medical device 4 is the target of operation, a white oval is shown inside a black square, and the arm number 60 (one of 1 to 4) is displayed inside the white oval. Also, as shown in Figure 22, when the medical device 4 is not the target of operation, a gray arm number 60 (one of 1 to 4) is displayed inside a black square.

[0105] Furthermore, as shown in Figure 20, if multiple of the medical instruments 4 attached to arm 60a, arm 60b, and arm 60d are located outside the field of view of the endoscope 6, multiple markers MK1 corresponding to the medical instruments 4 located outside the field of view of the endoscope 6 will be displayed in the order of the positions of arms 60a, arm 60b, and arm 60d. The order of the positions of arms 60a, arm 60b, and arm 60d refers to the order of the positions of arms 60a, arm 60b, and arm 60d when they are attached to the arm base 50.

[0106] Furthermore, as shown in Figure 20, if all three medical instruments 4 attached to arm 60a, arm 60b, and arm 60d are located outside the field of view of the endoscope 6, then the marker MK1 representing arm 60a ("4"), the marker MK1 representing arm 60b ("3"), and the marker MK1 representing arm 60d ("1") will be displayed in that order. In other words, the markers MK1 ("4", "3", and "1") will be displayed in the order of the hand area G3.

[0107] Furthermore, as shown in Figure 23, if one or two of the medical instruments 4 attached to arm 60a, arm 60b, and arm 60d are located outside the field of view of the endoscope 6, the marker MK1 corresponding to the medical instrument 4 located outside the field of view of the endoscope 6 will be positioned closer to the center of the screen gr of the monitor 24. In other words, if only the medical instrument 4 attached to arm 60a is located outside the left side of the field of view, the marker MK1 ("4") indicating the medical instrument 4 attached to arm 60a will be positioned closer to the center of the screen gr than the position of the marker MK1 ("4") when all three markers MK1 ("4", "3", and "1") are displayed (see Figure 20).

[0108] Similarly, if only the medical device 4 attached to arm 60a is located outside the lower part of the field of view, the marker MK1 ("4") indicating the medical device 4 attached to arm 60a will be positioned closer to the center of the screen gr than the position of marker MK1 ("4") when all three markers MK1 ("4", "3", and "1") are displayed (see Figure 20). The same applies if only the medical device 4 attached to arm 60a is located outside the upper part of the field of view (diagonally upper left, diagonally lower left).

[0109] Similarly, if only the medical device 4 attached to arm 60d is located outside the right side of the field of view, the marker MK1 ("1") indicating the medical device 4 attached to arm 60d is positioned closer to the center of the screen gr than the position of marker MK1 ("1") when all three markers MK1 ("4", "3", and "1") are displayed (see Figure 20). The same applies if only the medical device 4 attached to arm 60d is located outside the diagonally upper-right and diagonally lower-right sides of the field of view.

[0110] (Display method) Next, with reference to Figure 24, the method for displaying the graphical user interface G on the monitor 24 in the surgical system 100 will be described. The graphical user interface G is generated by the image processing device 8b.

[0111] First, in step S1, the image processing device 8b acquires the endoscopic image captured by the endoscope 6 for capturing an image of the surgical site.

[0112] Next, in step S2, the image processing device 8b determines whether a predetermined input has been received by the camera pedal 22c or the enable switch 81. If the answer in step S2 is yes, the process proceeds to step S3. If the answer in step S2 is no, the process returns to step S1.

[0113] Next, in step S3, the image processing device 8b determines whether at least one of the medical instruments 4 supported by arm 60a, the medical instrument 4 supported by arm 60b, and the medical instrument 4 supported by arm 60d is located outside the field of view of the endoscope 6. If the answer in step S3 is yes (i.e., a command has been received to allow the endoscope 6 to move, and the medical instrument 4 supported by arm 60 is located outside the field of view of the endoscope 6), the process proceeds to step S4. If the answer in step S4 is no, the process returns to step S1.

[0114] Next, in step S4, the image processing device 8b displays a graphical user interface G on the monitor 24 superimposed on the image acquired by the endoscope 6. This interface G is used to display a marker MK1 indicating a medical instrument 4 located outside the field of view in the area G11 near the outer edge of the level area G5, which includes the central part CN1 but does not include the area near the edge e of the monitor 24 screen gr.

[0115] The actions described in steps S1 to S4 above are performed continuously during the operation of the surgical system 100.

[0116] [Effects of the First Embodiment] In the first embodiment, the following effects can be obtained.

[0117] In the first embodiment, as described above, when at least one of the multiple medical instruments 4 is located outside the field of view of the endoscope 6, the image processing device 8b displays a marker MK1 indicating the medical instrument 4 located outside the field of view in the inner area of ​​the graphical user interface G, excluding the vicinity of the edge of the display screen, in response to a predetermined input from the camera pedal 22c or the enable switch 81. As a result, the marker MK1 indicating the medical instrument 4 located outside the field of view is displayed relatively towards the center of the display screen rather than near the edges. Consequently, the operator can easily see the marker MK1 indicating the medical instrument 4 located outside the field of view without moving their gaze significantly. Furthermore, since the marker MK1 indicating the medical instrument 4 located outside the field of view is displayed in response to a predetermined input from the camera pedal 22c or the enable switch 81, the marker MK1 can be hidden if no predetermined input is made to the camera pedal 22c or the enable switch 81. This prevents the marker MK1 from being displayed unnecessarily in a position easily visible to the operator.

[0118] Furthermore, in the first embodiment, as described above, the image processing device 8b is configured to display the marker MK1 on the graphical user interface G in response to a predetermined input which is an operation on the camera pedal 22c different from the camera movement operation performed on the camera pedal 22c to move the endoscope 6. This makes it possible to display the marker MK1 indicating a medical instrument 4 located outside the field of view using the camera pedal 22c which is operated when moving the endoscope 6. This suppresses an increase in the number of parts and simplifies the device configuration compared to the case where a separate dedicated input device for displaying the marker MK1 is provided. In addition, since the marker MK1 can be displayed when an operation different from the camera movement operation is performed on the camera pedal 22c, the marker MK1 can be prevented from being displayed when a camera movement operation is performed to move the endoscope 6. This prevents the display of unnecessary marker MK1 in a position easily visible to the operator when moving the endoscope 6.

[0119] Furthermore, in the first embodiment, as described above, the image processing device 8b is configured to display the indicator MK1 on the graphical user interface G in response to a predetermined input in which the camera pedal 22c is operated twice consecutively within a predetermined time, as an operation on the camera pedal 22c different from camera movement operation. This makes it easy to display the indicator MK1 indicating a medical device 4 located outside the field of view by operating the camera pedal 22c twice consecutively within a predetermined time.

[0120] Furthermore, in the first embodiment, as described above, the image processing device 8b displays the level of the endoscope 6 in the level area G5 of the graphical user interface G while the camera movement operation is being performed on the camera pedal 22c. As a result, the level LV of the endoscope 6 is displayed when the endoscope 6 is moved, allowing the operator to easily move the endoscope 6 while visually confirming the level LV.

[0121] Furthermore, in the first embodiment, as described above, the image processing device 8b is configured to display a marker MK1 in the direction in which the medical instrument 4 located outside the field of view is located within the area G11 near the outer edge of the level area G5. As a result, the marker MK1 is positioned so as to correspond to the direction in which the medical instrument 4 located outside the field of view is located relative to the central CN2 of the level area G5, so that the operator can easily recognize the direction in which the medical instrument 4 located outside the field of view is located relative to the central CN2 of the level area G5 without having to move their gaze significantly.

[0122] Furthermore, in the first embodiment, as described above, the image processing device 8b is configured to display the marker MK1 on the graphical user interface G in response to the operation of the enable switch 81 as a predetermined input. This makes it easy to display the marker MK1 indicating a medical device 4 located outside the field of view by operating the enable switch 81 provided on the manipulator.

[0123] Furthermore, in the first embodiment, as described above, the image processing device 8b is configured to overlay a graphical user interface G onto the endoscopic image captured by the endoscope 6 and display it on the display unit 8a. This allows the display unit 8a, which is viewed by the assistant assisting the operator, to show a marker MK1 indicating a medical instrument 4 located outside the field of view, in addition to the monitor 24 viewed by the operator. As a result, both the operator and the assistant can easily grasp the location of the medical instrument 4 located outside the field of view.

[0124] Furthermore, in the first embodiment, as described above, the image processing device 8b is configured to display the indicator MK1 on the graphical user interface G only when a predetermined input is received by the camera pedal 22c or the enable switch 81. This ensures that the display of unnecessary indicators MK1 without intent is prevented.

[0125] Furthermore, in the first embodiment, as described above, the label MK1 includes a number that identifies the arm 60 supporting the medical device 4 located outside the field of view. This allows the operator to easily identify and recognize the medical device 4 located outside the field of view.

[0126] Furthermore, in the first embodiment, as described above, the image processing device 8b is configured to display the indicator MK1 in the area G11 near the outer edge of the level indicator area G5 of the graphical user interface G. Furthermore, the level indicator area G5 is a rectangular region including the upper edge G5a, left edge G5b, right edge G5c, and lower edge G5d. The upper edge G5a is located between a position 1 / 10th of the vertical length of the screen above the center of the screen and a position 1 / 10th of the vertical length of the screen below the upper edge of the screen. The left edge G5b is located between a position 1 / 10th of the horizontal length of the screen to the left of the center of the screen and a position 1 / 10th of the horizontal length of the screen to the right of the left edge of the screen. The right edge G5c is located between a position 1 / 10th of the horizontal length of the screen to the right of the center of the screen and a position 1 / 10th of the horizontal length of the screen to the left of the right edge of the screen. The lower edge G5d is located between a position 1 / 10th of the vertical length of the screen below the center of the screen and a position 1 / 10th of the vertical length of the screen above the lower edge of the screen. As a result, the indicator MK1 displayed in area G11 near the outer edge of the level indicator area G5 is positioned closer to the center than the edge e of the screen gr. This improves the visibility of the indicator MK1, unlike when it is positioned at the edge e of the screen gr.

[0127] [Second Embodiment] Referring to Figures 25 to 27, the configuration of the surgical system 200 according to the second embodiment will be described.

[0128] In the second embodiment, as shown in Figures 25 and 26, the marker MK2 includes an arrow MK2a indicating the direction in which a medical device 4 located outside the field of view is located. Specifically, a number (for example, "4") is displayed near the outside of the arrow MK2a to identify the medical device 4 located outside the field of view. The number may also be displayed inside the arrow MK2a, as shown in the marker MK3 in Figure 27. The inside of the arrow MK2a is displayed as transparent or semi-transparent so that the image is visible through it. For a medical device 4 to be operated, the number on the marker MK2 is shown as the number of the black arm 60 (see "4") inside a white circle. For a medical device 4 not to be operated, the number on the marker MK2 is shown as the number of the gray arm 60 (see "3") inside a black circle. The arrow MK2a is shown in a planar view.

[0129] [Effects of the second embodiment] In the second embodiment, the following effects can be obtained.

[0130] In the second embodiment, as described above, the marker MK2 includes an arrow MK2a indicating the direction in which the medical device 4 located outside the field of view is located. This allows the operator to intuitively recognize the direction in which the medical device 4 located outside the field of view is located.

[0131] [Differentiation] It should be noted that the embodiments disclosed herein are illustrative and not restrictive in all respects. The scope of the present invention is defined by the claims rather than by the description of the embodiments above, and further includes all modifications (exceptions) within the meaning and scope equivalent to the claims.

[0132] For example, in the first and second embodiments described above, an example was shown in which the image processing device 8b acquires an image from the endoscope 6 and generates a graphical user interface G, but the present invention is not limited thereto. For example, the control unit 31 of the medical manipulator 1 may generate the graphical user interface G, or the control unit (not shown) of the remote control device 2 may generate the graphical user interface G. Furthermore, an image processing device that acquires an image from the endoscope 6 and performs image processing, and an image processing device that generates a graphical user interface G and overlays it on the image from the endoscope 6 may be provided separately.

[0133] Furthermore, while the first and second embodiments described above show examples in which the image processing device 8b determines whether or not input has been received from the camera pedal 22c or the enable switch 81, the present invention is not limited thereto. For example, the control unit 31 may determine whether or not input has been received from the camera pedal 22c or the enable switch 81 and transmit the determination result to the image processing device 8b.

[0134] Furthermore, while the first and second embodiments described above show examples in which the image processing device 8b determines whether or not at least one of the medical instruments 4 supported by arm 60a, the medical instruments 4 supported by arm 60b, and the medical instruments 4 supported by arm 60d is located outside the field of view of the endoscope 6, the present invention is not limited thereto. For example, the control unit 31 may determine whether or not the medical instrument is located outside the field of view of the endoscope and transmit the determination result to the image processing device 8b.

[0135] Furthermore, while the first and second embodiments described above show an example in which the marker MK1 is displayed in area G12 corresponding to the direction in which the medical device 4 outside the field of view is located relative to the center CN2 of the level area G5, the present invention is not limited thereto. For example, the marker MK1 may be displayed at a fixed position in area G11 near the outer edge of the level area G5.

[0136] Furthermore, in the first and second embodiments described above, the area G12 on which the indicator MK1 is displayed is shown to be one of eight areas G12 obtained by dividing the area G11 near the outer edge into eight radial areas G12 from the center CN2 of the level area G5. However, the present invention is not limited to this. For example, the number of areas G12 to be divided may be other than eight.

[0137] Furthermore, while the first and second embodiments described above show examples in which the indicator MK1 is displayed on the graphical user interface G in response to a predetermined input in which the camera pedal 22c is operated twice consecutively within a predetermined time, the present invention is not limited thereto. For example, the indicator MK1 may be displayed on the graphical user interface G in response to a predetermined input in which the camera pedal 22c is operated three or more times consecutively within a predetermined time.

[0138] Furthermore, while the first and second embodiments described above show examples of configurations in which an enable switch 81, a joystick 82, and a switch unit 83 are provided on multiple arms 60, the present invention is not limited thereto. For example, an enable switch 81, a joystick 82, and a switch unit 83 may be provided on at least one of the multiple arms 60.

[0139] Furthermore, while the first and second embodiments described above show examples in which the indicator MK1 is displayed on the graphical user interface G in response to the operation of the enable switch 81 of the arm 60, the present invention is not limited thereto. For example, the indicator MK1 may be displayed on the graphical user interface G in response to the operation of other switches of the arm 60 (joystick 82 and switch unit 83).

[0140] Furthermore, while the first and second embodiments described above show examples where the indicator MK1 is displayed on the graphical user interface G only when a predetermined input is made by the camera pedal 22c or the enable switch 81, the present invention is not limited thereto. For example, the indicator MK1 may also be displayed on the graphical user interface G when a predetermined input is made by another input device.

[0141] Furthermore, while the first and second embodiments described above show examples where the indicator MK1 is displayed outside the level area G5, the present invention is not limited thereto. For example, the indicator MK1 may be displayed inside the level area G5.

[0142] Furthermore, while the first and second embodiments described above show examples where the level indicator area G5 is a rectangular region, the present invention is not limited to this. For example, the level indicator area G5 may be a region with a shape other than a rectangle (such as an oval shape).

[0143] Furthermore, while the first and second embodiments described above show examples in which the camera area G2 and the hand area G3 (hand area G3a, hand area G3b, and hand area G3c) are displayed in the region near the lower edge ed of the screen gr of the monitor 24, the present invention is not limited thereto. For example, the camera area G2 and the hand area G3 (hand area G3a, hand area G3b, and hand area G3c) may be displayed in the region near the upper edge eu of the screen gr of the monitor 24. Specifically, the camera area G2 and the hand area G3 may be displayed between the upper edge eu of the screen gr and a position below the upper edge eu by a length of 1 / 10 (L11) of the vertical length of the screen gr.

[0144] Furthermore, in the first and second embodiments described above, when multiple medical instruments 4 are located outside the field of view of the endoscope 6, examples were shown in which multiple markers MK1 are displayed in the order of the positions of the arms 60 that support each of the multiple medical instruments 4. However, the present invention is not limited to this. For example, the order of the positions of the arms 60 that support each of the multiple medical instruments 4 may be different from the order in which the multiple markers MK1 are displayed.

[0145] Furthermore, while the first and second embodiments described above show examples where the marker MK1 is positioned towards the center of the screen gr when one or two medical instruments 4 are located outside the field of view of the endoscope 6, the present invention is not limited thereto. For example, the position where the marker MK1 corresponding to each of the multiple medical instruments 4 is positioned may be fixed.

[0146] Furthermore, while the first and second embodiments described above show examples where the display of the marker MK1 differs depending on whether the medical device 4 is the object of manipulation or not, the present invention is not limited thereto. For example, the display of the marker MK1 may be the same whether the medical device 4 is the object of manipulation or not.

[0147] Furthermore, although the first and second embodiments described above show examples in which four arms 60 are provided, the present invention is not limited thereto. In the present invention, the number of arms 60 may be any other number as long as at least one or more are provided.

[0148] Furthermore, while the first and second embodiments described above show examples in which the arm portion 61 and positioner 40 are composed of a 7-axis articulated robot, the present invention is not limited thereto. For example, the arm portion 61 and positioner 40 may be composed of an articulated robot with an axis configuration other than a 7-axis articulated robot (e.g., 6-axis or 8-axis).

[0149] Furthermore, while the first and second embodiments described above show examples in which the medical manipulator 1 includes a medical trolley 3, a positioner 40, and an arm base 50, the present invention is not limited to this. For example, the medical trolley 3, the positioner 40, and the arm base 50 are not necessarily required, and the medical manipulator 1 may consist only of an arm 60.

[0150] Furthermore, while the first embodiment described above shows an example where the number of arm 60 (one of 1 to 4) is displayed within a black square, the present invention is not limited to this. For example, as shown in the indicator MK4 of Figure 28, the outer edge e1 of the black square may be surrounded by another color (such as red). Alternatively, as shown in the indicator MK5 of Figure 29(a), the black square may not be displayed, and only the number of arm 60 may be displayed within a white oval (if it is the target of operation), or as shown in the indicator MK5 of Figure 29(b), only the number of arm 60 in gray may be displayed (if it is not the target of operation).

[0151] Furthermore, although the second embodiment described above shows an example where the arrow MK2a is shown in a planar manner, the present invention is not limited to this. For example, the arrow MK2a may be shown in three dimensions. This allows the operator to distinguish (recognize) whether the medical instrument 4 located outside the field of view of the endoscope 6 is located closer to the operator or further away from the operator.

[0152] Furthermore, while the first and second embodiments described above show examples in which the level indicator area G5, where the level indicator LV is displayed, is applied as the center area of ​​the present invention, the present invention is not limited thereto. In the present invention, an area other than the level indicator area G5, where the level indicator LV is displayed, may be applied as the center area of ​​the present invention.

[0153] The functions of the elements disclosed herein can be performed using circuits or processing circuits, including general-purpose processors, dedicated processors, integrated circuits, ASICs (Application Specific Integrated Circuits), conventional circuits, and / or combinations thereof, configured or programmed to perform the disclosed functions. A processor is considered a processing circuit or circuit because it includes transistors and other circuits. In this disclosure, a circuit, unit, or means is hardware that performs the enumerated functions, or hardware programmed to perform the enumerated functions. The hardware may be hardware disclosed herein, or other known hardware that is programmed or configured to perform the enumerated functions. If the hardware is a processor, which is considered a type of circuit, then the circuit, means, or unit is a combination of hardware and software, and the software is used to configure the hardware and / or the processor. [Explanation of Symbols]

[0154] 2 Remote control device 4. Medical Instruments (Surgical Instruments) 6 Endoscope 8a Display unit (second display device) 8b Image processing device (control device) 21, 21L, 21R Operating handle 22c Camera Pedal (Input Device) 24 Monitor (First display device) 60 Arms (Manipulators) 81 Enable switch (input device, switch) 82. Joystick (operating component) 83 Switch section (operating component) 100 Surgical Operating Systems (Surgical Systems) G Graphical User Interface G5 Leveling Instrument Area (Center Area) G5a Top edge G5b Left side G5c Right-hand side G5d lower edge G11 Outer edge vicinity area (outer edge vicinity region) LV spirit level MK1, MK2, MK3, MK4 MK5 label

Claims

1. a plurality of manipulators for respectively supporting an endoscope and a plurality of surgical instruments; a first display device for displaying an endoscopic image captured by the endoscope, and a remote control device including two operation handles for respectively operating two of the plurality of surgical instruments; a control device configured to display a graphical user interface on a display unit of the first display device by superimposing the graphical user interface on the endoscopic image captured by the endoscope; Among the plurality of manipulators, the manipulator supporting the endoscope includes an input device for an operator to input permission to move the endoscope, A surgical system, wherein when at least one of the plurality of surgical instruments is located outside the field of view of the endoscope, the control device is configured to display, in response to an input by the input device allowing movement of the endoscope, an indication indicating the surgical instrument located outside the field of view in an inner area of ​​the graphical user interface that does not include the edges of the screen of the display unit.

2. the input device includes a switch for performing an input for permitting movement of the endoscope, The surgical system of claim 1 , wherein the control device is configured to display the indicator on the graphical user interface while the switch is operated as an input to allow movement of the endoscope.

3. The manipulator supporting the endoscope includes an operating member for operating the manipulator supporting the endoscope to move the endoscope, and an enable switch for enabling operation of the manipulator supporting the endoscope by the operating member; the switch for inputting permission to move the endoscope is the enable switch, The surgical system of claim 2 , wherein the control device is configured to display the indicator on the graphical user interface while the enable switch is operated as an input to permit movement of the endoscope.

4. Further comprising a second display device provided separately from the first display device; The surgical system according to any one of claims 1 to 3, wherein the control device is configured to superimpose the graphical user interface on the endoscopic image captured by the endoscope and display it on the second display device.

5. The remote control device includes a camera pedal for causing the two operating handles to move the endoscope, The surgical system according to any one of claims 1 to 3, wherein the control device displays a spirit level for the endoscope in a center area of ​​the graphical user interface while a camera movement operation is being performed on the camera pedal to execute movement of the endoscope.

6. The surgical system of claim 5 , wherein the control device is configured to display the marker within an outer edge region of the center area.

7. The manipulator supporting the endoscope includes the operating member and a plurality of the switches including the enable switch, The surgical system of claim 3, wherein the control device is configured to display the indicator on the graphical user interface only when an input is made to permit movement of the endoscope by the operating member and the enable switch among the plurality of switches.

8. The surgical system according to any one of claims 1 to 3, wherein the markings include a number identifying the manipulator supporting the surgical instrument located outside the field of view.

9. The surgical system according to any one of claims 1 to 3, wherein the marking includes an arrow indicating the direction in which the surgical instrument located outside the field of view is located.

10. The surgical system according to any one of claims 1 to 3, wherein the control device is configured to determine whether the multiple surgical instruments are located outside the field of view of the endoscope based on imaging range information of the endoscope and positional information of the tips of each of the multiple surgical instruments supported by the multiple manipulators, respectively.

11. the control device is configured to display the indicia within an outer edge region of a center area of ​​the graphical user interface; the center area is a rectangular area including an upper side, a left side, a right side, and a lower side, the upper side portion is disposed between a position that is 1 / 10 of the vertical length of the screen from the center of the screen and a position that is 1 / 10 of the vertical length of the screen from the upper end of the screen and a position that is 1 / 10 of the vertical length of the screen; the left side portion is disposed between a position that is 1 / 10 of the horizontal length of the screen from the center of the screen to the left and a position that is 1 / 10 of the horizontal length of the screen from the left end of the screen to the right, the right side portion is disposed between a position on the right side of the center of the screen by a length of 1 / 10 of the horizontal length of the screen and a position on the left side of the right edge of the screen by a length of 1 / 10 of the horizontal length of the screen, The surgical system according to any one of claims 1 to 3, wherein the lower edge portion is positioned between a position 1 / 10 of the vertical length of the screen below the center of the screen and a position 1 / 10 of the vertical length of the screen above the lower end of the screen.

12. A display method executed by a control device provided in a surgical system having an endoscope and a plurality of manipulators for supporting a plurality of surgical instruments, comprising: acquiring an endoscopic image captured by the endoscope; determining whether the plurality of surgical instruments are located outside the field of view of the endoscope; determining whether an input for permitting movement of the endoscope has been made by an input device provided on a manipulator that supports the endoscope among the plurality of manipulators; and when at least one of the plurality of surgical instruments is located outside the field of view of the endoscope and an input to allow movement of the endoscope is made by the input device, displaying on the display unit a graphical user interface including a mark indicating the surgical instrument located outside the field of view in an inner area not including the edges of the screen of the display unit, superimposed on the endoscopic image captured by the endoscope.

13. In a surgical system including an endoscope and a plurality of manipulators for supporting each of the plurality of surgical instruments, a program for causing a computer to execute display of information relating to the plurality of surgical instruments, comprising: acquiring an endoscopic image captured by the endoscope; determining whether the plurality of surgical instruments are located outside the field of view of the endoscope; determining whether an input for permitting movement of the endoscope has been made by an input device provided on a manipulator that supports the endoscope among the plurality of manipulators; When at least one of the plurality of surgical instruments is located outside the field of view of the endoscope and an input to allow movement of the endoscope is made via the input device, a graphical user interface including a mark indicating the surgical instrument located outside the field of view in an inner area excluding the edges of the screen of the display unit is displayed on the display unit superimposed on the endoscopic image captured by the endoscope.

Images