Surgery system and display method

Abstract

To provide a surgery system which allows an operator to easily visually recognize a mark indicating a surgical instrument located outside a visual field without largely moving the sight line.SOLUTION: In a surgical system 100 (surgery system), an image processing device 8, when receiving a command for enabling movement of an endoscope 6 and a medical instrument 4 is located outside a visual field of the endoscope 6, is configured to display a graphical user interface G for displaying a mark MK1 indicating the medical instrument 4 located outside the visual field in an outer edge vicinity area G11 of a level area G5 including a center part CN1 without including the vicinity of an end e of a screen gr of a monitor 24 on the monitor 24 so as to be overlapped on an image captured by the endoscope 6.SELECTED DRAWING: Figure 13

Description

[Technical field]

[0001] The present invention relates to a surgical system and a display method, and more particularly to a surgical system and a display method for displaying a graphical user interface on a display device by superimposing the graphical user interface on an image captured by an endoscope. [Background technology]

[0002] 2. Description of the Related Art Conventionally, a robotic surgery system is known that displays a graphical user interface by superimposing it on an image captured by an endoscope (see, for example, Patent Document 1).

[0003] In the robotic surgery system of Patent Document 1, an endoscope captures an image of the surgical site for display in a display area of ​​a monitor. In addition, in this robotic surgery system, a current position of a tool is determined relative to the image (field of view) captured by the endoscope. If the determined current position of the tool is outside the image (field of view) displayed on the monitor, the current position of the tool is displayed as a symbol in a boundary area outside the display area of ​​the monitor screen. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2013-188574 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, in a robotic surgery system such as that described in Patent Document 1, the operator focuses on the center of the display area of ​​the monitor during surgery. Therefore, even if the current position of a tool (surgical instrument) that is outside the field of view is displayed as a symbol (sign) in a boundary area outside the display area of ​​the monitor screen as in Patent Document 1, the operator may not notice it. In addition, if the operator notices it, he or she must move his or her line of sight significantly.

[0006] The present invention has been made to solve the problems described above, and one object of the present invention is to provide a surgical system and display method that enables an operator to easily see signs indicating surgical instruments located outside the field of view without having to move their line of sight significantly. [Means for solving the problem]

[0007] A surgical system according to a first aspect of the present invention includes an endoscope that captures an image of a surgical site, a plurality of manipulators for supporting a first surgical instrument and a second surgical instrument, a display device for displaying an image captured by the endoscope, a remote control device including a first operating handle for operating the first surgical instrument and a second operating handle for operating the second surgical instrument, and a control device, wherein when the control device receives a command to enable movement of the endoscope and when at least one of the first surgical instrument and the second surgical instrument is located outside the field of view of the endoscope, the control device displays on a screen of the display device a sign indicating at least one of the first surgical instrument and the second surgical instrument that is located outside the field of view. No Outside the center area Edge A graphical user interface for displaying on the display device is configured to be superimposed on an image captured by the endoscope. The center area is a rectangular region including a top edge, a left edge, a right edge, and a bottom edge, the top edge being disposed between a position 1 / 10 of the vertical length of the screen from the center of the screen and a position 1 / 10 of the vertical length of the screen from the upper end of the screen, the left edge being disposed between a position 1 / 10 of the horizontal length of the screen from the center of the screen to the left and a position 1 / 10 of the horizontal length of the screen from the left end of the screen, the right edge being disposed between a position 1 / 10 of the horizontal length of the screen from the center of the screen to the right and a position 1 / 10 of the horizontal length of the screen from the right end of the screen, and the bottom edge being disposed between a position 1 / 10 of the vertical length of the screen from the center of the screen and a position 1 / 10 of the vertical length of the screen from the lower end of the screen. .

[0008] In the surgical system according to the first aspect of the present invention, as described above, the control device displays on the screen of the display device a sign indicating at least one of the first surgical instrument and the second surgical instrument that is located outside the field of view. No Outside the center area EdgeThe display device is configured to display a graphical user interface for displaying the graphical user interface on the display device in a manner superimposed on an image captured by the endoscope. As a result, a sign indicating at least one of the first surgical instrument and the second surgical instrument located outside the field of view is displayed on the screen of the display device. Club As a result, the operator can easily see the signs indicating the surgical instruments located outside the field of view without having to move his / her line of sight significantly.

[0009] A surgical system according to a second aspect of the present invention includes a first manipulator for supporting a first surgical instrument, a second manipulator for supporting a second surgical instrument, a third manipulator for supporting an endoscope that captures an image of a surgical site, a display device for displaying an image captured by the endoscope, a remote operation device including an operation pedal including a first operation handle for operating the first surgical instrument, a second operation handle for operating the second surgical instrument, and an endoscope pedal for executing movement of the endoscope by operating the first operation handle and the second operation handle, and a control device, wherein when the endoscope pedal is operated and at least one of the first surgical instrument and the second surgical instrument is located outside the field of view of the endoscope, the control device displays on a screen of the display device a sign indicating at least one of the first surgical instrument and the second surgical instrument that is located outside the field of view. No Outside the center area Edge A graphical user interface for displaying on the display device is configured to be superimposed on an image captured by the endoscope. The center area is a rectangular region including a top edge, a left edge, a right edge, and a bottom edge, the top edge being disposed between a position 1 / 10 of the vertical length of the screen from the center of the screen and a position 1 / 10 of the vertical length of the screen from the upper end of the screen, the left edge being disposed between a position 1 / 10 of the horizontal length of the screen from the center of the screen to the left and a position 1 / 10 of the horizontal length of the screen from the left end of the screen, the right edge being disposed between a position 1 / 10 of the horizontal length of the screen from the center of the screen to the right and a position 1 / 10 of the horizontal length of the screen from the right end of the screen, and the bottom edge being disposed between a position 1 / 10 of the vertical length of the screen from the center of the screen and a position 1 / 10 of the vertical length of the screen from the lower end of the screen. .

[0010] In the surgical system according to the second aspect of the present invention, as described above, the control device displays on the screen of the display device a sign indicating at least one of the first surgical instrument and the second surgical instrument that is located outside the field of view. No Outside the center area EdgeThe display device is configured to display a graphical user interface for displaying the graphical user interface on the display device in a manner superimposed on an image captured by the endoscope. As a result, a sign indicating at least one of the first surgical instrument and the second surgical instrument located outside the field of view is displayed on the screen of the display device. Club As a result, it is possible to provide a surgical system that allows the operator to easily view the signs indicating the surgical instruments positioned outside the field of view without having to move the operator's line of sight significantly.

[0011] A display method according to a third aspect of the present invention includes the steps of acquiring an image captured by an endoscope for capturing an image of a surgical site, and generating a graphical user interface and displaying the graphical user interface on a display device by overlaying it on the image captured by the endoscope. The step of displaying the graphical user interface includes, when a command is received to enable movement of the endoscope and when at least one of a first surgical instrument supported by a first manipulator and a second surgical instrument supported by a second manipulator is located outside the field of view of the endoscope, displaying a sign indicating at least one of the first surgical instrument and the second surgical instrument located outside the field of view on a screen of the display device. No Outside the center area Edge and displaying, on a display device, a graphical user interface for displaying on the display device the graphical user interface for displaying on the display device in a manner superimposed on an image captured by the endoscope. The center area is a rectangular region including a top edge, a left edge, a right edge, and a bottom edge, and the top edge is disposed between a position 1 / 10 of the vertical length of the screen from the center of the screen and a position 1 / 10 of the vertical length of the screen from the upper end of the screen, the left edge is disposed between a position 1 / 10 of the horizontal length of the screen from the center of the screen to the left and a position 1 / 10 of the horizontal length of the screen from the left end of the screen, the right edge is disposed between a position 1 / 10 of the horizontal length of the screen from the center of the screen to the right and a position 1 / 10 of the horizontal length of the screen from the right end of the screen, and the bottom edge is disposed between a position 1 / 10 of the vertical length of the screen from the center of the screen and a position 1 / 10 of the vertical length of the screen from the lower end of the screen. .

[0012] In the display method according to the third aspect of the present invention, as described above, the step of displaying the graphical user interface includes displaying on a screen of a display device a sign indicating at least one of the first surgical instrument and the second surgical instrument that is located outside the field of view. No Outside the center area Edge The method includes a step of displaying, on a display device, a graphical user interface for displaying the graphical user interface on the display device in a manner superimposed on an image captured by the endoscope, whereby a mark indicating at least one of the first surgical instrument and the second surgical instrument located outside the field of view is displayed at an edge on the screen of the display device. ClubAs a result, a display method can be provided that allows the operator to easily view the signs indicating the surgical instruments positioned outside the field of view without having to move the operator's line of sight significantly. Effect of the Invention

[0013] According to the present invention, an operator can easily visually recognize a sign indicating a surgical instrument located outside the field of view without significantly moving his / her line of sight. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram showing a configuration of a surgical operation system according to a first embodiment. [Diagram 2] 1A and 1B are diagrams showing a configuration of a medical manipulator according to a first embodiment. [Diagram 3] FIG. 2 is a diagram showing a configuration of an operating handle according to the first embodiment. [Figure 4] 1A and 1B are diagrams illustrating a configuration of a foot pedal according to a first embodiment. [Diagram 5] 1A and 1B are diagrams showing a configuration of an arm of a medical manipulator according to a first embodiment. [Figure 6] FIG. [Figure 7] FIG. 2 is a perspective view showing a configuration of an operation unit of the medical manipulator according to the first embodiment. [Figure 8] FIG. [Figure 9] FIG. 13 is a diagram showing a pivot position teaching tool. [Figure 10] FIG. 13 is a diagram for explaining translational movement of the arm. [Figure 11] FIG. 13 is a diagram for explaining the rotational movement of the arm. [Figure 12] 3 is a block diagram showing the configuration of a control unit of the medical manipulator according to the first embodiment. FIG. [Figure 13] FIG. 2 shows an image captured by an endoscope and a graphical user interface. [Figure 14]FIG. 2 illustrates an area of ​​a graphical user interface. [Figure 15] FIG. 13 is a diagram for explaining the display of a clutch area. [Figure 16] FIG. 13 is a diagram for explaining the display of a medical instrument status information area. [Figure 17] 13 is a diagram for explaining the display of a left popup area. FIG. [Figure 18] FIG. 13 is a diagram for explaining the display of a right popup area. [Figure 19] FIG. 2 is a diagram for explaining a display on a touch panel of a remote control device. [Figure 20] FIG. 2 is a diagram for explaining a sign according to the first embodiment. [Figure 21] FIG. 2 is a diagram for explaining a sign (operation object) according to the first embodiment. [Figure 22] FIG. 2 is a diagram for explaining a marker (non-operation target) according to the first embodiment. [Diagram 23] FIG. 2 is a diagram for explaining markers arranged toward the center according to the first embodiment. [Figure 24] FIG. 4 is a flow diagram for explaining a display method according to the first embodiment. [Diagram 25] FIG. 1 is a diagram for explaining a sign according to a second embodiment. [Figure 26] FIG. 11 is a diagram (2) for explaining a sign according to the second embodiment. [Figure 27] FIG. 11 is a diagram (3) for explaining a sign according to the second embodiment. [Figure 28] FIG. 1 is a diagram (1) for explaining a sign according to a first modified example. [Figure 29] FIG. 2 is a diagram for explaining a sign according to a first modified example. [Diagram 30] FIG. 1 is a diagram for explaining a sign according to a second modified example. [Diagram 31] FIG. 11 is a diagram (2) for explaining a sign according to a second modified example. [Diagram 32] FIG. 13 is a diagram for explaining a sign according to a third modified example. [Diagram 33] FIG. 13 is a diagram for explaining a sign according to a fourth modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings.

[0016] [First embodiment] A configuration of a surgical operation system 100 according to a first embodiment will be described with reference to Figs. 1 to 24. The surgical operation system 100 includes a medical manipulator 1, which is a patient P-side device, and a remote control device 2, which is an operator-side device for operating the medical manipulator 1. The medical manipulator 1 includes a medical cart 3 and is configured to be movable. The remote control device 2 is disposed at a position separated from the medical manipulator 1, and the medical manipulator 1 is configured to be remotely operated by the remote control device 2. An operator (such as a doctor) inputs a command to the remote control device 2 to cause the medical manipulator 1 to perform a desired operation. The remote control device 2 transmits the input command to the medical manipulator 1. The medical manipulator 1 operates based on the received command. The medical manipulator 1 is disposed in an operating room, which is a sterilized sterile field. The surgical operation system 100 is an example of a "surgery system" in the claims.

[0017] The remote control device 2 is placed, for example, inside or outside an operating room. The remote control device 2 includes an operating handle 21, a foot pedal 22, a touch panel 23, a monitor 24, a support arm 25, and a support bar 26. The operating handle 21 constitutes an operating handle for an operator (such as a doctor) to input commands. The monitor 24 is an example of a "display device" in the claims.

[0018] The operating handle 21 is configured to operate the medical instrument 4. The operating handle 21 also receives an operating amount for the medical instrument 4. The operating handle 21 includes an operating handle 21L arranged on the left side as viewed from an operator (such as a doctor) and operated by the operator's left hand, and an operating handle 21R arranged on the right side and operated by the operator's right hand. The operating handle 21L and the operating handle 21R are examples of a "first operating handle" and a "second operating handle" in the claims, respectively.

[0019] As shown in FIG. 3, the operating handle 21 includes a link portion 21a, a link portion 21b, a link portion 21c, and a link portion 21d operated by an operator (such as a doctor). The link portion 21a rotates around the A4 axis. When the link portion 21a rotates around the A4 axis, an arm portion 61 described later rotates around the JT4 axis. The link portion 21b rotates around the A5 axis relative to the link portion 21a. When the link portion 21b rotates around the A5 axis, the arm portion 61 described later rotates around the JT5 axis. The link portion 21c rotates around the A6 axis relative to the link portion 21b. When the link portion 21c rotates around the A6 axis, the arm portion 61 rotates around the JT6 axis. The link portion 21d rotates around the A7 axis relative to the link portion 21c. By rotating the link portion 21d around the A7 axis, the arm portion 61 rotates around the JT7 axis. The medical instrument 4 is an example of a "surgical instrument," a "first surgical instrument," and a "second surgical instrument" in the claims. The operating handles 21L and 21R are examples of a "first operating handle" and a "second operating handle," respectively, in the claims.

[0020] Furthermore, the operating handle 21 changes (scales) the amount of movement of the arm 60 (medical instrument 4) according to the amount of operation received by the operating handle 21. For example, when the magnification of the amount of movement is set to 1 / 2, the medical instrument 4 is controlled to move a distance that is 1 / 2 the movement distance of the operating handle 21. This makes it possible to perform delicate surgery with precision. The arm 60 is an example of a "manipulator" in the claims.

[0021] As shown in FIG. 4, a plurality of foot pedals 22 are provided to execute functions related to the medical instrument 4. The plurality of foot pedals 22 are arranged on the base 28. The foot pedals 22 include a switching pedal 22a, a clutch pedal 22b, a camera pedal 22c, an incision pedal 22d, and a coagulation pedal 22e. The switching pedal 22a, the clutch pedal 22b, the camera pedal 22c, the incision pedal 22d, and the coagulation pedal 22e are operated by the operator's foot. The incision pedal 22d includes an incision pedal 22dR for the right arm 60 and an incision pedal 22dL for the left arm 60. The coagulation pedal 22e includes a coagulation pedal 22eR for the right arm 60 and a coagulation pedal 22eL for the left arm 60. The foot pedal 22 is an example of an "operation pedal" in the claims. Moreover, the camera pedal 22c is an example of the "input unit" and "endoscope pedal" in the claims.

[0022] The switching pedal 22a is configured to switch the arm 60 operated by the operating handle 21. In the first embodiment, the clutch pedal 22b is configured to execute a clutch operation that temporarily disconnects the operational connection between the arm 60 and the operating handle 21. While the clutch pedal 22b is being depressed by the operator, the operation by the operating handle 21 is not transmitted to the arm 60.

[0023] Moreover, in the first embodiment, the camera pedal 22c is provided for inputting a command to move the endoscope 6. Specifically, the camera pedal 22c is depressed by the operator, thereby inputting a command to move the endoscope 6. Moreover, when a command enabling the movement of the endoscope 6 is input by the camera pedal 22c (that is, while the camera pedal 22c is being depressed by the operator), the endoscope 6 is configured to be moved by operating both the operating handle 21R and the operating handle 21L.

[0024] Furthermore, while the operator is depressing the incision pedal 22d (coagulation pedal 22e), an electrosurgical device (not shown) is activated.

[0025] As shown in FIG. 1, the monitor 24 is a scope-type display device for displaying an image (see FIG. 13) captured by the endoscope 6. The support arm 25 supports the monitor 24 so that the height of the monitor 24 matches the height of the face of an operator (such as a doctor). The touch panel 23 is disposed on a support bar 26. The medical manipulator 1 can be operated by the remote control device 2 when a sensor (not shown) provided near the monitor 24 detects the head of the operator. The operator operates the operation handle 21 and the foot pedal 22 while visually checking the affected area on the monitor 24. This inputs a command to the remote control device 2. The command input to the remote control device 2 is transmitted to the medical manipulator 1.

[0026] The medical cart 3 is provided with a control unit 31 that controls the operation of the medical manipulator 1, and a storage unit 32 that stores programs for controlling the operation of the medical manipulator 1. Based on a command input to the remote control device 2, the control unit 31 of the medical cart 3 controls the operation of the medical manipulator 1.

[0027] The medical cart 3 is also provided with an input device 33. The input device 33 is configured to receive operations for moving and changing the posture of the positioner 40, the arm base 50, and the multiple arms 60, mainly for preparation of surgery before the treatment.

[0028] The medical manipulator 1 shown in FIG. 1 and FIG. 2 is placed in an operating room. The medical manipulator 1 includes a medical cart 3, a positioner 40, an arm base 50, and a plurality of arms 60. The arm base 50 is attached to the tip of the positioner 40. The arm base 50 has a relatively long rod shape (long shape). The base of each of the arms 60 is attached to the arm base 50. The arms 60 are configured to be capable of taking a folded position (storage position). The arm base 50 and the arms 60 are used covered with a sterile drape (not shown). The arm 60 supports a medical instrument 4. The medical instrument 4 is an example of a "surgical instrument" in the claims.

[0029] The positioner 40 is, for example, a seven-axis articulated robot. The positioner 40 is disposed on the medical cart 3. The positioner 40 moves the arm base 50. Specifically, the positioner 40 is configured to move the position of the arm base 50 in three dimensions.

[0030] The positioner 40 also includes a base portion 41 and a plurality of link portions 42 connected to the base portion 41. The plurality of link portions 42 are connected to each other by joint portions 43.

[0031] 1, a medical instrument 4 is attached to the tip of each of the multiple arms 60. The medical instrument 4 includes, for example, replaceable instruments, an endoscope 6 (see FIG. 8) for capturing an image of a surgical site, and the like.

[0032] 5, the instrument is provided with a driven unit 4a that is driven by a servo motor M2 provided on a holder 71 of an arm 60. In addition, a forceps 4b is provided at the tip of the instrument.

[0033] As shown in FIG. 6, the instrument includes a first support 4e that supports the base end side of the end effector members 104a and 104b rotatably around the JT11 axis at the tip side, a second support 4f that supports the base end side of the first support 4e rotatably around the JT10 axis at the tip side, and a shaft 4c connected to the base end side of the second support 4f. The driven unit 4a, the shaft 4c, the second support 4f, the first support 4e, and the forceps 4b are arranged along the Z direction. The JT11 axis is perpendicular to the direction in which the shaft 4c extends (Z direction). The JT10 axis is separated from the JT11 axis in the direction in which the shaft 4c extends and perpendicular to the direction in which the shaft 4c extends and the JT11 axis.

[0034] The forceps 4b are attached to the first support 4e so as to rotate about the axis of the JT11 shaft. The second support 4f supports the first support 4e rotatably about the JT10 shaft. That is, the first support 4e is attached to the second support 4f so as to rotate about the axis of the JT10 shaft. The tip side (Z1 direction side) of the first support 4e has a U-shape. A tool center point (TCP1, clevis) is set in the center of the axis of the JT11 shaft of the tip side of the U-shape of the first support 4e.

[0035] The medical instrument 4 (forceps 4b) also includes a JT9 axis as a rotation axis of the shaft 4c (axis along the extension direction of the shaft 4c) and a JT12 axis as an opening / closing axis of the forceps 4b. A plurality of (for example, four) servo motors M2 are provided on the holder 71 of the arm 60, and the rotating body of the driven unit 4a is driven by the plurality of servo motors M2. As a result, the medical instrument 4 is driven around the J9 to J12 axes.

[0036] As shown in FIG. 8, the TCP2 of the endoscope 6 is set at the tip of the endoscope 6.

[0037] Next, the configuration of the arm 60 will be described in detail.

[0038] 5, the arm 60 includes an arm section 61 (a base section 62, a link section 63, a joint section 64) and a translation mechanism section 70 provided at the tip of the arm section 61. The arm 60 is configured to move the tip side relative to the base side (arm base 50) of the arm 60 in three dimensions. The arm section 61 is also configured as a seven-axis articulated robot arm. The multiple arms 60 have similar configurations.

[0039] As shown in Fig. 5, the arm 60 has axes JT1 to JT7 as rotation axes and axis J8 as a linear motion axis. The axes JT1 to JT7 correspond to the rotation axis of the joint 64 of the arm section 61. The JT7 axis corresponds to the base end link section 72 of the translational movement mechanism 70. The JT8 axis corresponds to an axis that moves the tip end link section 73 of the translational movement mechanism 70 relatively to the base end link section 72 in the Z direction. That is, the servo motor M1 shown in Fig. 12 is provided to correspond to the JT1 to JT7 axes of the arm 60. The servo motor M3 is provided to correspond to the JT8 axis.

[0040] The translational movement mechanism 70 is provided on the tip side of the arm unit 61, and has the medical instrument 4 attached thereto. The translational movement mechanism 70 translates the medical instrument 4 in a direction in which the medical instrument 4 is inserted into the patient P. The translational movement mechanism 70 is configured to translate the medical instrument 4 relative to the arm unit 61. Specifically, the translational movement mechanism 70 is provided with a holder 71 that holds the medical instrument 4. The holder 71 houses a servo motor M2 (see FIG. 12).

[0041] 7, the medical manipulator 1 is attached to the arm 60 and includes an operation unit 80 for operating the arm 60. The operation unit 80 includes an enable switch 81, a joystick 82, and a switch unit 83. The enable switch 81 permits or prohibits movement of the arm 60 by the joystick 82 and the switch unit 83. When an operator (such as a nurse or an assistant) holds the operation unit 80 and presses it, the enable switch 81 enters a state that permits movement of the medical instrument 4 by the arm 60.

[0042] The switch unit 83 includes a switch unit 83a that moves the medical instrument 4 in the direction along the longitudinal direction of the medical instrument 4 in which the medical instrument 4 is inserted into the patient P, and a switch unit 83b that moves the medical instrument 4 in the opposite direction to the direction in which the medical instrument 4 is inserted into the patient P. Both the switch unit 83a and the switch unit 83b are composed of push button switches.

[0043] 7, the operation unit 80 includes a pivot button 85 for teaching a pivot position PP that serves as a fulcrum (see FIG. 11) for the movement of the medical instrument 4 attached to the arm 60. The pivot button 85 is provided on a surface 80b of the operation unit 80 adjacent to the enable switch 81. When the tip of the endoscope 6 (see FIG. 8) or the pivot position teaching instrument 7 (FIG. 9) is moved to a position corresponding to the insertion position of the trocar T inserted into the body surface S of the patient P, the pivot button 85 is pressed to teach the pivot position PP, which is then stored in the memory unit 32. In teaching the pivot position PP, the pivot position PP is set as one point (coordinates), and teaching the pivot position PP does not set the direction of the medical instrument 4.

[0044] As shown in FIG. 1, an endoscope 6 is attached to one arm 60 (for example, arm 60c) of the multiple arms 60, and medical instruments 4 other than the endoscope 6 are attached to the remaining arms 60 (for example, arms 60a, 60b, and 60d). Specifically, in a surgery, an endoscope 6 is attached to one arm 60 of the four arms 60, and medical instruments 4 other than the endoscope 6 (such as forceps 4b) are attached to the three arms 60. Then, a pivot position PP is taught to the arm 60 to which the endoscope 6 is attached in a state in which the endoscope 6 is attached. Also, a pivot position PP is taught to the arm 60 to which the medical instruments 4 other than the endoscope 6 are attached in a state in which the pivot position teaching instrument 7 is attached. The endoscope 6 is attached to one of the two arms 60 (arms 60b and 60c) arranged in the center of the four arms 60 arranged adjacent to each other. That is, the pivot position PP is set individually for each of the multiple arms 60. The arm 60 (for example, the arm 60c) on which the endoscope 6 is supported is an example of a "third manipulator" in the claims.

[0045] 7, an adjustment button 86 for optimizing the position of the arm 60 is provided on the surface 80b of the operation unit 80. After teaching the pivot position PP for the arm 60 to which the endoscope 6 is attached, the adjustment button 86 is pressed to optimize the position of the other arm 60 (arm base 50).

[0046] As shown in Fig. 7, the operation unit 80 includes a mode switching button 84 for switching between a mode for translating (see Fig. 10) the medical instrument 4 attached to the arm 60 and a mode for rotating (see Fig. 11). A mode indicator 84a is provided near the mode switching button 84. The mode indicator 84a displays the switched mode. Specifically, the mode indicator 84a is turned on (rotational movement mode) or off (translational movement mode) to display the current mode (translational movement mode or rotational movement mode).

[0047] The mode indicator 84a also serves as a pivot position indicator that indicates that the pivot position PP has been taught.

[0048] As shown in Fig. 10, in the mode in which the arm 60 is translated, the arm 60 is moved so that the tip 4d of the medical instrument 4 moves on the XY plane. Also, as shown in Fig. 11, in the mode in which the arm 60 is rotationally moved, when the pivot position PP is not taught, the arm 60 is moved so that the medical instrument 4 rotates around the forceps 4b, and when the pivot position PP is taught, the arm 60 is moved so that the medical instrument 4 rotates around the pivot position PP as a fulcrum. Note that the medical instrument 4 is rotationally moved with the shaft 4c of the medical instrument 4 inserted into the trocar T.

[0049] 12, the arm 60 is provided with a plurality of servo motors M1, an encoder E1, and a reducer (not shown) so as to correspond to a plurality of joints 64 of the arm portion 61. The encoder E1 is configured to detect the rotation angle of the servo motor M1. The reducer is configured to reduce the speed of rotation of the servo motor M1 to increase the torque.

[0050] 12, the translation mechanism 70 includes a servomotor M2 for rotating a rotor provided in the driven unit 4a of the medical instrument 4, a servomotor M3 for translating the medical instrument 4, an encoder E2, an encoder E3, and a reducer (not shown). The encoders E2 and E3 are configured to detect the rotation angles of the servomotors M2 and M3, respectively. The reducers are configured to reduce the rotation speed of the servomotors M2 and M3 to increase the torque.

[0051] The positioner 40 is also provided with a plurality of servo motors M4, an encoder E4, and a reducer (not shown) so as to correspond to the plurality of joints 43 of the positioner 40. The encoder E4 is configured to detect the rotation angle of the servo motor M4. The reducer is configured to reduce the speed of rotation of the servo motor M4 to increase the torque.

[0052] The medical cart 3 is provided with a servo motor M5 for driving each of a plurality of front wheels (not shown) of the medical cart 3, an encoder E5, and a reducer and a brake (not shown). The reducer is configured to reduce the rotation of the servo motor M5 to increase the torque. The throttle unit 34a of the medical cart 3 is provided with a potentiometer P1 (see FIG. 1), and the servo motor M5 of the front wheel is driven based on the rotation angle detected by the potentiometer P1 according to the twist of the throttle unit 34a. The rear wheels (not shown) of the medical cart 3 are of a double wheel type, and the rear wheels are steered based on the left and right rotation of the operation handle 34. The operation handle 34 of the medical cart 3 is provided with a potentiometer P2 (see FIG. 2), and the rear wheels of the medical cart 3 are provided with a servo motor M6, an encoder E6, and a reducer (not shown). The reducer is configured to reduce the rotation of the servo motor M6 to increase the torque. The servo motor M6 is driven based on the rotation angle detected by the potentiometer P2 in response to the left and right rotation of the operating handle 34. In other words, the steering of the rear wheels caused by the left and right rotation of the operating handle 34 is configured to be power-assisted by the servo motor M6.

[0053] In addition, the medical cart 3 moves in the front-rear direction by driving the front wheels, and the rear wheels are steered by turning the operating handle 34 of the medical cart 3, so that the medical cart 3 turns in the left-right direction.

[0054] The control unit 31 of the medical cart 3 includes an arm control unit 31a that controls the movement of the multiple arms 60 based on a command, and a positioner control unit 31b that controls the movement of the positioner 40 and the drive of the front wheels (not shown) of the medical cart 3 based on a command. A servo control unit C1 for controlling a servo motor M1 for driving the arm 60 is electrically connected to the arm control unit 31a. In addition, an encoder E1 for detecting the rotation angle of the servo motor M1 is electrically connected to the servo control unit C1.

[0055] A servo control unit C2 for controlling a servo motor M2 for driving the medical instrument 4 is electrically connected to the arm control unit 31a. An encoder E2 for detecting the rotation angle of the servo motor M2 is electrically connected to the servo control unit C2. A servo control unit C3 for controlling a servo motor M3 for translating the translational movement mechanism 70 is electrically connected to the arm control unit 31a. An encoder E3 for detecting the rotation angle of the servo motor M3 is electrically connected to the servo control unit C3.

[0056] Then, the operation command input to the remote operation device 2 is input to the arm control unit 31a. The arm control unit 31a generates a position command based on the input operation command and the rotation angle detected by the encoder E1 (E2, E3), and outputs the position command to the servo control unit C1 (C2, C3). The servo control unit C1 (C2, C3) generates a torque command based on the position command input from the arm control unit 31a and the rotation angle detected by the encoder E1 (E2, E3), and outputs the torque command to the servo motor M1 (M2, M3). As a result, the arm 60 is moved in accordance with the operation command input to the remote operation device 2.

[0057] 12, the control unit 31 (arm control unit 31a) is configured to operate the arm 60 based on an input signal from a joystick 82 of the operation unit 80. Specifically, the arm control unit 31a generates a position command based on the input signal (operation command) input from the joystick 82 and a rotation angle detected by the encoder E1, and outputs the position command to the servo control unit C1. The servo control unit C1 generates a torque command based on the position command input from the arm control unit 31a and the rotation angle detected by the encoder E1, and outputs the torque command to the servo motor M1. As a result, the arm 60 is moved in accordance with the operation command input to the joystick 82.

[0058] The control unit 31 (arm control unit 31a) is configured to operate the arm 60 based on an input signal from the switch unit 83 of the operation unit 80. Specifically, the arm control unit 31a generates a position command based on the input signal (operation command) input from the switch unit 83 and the rotation angle detected by the encoder E1 or E3, and outputs the position command to the servo control unit C1 or C3. The servo control unit C1 or C3 generates a torque command based on the position command input from the arm control unit 31a and the rotation angle detected by the encoder E1 or E3, and outputs the torque command to the servo motor M1 or M3. As a result, the arm 60 is moved in accordance with the operation command input to the switch unit 83.

[0059] 12, a servo control unit C4 for controlling a servo motor M4 that moves the positioner 40 is electrically connected to the positioner control unit 31b. An encoder E4 for detecting the rotation angle of the servo motor M4 is electrically connected to the servo control unit C4. A servo control unit C5 for controlling a servo motor M5 that drives the front wheels (not shown) of the medical cart 3 is electrically connected to the positioner control unit 31b. An encoder E5 for detecting the rotation angle of the servo motor M5 is electrically connected to the servo control unit C5.

[0060] Further, an operation command related to setting of a preparation position and the like is inputted from the input device 33 to the positioner control unit 31b. The positioner control unit 31b generates a position command based on the operation command inputted from the input device 33 and the rotation angle detected by the encoder E4, and outputs the position command to the servo control unit C4. The servo control unit C4 generates a torque command based on the position command inputted from the positioner control unit 31b and the rotation angle detected by the encoder E4, and outputs the torque command to the servo motor M4. As a result, the positioner 40 is moved so as to follow the operation command inputted to the input device 33. Similarly, the positioner control unit 31b moves the medical cart 3 based on the operation command from the input device 33.

[0061] Here, in the first embodiment, the surgical system 100 includes an image processing device 8. The image processing device 8 is configured to generate a graphical user interface G (see FIG. 14) and display the graphical user interface G on the monitor 24 of the remote control device 2 by superimposing it on an image captured by the endoscope 6 (see FIG. 13). The image processing device 8 is configured to capture an image from the endoscope 6. The image processing device 8 is an example of a "control device" in the claims.

[0062] As shown in Fig. 14, the graphical user interface G includes a clutch area G1. As shown in Fig. 15, the clutch area G1 shows the state of the clutch pedal 22b. Fig. 15(a) shows a state in which the clutch pedal 22b is not depressed (OFF state). Fig. 15(b) shows a state in which the operator places his / her foot on the clutch pedal 22b (hover state). Fig. 15(c) shows a state in which the clutch pedal 22b is depressed (ON state).

[0063] As shown in Fig. 14, in the first embodiment, the graphical user interface G includes a camera area G2 that displays information related to the endoscope 6. The camera area G2 is displayed in an area near the lower end ed (see Fig. 13) of the screen gr of the monitor 24. The camera area G2 is an example of the "fourth area" in the claims.

[0064] As shown in FIG. 14, the graphical user interface G includes a hand area G3. The hand area G3 displays information on each medical instrument 4, information on each arm 60, and information on the state of the coagulation pedal 22e and the incision pedal 22d. The hand area G3 includes a hand area G3a showing information on the medical instrument 4 and the arm 60a (numbered "4" in the hand area G3) operated by the left-hand operating handle 21L, a hand area G3b showing information on the replacement medical instrument 4 and the arm 60b (numbered "3" in the hand area G3), and a hand area G3c showing information on the medical instrument 4 and the arm 60d (numbered "1" in the hand area G3) operated by the left-hand operating handle 21L. The hand area G3 is displayed in an area near the lower end ed of the screen gr of the monitor 24. The clutch area G1 is also displayed in an area near the lower end ed of the screen gr. Specifically, as shown in Fig. 14, the clutch area G1, camera area G2, and hand area G3 are displayed between the lower end ed of the screen gr and a position 1 / 10 (L11) of the vertical length of the screen gr from the end ed. The hand areas G3a, G3b, and G3c are examples of the "first area," "second area," and "third area" in the claims, respectively.

[0065] In addition, the information regarding arm 60 includes the arm number (arm number such as "1" or "2") and an arrow icon that is displayed when the arm 60 is set as the replacement arm for the attached medical instrument 4, the information regarding the medical instrument 4 includes the name of the medical instrument 4, and the information regarding the status of coagulation pedal 22e and incision pedal 22d includes the operation status of clutch pedal 22b, the operation status of incision pedal 22d, and the operation status of coagulation pedal 22e.

[0066] 13, the hand area (hand area G3a and hand area G3c) of the arm 60 to be operated is displayed in dark gray with a white oval therein and the arm 60 number (1 and 4) is displayed within the white oval. The hand area (hand area G3b) of the arm 60 not to be operated is displayed in light gray with the arm 60 number (3) displayed in an even lighter gray.

[0067] As shown in FIG. 14, the graphical user interface G includes a medical instrument status information area G4 which is a pop-up area. In the medical instrument status information area G4, the current number of uses / maximum number of uses (see FIG. 16) of the medical instrument 4 attached to each arm 60 are displayed in a pop-up. When the current number of uses is equal to the maximum number of uses, the current number of uses is displayed in red. When an error occurs in the medical instrument 4 attached to each arm 60, error information is displayed in a pop-up. When no medical instrument 4 is attached to the arm 60, nothing is displayed in the medical instrument status information area G4. The medical instrument status information area G4 is displayed in an adjacent area above the clutch area G1, the camera area G2, and the hand area G3 on the monitor 24. The medical instrument status information area G4 is an example of a "surgical instrument status information area" in the claims.

[0068] As shown in Fig. 14, the graphical user interface G includes a spirit level area G5. In the spirit level area G5, angle information of the endoscope 6 is displayed. In addition, the spirit level area G5 is only displayed while the camera pedal 22c is depressed. That is, when the image processing device 8 receives a command to enable the movement of the endoscope 6, it displays the spirit level LV of the endoscope 6 in the spirit level area G5. The spirit level area G5 is an example of the "center area" in the claims.

[0069] As shown in FIG. 14, the graphical user interface G includes a left pop-up area G6. In the left pop-up area G6, when the foot is placed on the foot pedal 22 and in a hover state, icons shown in FIG. 17(a) to (c) are displayed. FIG. 17(a) is an icon displayed when the feet are placed on the coagulation pedal 22eL and the incision pedal 22dL. FIG. 17(b) is an icon displayed when the foot is placed on the clutch pedal 22b. FIG. 17(c) is an icon displayed when the foot is placed on the camera pedal 22c. The left pop-up area G6 is displayed on the left side of the monitor 24. The left pop-up area G6 is an example of the "operation pedal area" in the claims.

[0070] As shown in Fig. 14, the graphical user interface G includes a right pop-up area G7. In the right pop-up area G7, icons (Fig. 18) are displayed when the feet are placed on the coagulation pedal 22eR and the incision pedal 22dR. The right pop-up area G7 is displayed on the right side of the monitor 24. The right pop-up area G7 is an example of the "operation pedal area" in the claims.

[0071] 14, the graphical user interface G includes a first area G8 that displays the operable range of the arm 60 and the operable range of the arm 60 that can be operated by the operating handle 21. The graphical user interface G also includes a second area G9 that displays the operating direction of the operating handle 21 required for returning the operating handle 21 to the operable range and / or (in the first embodiment, "and") for returning the arm 60 to the operable range.

[0072] There are two arms 60 that can be operated by the operating handle 21. For example, the left arm 60L (e.g., arm 60a, see FIG. 1) for supporting the medical instrument 4 is operated by the operating handle 21L. The right arm 60R (e.g., arm 60d, see FIG. 1) for supporting the medical instrument 4 is operated by the operating handle 21R. The first area G8 (first area G8L, first area G8R) and the second area G9 (second area G9L, second area G9R) are provided separately for the left arm 60L and the right arm 60R, respectively. The left arm 60L and the right arm 60R are examples of the "first manipulator" and the "second manipulator" in the claims, respectively.

[0073] 14, the graphical user interface G also includes an error notification area G15 (G15a, G15b). The error notification area G15a is displayed as a pop-up to display warning or error information when a warning or error occurs. The error notification area G15b is displayed as a pop-up to display detailed notes regarding the warning or error displayed in the error notification area G15a.

[0074] In the first embodiment, the indicator MK1 indicating the medical instrument 4 located outside the field of view of the endoscope 6 shown in FIG. 13 and the indicator MK2 shown in FIG. 25 are configured to be switchable between a displayed state and a non-displayed state. Specifically, as shown in FIG. 19, the touch panel 23 of the remote control device 2 is operated to display a button for "displaying forceps outside the field of view". When "ON" is selected for "displaying forceps outside the field of view", the indicator MK1 or the indicator MK2 is displayed when the operator depresses the camera pedal 22c and there is a medical instrument 4 located outside the field of view of the endoscope 6. When "OFF" is selected for "displaying forceps outside the field of view", the indicator MK1 or the indicator MK2 is not displayed. This allows the display setting for "displaying forceps outside the field of view" to be easily switched depending on the operator's operating skill level and the operator's needs.

[0075] As shown in Fig. 14, the graphical user interface G includes a status area G10. In the status area G10, information such as the remaining charge of the built-in battery of the medical manipulator 1, the brightness / contrast of the monitor 24, lap time, and surgery elapsed time is displayed. Specifically, as shown in Fig. 14, the status area G10 is displayed between the upper end eu of the screen gr and a position 1 / 10 (L11) of the vertical length of the screen gr from the end eu. The status area G10 is an example of a "surgery system area" in the claims.

[0076] Here, in the first embodiment, as shown in FIG. 13 and FIG. 14, the image processing device 8 displays on the monitor 24 a graphical user interface G for displaying a mark MK1 indicating a medical instrument 4 located outside the field of view of the endoscope 6, superimposed on an image captured by the endoscope 6. This mark MK1 (graphical user interface G) is displayed when the image processing device 8 receives a command to enable the movement of the endoscope 6 (when the camera pedal 22c is depressed by the operator) and when at least one of the medical instruments 4 (at least one of the medical instruments 4 attached to each of the arms 60a, 60b, and 60d) is located outside the field of view of the endoscope 6. In addition, this mark MK1 (graphical user interface G) is displayed in an area G11 near the outer edge of the level area G5 including the center CN1 but not including the vicinity of the end e of the screen gr of the monitor 24. The area G11 near the outer edge is an area located outside the level LV described later and near the level LV.

[0077] The image processing device 8 acquires the position of the medical instrument 4 based on the attitude and position of the arm 60. The image processing device 8 also acquires the imaging direction of the endoscope 6 based on the attitude and position of the arm 60. The image processing device 8 also acquires the angle of view (field of view range) of the endoscope 6 based on the zoom state of the endoscope 6. The image processing device 8 acquires the angle of view (field of view range) of the endoscope 6 using values ​​set as the mechanical mechanism (lens, etc.) of the endoscope 6. The image processing device 8 then acquires the coordinates of the tip of the medical instrument 4 relative to the field of view of the endoscope 6 from the field of view of the endoscope 6, the attitude and position of the endoscope 6, and position information of the arm 60. As a result, the image processing device 8 determines whether or not the medical instrument 4 is located outside the field of view of the endoscope 6.

[0078] In the example shown in FIG. 13 and FIG. 14, among the arms 60a, 60b, 60c, and 60d, the arm 60a (number "4" in the hand area G3a), the arm 60b (number "3" in the hand area G3b), and the arm 60d (number "1" in the hand area G3c) are attached with a medical instrument 4 (a medical instrument 4 such as forceps 4b other than an endoscope 6). The arm 60c (number "2" in the camera area G2) is attached with an endoscope 6. The arm 60a (number "4" in the hand area G3a) and the arm 60d (number "1" in the hand area G3) are in a state in which they can be operated by the operation handle 21 (active), and the hand areas G3a and G3c are displayed in dark gray. The arm 60b is in a state in which they cannot be operated by the operation handle 21 (inactive), and the hand area G3b is displayed in light gray.

[0079] 13, the medical instruments 4 supported by the arm 60a (numbered "4" in the hand area G3a) and the arm 60d (numbered "1" in the hand area G3c) are located within the field of view of the endoscope 6. On the other hand, the medical instrument 4 supported by the arm 60b (numbered "3" in the hand area G3b) is located outside the field of view of the endoscope 6.

[0080] In the first embodiment, as shown in Fig. 13, the graphical user interface G is configured to display a mark MK1 indicating the medical instrument 4 outside the field of view in an area G12 (see Fig. 20) in the outer edge vicinity area G11 corresponding to the direction in which the medical instrument 4 outside the field of view is located with respect to the center CN2 of the spirit level area G5. In Fig. 13, the mark MK1 (the character "3") is displayed.

[0081] In the first embodiment, as shown in FIG. 20, the area G12 in which the marker MK1 indicating the medical instrument 4 outside the field of view is displayed is one of the areas G12 obtained by radially dividing the outer edge vicinity area G11 from the center CN2 of the level area G5. In the first embodiment, the outer edge vicinity area G11 is radially divided into eight areas from the center CN2. The eight areas G12a to G12h are arranged at 45 degree intervals with respect to the center CN2. In FIG. 13, the marker MK1 (the character "3") is displayed in the area G12d (see FIG. 20) as an example. Note that FIG. 20 displays all patterns in which the medical instrument 4 is located outside the field of view.

[0082] In the first embodiment, as shown in Fig. 14, when the image processing device 8 receives a command to enable the movement of the endoscope 6, it displays a spirit level LV of the endoscope 6 in the spirit level area G5. Then, the image processing device 8 displays, on the monitor 24, a graphical user interface G for displaying a mark MK1 in an area G11 near the outer edge outside the spirit level area G5. The spirit level LV represents the inclination of the field of view of the endoscope 6 relative to the patient P. The spirit level LV is displayed when the image processing device 8 receives a command to enable the movement of the endoscope 6 when the camera pedal 22c is operated (specifically, when it is depressed).

[0083] In the first embodiment, as shown in Fig. 14, the level area G5 is a rectangular area (rectangle long in the horizontal direction) including an upper side G5a, a left side G5b, a right side G5c, and a lower side G5d. By displaying the mark MK1 in an area G11 near the outer edge of the level area G5 of a predetermined size including the center of the screen gr, the visibility of the operator can be improved. The upper side G5a is located between a position 1 / 10 (L11) above the center CN1 of the screen gr of the monitor 24 and a position 1 / 10 (L11) below the upper end eu of the screen gr. Preferably, the upper edge portion G5a is located between a position 1 / 8 of the vertical length of the screen gr above the center CN1 of the screen gr of the monitor 24 and a position 1 / 8 of the vertical length of the screen gr below the upper end eu of the screen gr, and more preferably, the upper edge portion G5a is located between a position 1 / 6 of the vertical length of the screen gr above the center CN1 of the screen gr of the monitor 24 and a position 1 / 6 of the vertical length of the screen gr below the upper end eu of the screen gr.

[0084] Moreover, the upper side portion G5a is disposed below a status area G10 in which the remaining charge of the built-in battery of the medical manipulator 1 and the like are displayed.

[0085] The left side G5b is disposed between a position 1 / 10 (L12) of the horizontal length of the screen gr from the center CN1 of the screen gr of the monitor 24 to the left and a position 1 / 10 (L12) of the horizontal length of the screen gr from the left end el of the screen gr to the right. Preferably, the left side G5b is disposed between a position 1 / 8 of the horizontal length of the screen gr from the center CN1 of the screen gr of the monitor 24 to the left and a position 1 / 8 of the horizontal length of the screen gr from the left end el of the screen gr to the right, and more preferably, the left side G5b is disposed between a position 1 / 6 of the horizontal length of the screen gr from the center CN1 of the screen gr of the monitor 24 to the left and a position 1 / 6 of the horizontal length of the screen gr from the left end el of the screen gr to the right.

[0086] The right side G5c is disposed between a position 1 / 10 (L12) of the horizontal length of the screen gr from the center CN1 of the screen gr of the monitor 24 to the right and a position 1 / 10 (L12) of the horizontal length of the screen gr from the right end er of the screen gr to the left. Preferably, the right side G5c is disposed between a position 1 / 8 of the horizontal length of the screen gr from the center CN1 of the screen gr of the monitor 24 to the right and a position 1 / 8 of the horizontal length of the screen gr from the right end er of the screen gr to the left, and more preferably, the right side G5c is disposed between a position 1 / 6 of the horizontal length of the screen gr from the center CN1 of the screen gr of the monitor 24 to the right and a position 1 / 6 of the horizontal length of the screen gr from the right end er of the screen gr to the left.

[0087] The lower side G5d is disposed between a position 1 / 10 (L11) of the vertical length of the screen gr from the center CN1 of the screen gr of the monitor 24 and a position 1 / 10 (L11) of the vertical length of the screen gr from the lower end ed of the screen gr. Preferably, the lower side G5d is disposed between a position 1 / 8 of the vertical length of the screen gr from the center CN1 of the screen gr of the monitor 24 and a position 1 / 8 of the vertical length of the screen gr from the lower end ed of the screen gr, and more preferably, the lower side G5d is disposed between a position 1 / 6 of the vertical length of the screen gr from the center CN1 of the screen gr of the monitor 24 and a position 1 / 6 of the vertical length of the screen gr from the lower end ed of the screen gr.

[0088] In the first embodiment, the bottom side G5d of the level area G5 is disposed above the hand area G3a, the hand area G3b, the hand area G3c, and the camera area G2. Furthermore, the bottom side G5d of the level area G5 is disposed above the medical instrument status information area G4.

[0089] The center CN1 of the screen gr of the monitor 24 and the center CN2 of the level area G5 are at substantially the same position.

[0090] In the first embodiment, the mark MK1 indicating the medical instrument 4 located outside the field of view is disposed between the spirit level area G5 and at least one of the medical instrument status information area G4, the left pop-up area G6, the right pop-up area G7, and the status area G10 (all of them in the first embodiment). The medical instrument status information area G4 indicates the status of the medical instrument 4. The left pop-up area G6 is displayed when the foot pedal 22 is operated (when the foot is placed on the foot pedal 22 in a hover state). The right pop-up area G7 is displayed when the coagulation pedal 22eR and the incision pedal 22dR are operated (when the operator's foot is placed on them). The status area G10 indicates the status of the surgical operation system 100.

[0091] In the first embodiment, as shown in Fig. 13, the mark MK1 includes a number that identifies the arm 60 supporting the medical instrument 4 that is located outside the field of view. In the example shown in Fig. 13, the number "3" is displayed as MK1, indicating that the medical instrument 4 supported by the arm 60b (number "3" in the hand area G3) is located outside the field of view of the endoscope 6.

[0092] In the first embodiment, as shown in Fig. 21 and Fig. 22, the marker MK1 is displayed in a different manner depending on whether the medical instrument 4 located outside the field of view is an operation target or not. Specifically, as shown in Fig. 21, when the medical instrument 4 is an operation target, a white oval is displayed in a black square, and the number of the arm 60 (any of 1 to 4) is displayed in the white oval. Also, as shown in Fig. 22, when the medical instrument 4 is not an operation target, a gray number of the arm 60 (any of 1 to 4) is displayed in a black square.

[0093] 20, in the first embodiment, when multiple medical instruments 4 attached to the arm 60a, the arm 60b, and the arm 60d are located outside the field of view of the endoscope 6, multiple markers MK1 corresponding to the medical instruments 4 located outside the field of view of the endoscope 6 are displayed in a manner corresponding to the order of the positions of the arms 60a, 60b, and 60d. The order of the positions of the arms 60a, 60b, and 60d means the order of the positions of the arms 60a, 60b, and 60d when attached to the arm base 50.

[0094] 20, in the first embodiment, when all (three) of the medical instruments 4 attached to the arm 60a, the medical instruments 4 attached to the arm 60b, and the medical instruments 4 attached to the arm 60d are located outside the field of view of the endoscope 6, the mark MK1 ("4") representing the arm 60a, the mark MK1 ("3") representing the arm 60b, and the mark MK1 ("1") representing the arm 60d are displayed in the order of the arm 60a, the arm 60b, and the arm 60d. That is, the marks MK1 ("4," "3," and "1") are displayed in the order of the arrangement of the hand area G3.

[0095] 23, in the first embodiment, when one or two of the medical instruments 4 attached to the arm 60a, the medical instruments 4 attached to the arm 60b, and the medical instruments 4 attached to the arm 60d are located outside the field of view of the endoscope 6, the mark MK1 corresponding to the medical instrument 4 located outside the field of view of the endoscope 6 is positioned closer to the center of the screen gr of the monitor 24. In other words, when only the medical instrument 4 attached to the arm 60a is located outside the left side of the field of view, the mark MK1 ("4") indicating the medical instrument 4 attached to the arm 60a is positioned closer to the center of the screen gr than the position of the mark MK1 ("4") when all three marks MK1 ("4", "3", and "1") are displayed (see FIG. 20).

[0096] Similarly, when only the medical instrument 4 attached to the arm 60a is located outside the lower side of the field of view, the mark MK1 ("4") indicating the medical instrument 4 attached to the arm 60a is positioned closer to the center of the screen gr than the position of the mark MK1 ("4") when all three marks MK1 ("4", "3", and "1") are displayed (see FIG. 20). The same is true when only the medical instrument 4 attached to the arm 60a is located outside the upper side (diagonally upper left or diagonally lower left) of the field of view.

[0097] Similarly, when only the medical instrument 4 attached to the arm 60d is located outside the right side of the visual field, the mark MK1 ("1") indicating the medical instrument 4 attached to the arm 60d is positioned closer to the center of the screen gr than the position of the mark MK1 ("1") when all three marks MK1 ("4", "3" and "1") are displayed (see FIG. 20). The same is true when only the medical instrument 4 attached to the arm 60d is located outside the diagonally upper right and lower right sides of the visual field.

[0098] (Display method) Next, a method of displaying the graphical user interface G on the monitor 24 in the surgical operation system 100 will be described with reference to Fig. 24. The graphical user interface G is generated by the image processing device 8.

[0099] First, in step S1, the image processing device 8 acquires an image captured by the endoscope 6 for capturing an image of a surgical site.

[0100] Next, in step S2, the image processing device 8 judges whether or not it has received a command to enable the movement of the endoscope 6 (whether or not the operator has depressed the camera pedal 22c). If the answer is yes in step S2, it proceeds to step S3. If the answer is no in step S2, it returns to step S1.

[0101] Next, in step S3, the image processing device 8 determines whether or not at least any of the medical instruments 4 supported by the arm 60a, the medical instruments 4 supported by the arm 60b, and the medical instruments 4 supported by the arm 60d are located outside the field of view of the endoscope 6. If the answer is yes in step S3 (i.e., if a command to enable movement of the endoscope 6 has been received and the medical instruments 4 supported by the arm 60 are located outside the field of view of the endoscope 6), the process proceeds to step S4. If the answer is no in step S4, the process returns to step S1.

[0102] Next, in step S4, the image processing device 8 displays on the monitor 24 a graphical user interface G for displaying a mark MK1 indicating a medical instrument 4 located outside the field of view in an area G11 near the outer edge of the spirit level area G5 including the central part CN1 but not including the area near the end e of the screen gr of the monitor 24, superimposed on the image captured by the endoscope 6.

[0103] The operations of steps S1 to S4 described above are always performed while the surgical operation system 100 is in operation.

[0104] [Effects of the first embodiment] In the first embodiment, the following effects can be obtained.

[0105] In the first embodiment, as described above, the image processing device 8 is configured to display on the monitor 24 a graphical user interface G for displaying a mark MK1 indicating a medical instrument 4 located outside the field of view in an area G11 near the outer edge of a level area G5 including a central part CN1 but not including the vicinity of an end part e of the screen gr of the monitor 24, superimposed on an image captured by the endoscope 6. This causes the mark MK1 indicating a medical instrument 4 located outside the field of view to be displayed relatively closer to the center than the vicinity of the end part e on the screen gr of the monitor 24. As a result, the operator can easily visually recognize the mark MK1 indicating a medical instrument 4 located outside the field of view without moving his / her line of sight significantly.

[0106] In the first embodiment, as described above, the graphical user interface G is configured to display the mark MK1 in the area G11 near the outer edge of the level area G5 and in the area G12 corresponding to the direction in which the medical instrument 4 outside the field of view is located with respect to the center CN2 of the level area G5. As a result, the mark MK1 is arranged so as to correspond to the direction in which the medical instrument 4 outside the field of view is located with respect to the center CN2 of the level area G5, so that the operator can easily recognize in which direction the medical instrument 4 outside the field of view is located with respect to the center CN2 of the level area G5 without moving his / her line of sight significantly. In addition, the position of the mark MK1 on the screen gr of the monitor 24 (the position with respect to the center CN1 of the level area G5) corresponds to the position of the medical instrument 4 outside the field of view, so that the position (direction) of the medical instrument 4 outside the field of view can be intuitively and easily recognized.

[0107] In the first embodiment, as described above, the area G12 in which the mark MK1 indicating the medical instrument 4 outside the field of view is displayed is one of the areas G12 (G12a to G12h) obtained by dividing the outer edge vicinity area G11 radially from the center CN2 of the level area G5. As a result, since the multiple areas G12 are provided radially from the center CN2 of the level area G5, the position of the medical instrument 4 outside the field of view can be more intuitively recognized, unlike the case where the multiple areas G12 are provided in one biased region on the screen gr of the monitor 24.

[0108] Furthermore, in the first embodiment, as described above, when the image processing device 8 receives a command to enable movement of the endoscope 6, it displays the spirit level LV of the endoscope 6 in the spirit level area G5 and displays a graphical user interface G for displaying the mark MK1 outside the spirit level LV on the monitor 24. As a result, while the operator is visually checking the spirit level LV of the endoscope 6 in order to operate the endoscope 6, the mark MK1 indicating the medical instrument 4 located outside the field of view is displayed together with the spirit level LV, so that the operator can recognize the mark MK1 even more easily.

[0109] In the first embodiment, as described above, the mark MK1 indicating the medical instrument 4 located outside the field of view is disposed between the spirit level area G5 and at least one (in the first embodiment, all) of the medical instrument status information area G4, the left pop-up area G6, the right pop-up area G7, and the status area G10. This causes the mark MK1 to be displayed closer to the center than at least one (in the first embodiment, all) of the medical instrument status information area G4, the left pop-up area G6, the right pop-up area G7, and the status area G10, and therefore the visibility of the mark MK1 can be improved, unlike when the mark MK1 is disposed at the edge e of the screen gr.

[0110] In the first embodiment, as described above, the level area G5 is a rectangular area including an upper side G5a, a left side G5b, a right side G5c, and a lower side G5d, and the upper side G5a is disposed closer to the upper end eu of the screen gr than the center CN1 of the screen gr of the monitor 24, the left side G5b is disposed closer to the left end el of the screen gr than the center CN1 of the screen gr of the monitor 24, the right side G5c is disposed closer to the right end er of the screen gr than the center CN1 of the screen gr of the monitor 24, and the lower side G5d is disposed closer to the lower end ed of the screen gr than the center CN1 of the screen gr of the monitor 24. As a result, the mark MK1 is displayed in the area G11 near the outer edge of the level area G5, and the mark MK1 is not displayed in the center CN1 of the screen gr of the monitor 24. This makes it possible to prevent the mark MK1 from interfering with the operator's view of the center CN1 of the screen gr of the monitor 24.

[0111] In the first embodiment, as described above, the upper side G5a is disposed between a position 1 / 10 (L11) of the vertical length of the screen gr from the center CN1 of the screen gr of the monitor 24 and a position 1 / 10 (L11) of the vertical length of the screen gr from the upper end eu of the screen gr, and the left side G5b is disposed between a position 1 / 10 (L10) of the horizontal length of the screen gr from the center CN1 of the screen gr of the monitor 24 to the left and a position 1 / 10 (L12) of the horizontal length of the screen gr to the right of the left end el of the screen gr. The right side G5c is disposed between a position 1 / 10 of the horizontal length of the screen gr from the center CN1 of the screen gr of the monitor 24 to the right and a position 1 / 10 (L12) of the horizontal length of the screen gr from the right end er of the screen gr to the left, and the bottom side G5d is disposed between a position 1 / 10 of the vertical length of the screen gr from the center CN1 of the screen gr of the monitor 24 below and a position 1 / 10 (L11) of the vertical length of the screen gr from the lower end ed of the screen gr. This causes the mark MK1 displayed in the area G11 near the outer edge of the level area G5 to be displayed closer to the center than the end e of the screen gr, which improves the visibility of the mark MK1, unlike the case where the mark MK1 is disposed at the end e of the screen gr.

[0112] The upper side G5a is disposed between a position 1 / 8 of the vertical length of the screen gr from the center CN1 of the screen gr of the monitor 24 and a position 1 / 8 of the vertical length of the screen gr from the upper end eu of the screen gr. The left side G5b is disposed between a position 1 / 8 of the horizontal length of the screen gr from the center CN1 of the screen gr of the monitor 24 to the left and a position 1 / 8 of the horizontal length of the screen gr from the left end el of the screen gr. The right side G5c is disposed between a position 1 / 8 of the horizontal length of the screen gr from the center CN1 of the screen gr of the monitor 24 and a position 1 / 8 of the horizontal length of the screen gr from the left end el of the screen gr. By positioning the lower edge G5d between a position 1 / 8 of the horizontal length of the screen gr from the center CN1 of the screen gr of the monitor 24 to the right and 1 / 8 of the horizontal length of the screen gr from the right end er of the screen gr, and between a position 1 / 8 of the vertical length of the screen gr below the center CN1 of the screen gr of the monitor 24 and a position 1 / 8 of the vertical length of the screen gr above the lower end ed of the screen gr, the visibility of the sign MK1 can be further improved.

[0113] In the first embodiment, as described above, the arms 60 include an arm 60b for supporting a medical instrument 4 that is not the object of operation (for replacement), and the graphical user interface G displays a hand area G3a showing information on the medical instrument 4 supported by the arm 60d, a hand area G3b displaying information on the medical instrument 4 supported by the arm 60b, a hand area G3c displaying information on the medical instrument 4 supported by the arm 60a, and a camera area G2 showing information on the endoscope 6, all of which are arranged in a region near the lower end ed of the screen gr of the monitor 24. As a result, the mark MK1 is displayed closer to the center of the screen gr than the hand areas G3a, G3b, G3c, and the camera area G2, and therefore the visibility of the mark MK1 can be further improved.

[0114] In the first embodiment, as described above, the level area G5 is a rectangular area including an upper side G5a, a left side G5b, a right side G5c, and a lower side G5d, and the lower side G5d is disposed above the hand areas G3a, G3b, G3c, and the camera area G2. As a result, the marker MK1 is disposed above compared to a case in which the lower side G5d of the level area G5 is disposed at the lower end ed of the screen gr, and therefore the marker MK1 can be visually recognized while suppressing the operator's downward movement of the line of sight.

[0115] Furthermore, in the first embodiment, as described above, when two or more of the arms 60a, 60b, and 60d are positioned outside the field of view of the endoscope 6, a plurality of markers MK1 corresponding to the medical instruments 4 positioned outside the field of view of the endoscope 6 are displayed so as to correspond to the order of the arrangement positions of the arms 60a, 60b, and 60d on which the medical instruments 4 are provided. This allows the operator to easily recognize the arrangement order of the medical instruments 4 positioned outside the field of view.

[0116] In the first embodiment, as described above, the marker MK1 is configured to be switchable between a displayed state and a hidden state, so that the marker MK1 can be switched between a displayed state and a hidden state as needed.

[0117] Furthermore, in the first embodiment, as described above, the multiple markers MK1 are configured to be displayable in a line corresponding to the order of the arrangement position of the arm 60 even when all of the medical instruments 4 are located outside the field of view of the endoscope 6, and when one or two of the multiple medical instruments 4 are located outside the field of view of the endoscope 6, the markers MK1 corresponding to the medical instruments 4 located outside the field of view of the endoscope 6 are positioned closer to the center of the screen gr of the monitor 24. As a result, the markers MK1 are positioned closer to the center, so that the visibility of the markers MK1 can be further improved.

[0118] Also, in the first embodiment, as described above, the remote control device 2 includes the camera pedal 22c for inputting a command to move the endoscope 6, and is configured so that when a command enabling the movement of the endoscope 6 is input by the camera pedal 22c, the endoscope 6 is moved by operating both the operation handles 21L and 21R. This allows the operator to easily move the endoscope 6 by operating the camera pedal 22c.

[0119] In the first embodiment, as described above, the marker MK1 is displayed in a different manner depending on whether the medical instrument 4 located outside the field of view is an operation target or not, thereby allowing the operator to easily recognize whether the medical instrument 4 located outside the field of view is an operation target or not.

[0120] In the first embodiment, as described above, the mark MK1 includes a number that identifies the arm 60 that supports the medical instrument 4 that is located outside the field of view. This allows the operator to easily identify and recognize the medical instrument 4 that is located outside the field of view.

[0121] [Second embodiment] The configuration of a surgical operation system 200 according to the second embodiment will be described with reference to FIGS.

[0122] In the second embodiment, as shown in FIG. 25 and FIG. 26, the mark MK2 includes an arrow MK2a indicating the direction in which the medical instrument 4 located outside the field of view is located. Specifically, a number (for example, "4") for identifying the medical instrument 4 located outside the field of view is displayed near the outside of the arrow MK2a. The number may be displayed inside the arrow MK2a, as in the mark MK3 shown in FIG. 27. The inside of the arrow MK2a is displayed transparently or semi-transparently so that the image can be seen through. The number of the mark MK2 for the medical instrument 4 to be operated is displayed as the number of the black arm 60 (see "4") in a white circle. The number of the mark MK2 for the medical instrument 4 not to be operated is displayed as the number of the gray arm 60 (see "3") in a black circle. The arrow MK2a is displayed two-dimensionally.

[0123] [Effects of the second embodiment] In the second embodiment, the following effects can be obtained.

[0124] In the second embodiment, as described above, the mark MK2 includes an arrow MK2a indicating the direction in which the medical instrument 4 located outside the field of view is located. This allows the operator to intuitively recognize the direction in which the medical instrument 4 located outside the field of view is located.

[0125] [Variations] The embodiments disclosed herein should be considered to be illustrative and not restrictive in all respects. The scope of the present invention is indicated by the claims, not by the description of the embodiments above, and further includes all modifications (variations) within the meaning and scope of the claims.

[0126] For example, in the above first and second embodiments, an example has been shown in which the image processing device 8 captures an image from the endoscope 6 and generates a graphical user interface G, but the present invention is not limited to this. For example, the control unit 31 of the medical manipulator 1 may generate the graphical user interface G, or a control unit (not shown) of the remote control device 2 may generate the graphical user interface G. Also, an image processing device that captures an image from the endoscope 6 and an image processing device that generates the graphical user interface G and overlays it on the image of the endoscope 6 may be provided separately.

[0127] In the above first and second embodiments, the mark MK1 is displayed in the area G12 corresponding to the direction in which the medical instrument 4 outside the field of view is located relative to the center CN2 of the spirit level area G5, but the present invention is not limited to this. For example, the mark MK1 may be displayed at a fixed position in the area G11 near the outer edge of the spirit level area G5.

[0128] In the above first and second embodiments, the area G12 in which the mark MK1 is displayed is one of the eight areas G12 obtained by radially dividing the outer edge vicinity area G11 from the center CN2 of the level area G5, but the present invention is not limited to this. For example, the number of divided areas G12 may be a number other than eight.

[0129] In the above first and second embodiments, the mark MK1 is displayed outside the level area G5, but the present invention is not limited to this. For example, the mark MK1 may be displayed inside the level area G5.

[0130] In the first and second embodiments, the level area G5 is a rectangular area, but the present invention is not limited to this. For example, the level area G5 may be an area having a shape other than a rectangle (such as an oval shape).

[0131] In the above first and second embodiments, the camera area G2 and the hand area G3 (hand area G3a, hand area G3b, and hand area G3c) are displayed in a region near the lower end ed of the screen gr of the monitor 24, but the present invention is not limited to this. For example, the camera area G2 and the hand area G3 (hand area G3a, hand area G3b, and hand area G3c) may be displayed in a region near the upper end eu of the screen gr of the monitor 24. Specifically, the camera area G2 and the hand area G3 may be displayed between the upper end eu of the screen gr and a position 1 / 10 (L11) of the vertical length of the screen gr downward from the upper end eu.

[0132] In the above first and second embodiments, when the multiple medical instruments 4 are located outside the field of view of the endoscope 6, the multiple markers MK1 are displayed in the order of the positions of the arm 60 on which the multiple medical instruments 4 are supported, but the present invention is not limited to this. For example, the order of the positions of the arm 60 on which the multiple medical instruments 4 are supported may be different from the order in which the multiple markers MK1 are displayed.

[0133] In the above first and second embodiments, the marker MK1 is disposed closer to the center of the screen gr when one or two medical instruments 4 are located outside the field of view of the endoscope 6, but the present invention is not limited to this. For example, the marker MK1 corresponding to each of the multiple medical instruments 4 may be disposed at a fixed position.

[0134] In the above first and second embodiments, the display mode of the marker MK1 is different between when the medical instrument 4 is an operation target and when it is not an operation target, but the present invention is not limited to this. For example, the display mode of the marker MK1 may be the same between when the medical instrument 4 is an operation target and when it is not an operation target.

[0135] In the above first and second embodiments, the mark MK1 indicating the medical instrument 4 located outside the field of view is displayed in the area G11 near the outer edge of the level area G5, but the present invention is not limited to this. For example, as shown in Fig. 28 and Fig. 29, in addition to the area G11 near the outer edge of the level area G5, an area G13 divided into eight parts is displayed inside (for example, at the left end) of the hand area G3a (hand area G3b and hand area G3c) (see Fig. 29(a)). The area G13 corresponding to the medical instrument 4 located outside the field of view may be filled in white (see Fig. 29(b)) to indicate the medical instrument 4 located outside the field of view.

[0136] In the above first and second embodiments, the mark MK1 indicating the medical instrument 4 located outside the field of view is displayed in the area G11 near the outer edge of the level area G5, but the present invention is not limited to this. For example, as shown in Fig. 30 and Fig. 31, in addition to the area G11 near the outer edge of the level area G5, an area G14 is prepared by dividing the vicinity of the outer edge of the hand area G3a (hand area G3b and hand area G3c) into eight. Then, the area G14 corresponding to the medical instrument 4 located outside the field of view may be filled in white to indicate the medical instrument 4 located outside the field of view.

[0137] In addition, in the above first and second embodiments, an example in which four arms 60 are provided is shown, but the present invention is not limited to this. In the present invention, the number of arms 60 may be any other number as long as at least one or more is provided.

[0138] In the above first and second embodiments, the arm unit 61 and the positioner 40 are configured as a 7-axis articulated robot, but the present invention is not limited to this. For example, the arm unit 61 and the positioner 40 may be configured as an articulated robot having an axis configuration other than a 7-axis articulated robot (for example, a 6-axis or 8-axis articulated robot).

[0139] In the above first and second embodiments, the medical manipulator 1 includes the medical cart 3, the positioner 40, and the arm base 50, but the present invention is not limited to this. For example, the medical cart 3, the positioner 40, and the arm base 50 are not necessarily required, and the medical manipulator 1 may be composed of only the arm 60.

[0140] In the first embodiment, the number of the arm 60 (any one of 1 to 4) is displayed in a black square, but the present invention is not limited to this. For example, as shown in the indicator MK4 in Fig. 32, the outer edge e1 of the black square may be surrounded by another color (such as red). As shown in the indicator MK5 in Fig. 33(a), no black square may be displayed, and only the number of the arm 60 may be displayed in a white oval (if the arm 60 is to be operated), or as shown in the indicator MK5 in Fig. 33(b), only the number of the arm 60 in gray may be displayed (if the arm 60 is not to be operated).

[0141] In the second embodiment, the arrow MK2a is shown in a two-dimensional manner, but the present invention is not limited to this. For example, the arrow MK2a may be shown in a three-dimensional manner. This allows the operator to distinguish (recognize) whether the medical instrument 4 located outside the field of view of the endoscope 6 is located closer to the operator than the medical instrument 4 located farther from the operator.

[0142] In the above first and second embodiments, an example was shown in which the level area G5 in which the level LV is displayed is applied as the center area of ​​the present invention, but the present invention is not limited to this. In the present invention, an area other than the level area G5 in which the level LV is displayed may be applied as the center area of ​​the present invention.

[0143] The functions of the elements disclosed herein can be performed using circuits or processing circuits, including general purpose processors, special purpose processors, integrated circuits, Application Specific Integrated Circuits (ASICs), conventional circuits, and / or combinations thereof, configured or programmed to perform the disclosed functions. Processors are considered processing circuits or circuits because they include transistors and other circuits. In this disclosure, a circuit, unit, or means is hardware that performs the recited functions or hardware that is programmed to perform the recited functions. The hardware may be hardware disclosed herein or other known hardware that is programmed or configured to perform the recited functions. Where the hardware is a processor, which is considered a type of circuit, the circuit, means, or unit is a combination of hardware and software, and the software is used to configure the hardware and / or the processor. [Explanation of symbols]

[0144] 2 Remote control device 4 Medical instruments (first surgical instruments, second surgical instruments, replacement surgical instruments) 6 Endoscopy 8 Image processing device (control device) 21L Operating handle (first operating handle) 21R Operating handle (second operating handle) 22 Foot pedal (operation pedal) 22c Camera pedal (endoscope pedal, operation pedal) 24 Monitor (display device) 60 Arm (Manipulator) 60L Left arm (first manipulator) 60R Right arm (second manipulator) 60c Arm (3rd manipulator) 100 Surgical system (surgical system) CN1 (center of image) CN2 (Center area) Center e (of an image) ed (lower) end el (left) end er (right) end eu (upper) end gr image G Graphical User Interface G2 Camera Area (4th Area) G3a Hand Area (1st Area) G3b Hand Area (Second Area) G3c Hand Area (3rd Area) G4 Medical Instrument Status Information Area (Surgical Instrument Status Information Area) G5 Spirit Level Area (Center Area) G5a Top G5b Left side G5c right side G5d Lower part G6 Left pop-up area (operation pedal area) G7 Right pop-up area (operation pedal area) G10 Status Area (Surgery System Area) G11 Outer edge area G12 (corresponding to the direction in which the medical device is located) area LV spirit level MK1, MK2, MK3, MK4 MK5 label MK2a Arrow

Claims

1. an endoscope for capturing an image of a surgical site, a plurality of manipulators for supporting a first surgical instrument and a second surgical instrument, respectively; a remote control device including a display device for displaying an image captured by the endoscope, a first operating handle for operating the first surgical instrument, and a second operating handle for operating the second surgical instrument; A control device, the control device is configured to, when receiving a command to enable movement of the endoscope and when at least one of the first surgical instrument and the second surgical instrument is located outside the field of view of the endoscope, display on the display device a graphical user interface for displaying a sign indicating at least one of the first surgical instrument and the second surgical instrument located outside the field of view in an outer edge area of ​​a center area of ​​a screen of the display device, superimposed on the image captured by the endoscope; the center area is a rectangular area including an upper side, a left side, a right side, and a lower side, the upper side portion is disposed between a position that is 1 / 10 of the vertical length of the screen from the center of the screen and a position that is 1 / 10 of the vertical length of the screen from the upper end of the screen, the left side portion is disposed between a position that is 1 / 10 of the horizontal length of the screen from the center of the screen to the left and a position that is 1 / 10 of the horizontal length of the screen from the left end of the screen to the right, the right side portion is disposed between a position on the right side of the center of the screen by a length of 1 / 10 of the horizontal length of the screen and a position on the left side of the right edge of the screen by a length of 1 / 10 of the horizontal length of the screen, A surgical system, wherein the lower edge portion is positioned between a position 1 / 10 of the vertical length of the screen below the center of the screen and a position 1 / 10 of the vertical length of the screen above the lower end of the screen.

2. The surgical system of claim 1, wherein the graphical user interface is configured to position and display the marker in an area that corresponds to a direction in which at least one of the first surgical instrument and the second surgical instrument is located outside the field of view relative to the center of the center area, in the outer edge area of ​​the center area.

3. The surgical system of claim 2, wherein the area in which the sign indicating at least one of the first surgical instrument and the second surgical instrument outside the field of view is displayed is one of multiple areas obtained by dividing the outer edge area radially from the center of the center area.

4. The surgical system according to any one of claims 1 to 3, wherein when the control device receives a command to enable movement of the endoscope, the control device displays a spirit level for the endoscope in the center area and displays the graphical user interface on the display device, the graphical user interface displaying the mark outside the spirit level.

5. The remote control device includes an operation pedal, The graphical user interface further includes at least one of a surgical instrument status information area showing status information of a surgical instrument, an operation pedal area displayed when the operation pedal is operated, and a surgical system area showing a status of the surgical system; The surgical system according to any one of claims 1 to 4, wherein the sign is positioned between the center area and at least one of the surgical instrument status information area, the operation pedal area, and the surgical system area.

6. the upper side portion is disposed between a position 1 / 8 of the vertical length of the screen from the center of the screen and a position 1 / 8 of the vertical length of the screen from the upper end of the screen, the left side portion is disposed between a position that is 1 / 8 of the horizontal length of the screen from a center portion of the screen to the left and a position that is 1 / 8 of the horizontal length of the screen from a left end portion of the screen to the right, the right side portion is disposed at a position to the right of the center of the screen by a length of 1 / 8 of the horizontal length of the screen, and at a position to the left of the right end of the screen by a length of 1 / 8 of the horizontal length of the screen, The surgical system according to any one of claims 1 to 5, wherein the lower edge portion is positioned between a position 1 / 8 of the vertical length of the screen below the center of the screen and a position 1 / 8 of the vertical length of the screen above the lower end of the screen.

7. the plurality of manipulators includes a manipulator for supporting an exchange surgical instrument; The surgical system according to any one of claims 1 to 6, wherein the graphical user interface displays a first area showing information regarding the first surgical instrument, a second area showing information regarding the replacement surgical instrument, a third area showing information regarding the second surgical instrument, and a fourth area showing information regarding the endoscope, which are arranged in a lower or upper end region of the screen.

8. the first area, the second area, the third area, and the fourth area are arranged and displayed in a lower end region of the screen, The surgical system according to claim 7 , wherein the lower side portion is positioned higher than the first area, the second area, the third area, and the fourth area.

9. 9. The surgical system according to claim 7 or 8, wherein when a plurality of the first surgical instrument, the replacement surgical instrument and the second surgical instrument are located outside the field of view of the endoscope, a plurality of the markers corresponding to the surgical instruments located outside the field of view of the endoscope are displayed in a manner corresponding to the order of the positions of the manipulator on which each of the first surgical instrument, the replacement surgical instrument and the second surgical instrument is provided.

10. The surgical system according to any one of claims 1 to 9, wherein the marker is configured to be switchable between a displayed state and a non-displayed state.

11. the remote control device includes an input unit for inputting a command for moving the endoscope, A surgical system according to any one of claims 1 to 10, wherein when a command enabling movement of the endoscope is input by the input unit, the endoscope is configured to be moved by operating both the first operating handle and the second operating handle.

12. The surgical system according to any one of claims 1 to 11, wherein the marker is displayed in a different manner when at least one of the first surgical instrument and the second surgical instrument located outside the field of view is the object of operation and when the instrument is not the object of operation.

13. The surgical system of any one of claims 1 to 12, wherein the markings include a number identifying the manipulator supporting at least one of the first surgical instrument and the second surgical instrument that is located outside the field of view.

14. The surgical system of any one of claims 1 to 13, wherein the markings further include an arrow indicating the direction in which at least one of the first surgical instrument and the second surgical instrument located outside the field of view is located.

15. a first manipulator for supporting a first surgical instrument; a second manipulator for supporting a second surgical instrument; a third manipulator for supporting an endoscope for taking images of the surgical site; a remote control device including a display device for displaying an image captured by the endoscope, a first operating handle for operating the first surgical instrument, a second operating handle for operating the second surgical instrument, and an endoscope pedal for executing movement of the endoscope by operating the first operating handle and the second operating handle; A control device, the control device is configured to display, on the display device, a graphical user interface for displaying a sign indicating at least one of the first surgical instrument and the second surgical instrument that is located outside the field of view in an outer edge area of ​​a center area of ​​a screen of the display device when the endoscope pedal is operated and at least one of the first surgical instrument and the second surgical instrument is located outside the field of view of the endoscope, superimposed on the image captured by the endoscope; the center area is a rectangular area including an upper side, a left side, a right side, and a lower side, the upper side portion is disposed between a position that is 1 / 10 of the vertical length of the screen from the center of the screen and a position that is 1 / 10 of the vertical length of the screen from the upper end of the screen, the left side portion is disposed between a position that is 1 / 10 of the horizontal length of the screen from the center of the screen to the left and a position that is 1 / 10 of the horizontal length of the screen from the left end of the screen to the right, the right side portion is disposed between a position on the right side of the center of the screen by a length of 1 / 10 of the horizontal length of the screen and a position on the left side of the right edge of the screen by a length of 1 / 10 of the horizontal length of the screen, A surgical system, wherein the lower edge portion is positioned between a position 1 / 10 of the vertical length of the screen below the center of the screen and a position 1 / 10 of the vertical length of the screen above the lower end of the screen.

16. obtaining an image captured by an endoscope for capturing an image of the surgical site; generating a graphical user interface and displaying the graphical user interface on a display device overlaid on an image captured by the endoscope; the step of displaying the graphical user interface includes, when a command to enable movement of the endoscope is received and at least one of a first surgical instrument supported by a first manipulator and a second surgical instrument supported by a second manipulator is located outside the field of view of the endoscope, a step of displaying on the display device the graphical user interface for displaying a sign indicating at least one of the first surgical instrument and the second surgical instrument located outside the field of view in an outer edge area of ​​a center area of ​​a screen of the display device, superimposed on the image captured by the endoscope; the center area is a rectangular area including an upper side, a left side, a right side, and a lower side, the upper side portion is disposed between a position that is 1 / 10 of the vertical length of the screen from the center of the screen and a position that is 1 / 10 of the vertical length of the screen from the upper end of the screen, the left side portion is disposed between a position that is 1 / 10 of the horizontal length of the screen from the center of the screen to the left and a position that is 1 / 10 of the horizontal length of the screen from the left end of the screen to the right, the right side portion is disposed between a position on the right side of the center of the screen by a length of 1 / 10 of the horizontal length of the screen and a position on the left side of the right edge of the screen by a length of 1 / 10 of the horizontal length of the screen, A display method in which the lower edge portion is positioned between a position 1 / 10 of the vertical length of the screen below the center of the screen and a position 1 / 10 of the vertical length of the screen above the lower end of the screen.

Images