Surgery system

Abstract

To provide a surgery system which allows an operator to easily confirm a position of a surgical instrument even when there is a positional deviation of the surgical instrument.SOLUTION: A surgical system 100 (surgery system) comprises: an arm 60c which supports an endoscope 6; arms 60a, 60b and 60d which support a medical instrument 4; and an image processing device 8b which includes one or a plurality of processors. The image processing device 8b acquires the tip position of the medical instrument 4, acquires a second imaging range A2 that reflects a positional deviation related to the tip position of the medical instrument 4 and the diameter of the shaft of the medical instrument 4 with respect to a first imaging range A1 corresponding to the field angle of an endoscope 6, and determines whether or not the tip position of the medical instrument 4 is within the second imaging range A2.SELECTED DRAWING: Figure 27

Description

Technical Field

[0001] The present invention relates to a surgical system, and particularly to a surgical system including a plurality of manipulators that respectively support an endoscope and a surgical instrument.

Background Art

[0002] Conventionally, a surgical system including a plurality of manipulators that respectively support an endoscope and a surgical instrument has been known (see, for example, Patent Document 1).

[0003] In the robotic surgical system of Patent Document 1, a surgical system including a plurality of manipulators that respectively support an endoscope and a tool (surgical instrument) and a control device is disclosed. In the surgical system of Patent Document 1, the control device determines whether the current position of the tip of the tool (surgical instrument) is outside the field of view of the endoscope, and configures to display information of the tool outside the field of view as a symbol (mark) in a boundary area outside the display area of the monitor screen.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the surgical system described in Patent Document 1, the tip position of the tool as determined by the control device may deviate significantly from its actual position due to manufacturing errors in the distance between the endoscope's manipulator and the tool's (surgical instrument's) manipulator, misalignment of the endoscope's field of view, and misalignment of the tool's tip position. In such cases, even if the endoscope is pointed in the direction indicated by the symbol (marker) to search for the tool, the tool may not be found. Furthermore, the relative positions of the endoscope and the tool can make it difficult for the operator to locate the tool. In addition, it is necessary to inform the operator whether the tip of the tool is outside the field of view or not in a way that does not cause discomfort to the operator. For these reasons, it can be difficult for the operator to easily confirm the position of the tool (surgical instrument).

[0006] This invention was made to solve the above-mentioned problems, and one of its objectives is to provide a surgical system that allows the operator to easily confirm the position of surgical instruments even when there is misalignment of the surgical instruments. [Means for solving the problem]

[0007] A surgical system according to the first aspect of this invention comprises a first manipulator for supporting an endoscope, a second manipulator for supporting a surgical instrument, and a control device including one or more processors, wherein the control device acquires the tip position of the surgical instrument, acquires a second imaging range that reflects the positional displacement related to the tip position of the surgical instrument and the diameter of the shaft of the surgical instrument with respect to a first imaging range corresponding to the field of view of the endoscope, and determines whether the tip position of the surgical instrument is within the second imaging range.

[0008] In the surgical system according to the first aspect of this invention, as described above, the control device acquires the tip position of the surgical instrument, acquires a second imaging range that reflects the positional displacement related to the tip position of the surgical instrument and the diameter of the shaft of the surgical instrument, relative to a first imaging range corresponding to the field of view of the endoscope, and determines whether the tip position of the surgical instrument is within the second imaging range. This makes it possible to determine whether the tip position of the surgical instrument is outside the field of view of the endoscope based on the second imaging range that reflects the positional displacement related to the tip position of the surgical instrument. As a result, when the position of the tip position of the surgical instrument relative to the field of view of the endoscope is uncertain due to a displacement of the tip position of the surgical instrument relative to the field of view of the endoscope, it is possible to suppress the indication of the direction of the uncertain tip position of the surgical instrument. Furthermore, since it is possible to determine whether the tip position of the surgical instrument is outside the field of view of the endoscope based on the second imaging range that reflects the diameter of the shaft of the surgical instrument, it is possible to determine whether the tip position of the surgical instrument, which has a certain size, is within the field of view of the endoscope, not just a specific point at the tip of the surgical instrument. As a result, even if there is a displacement of the surgical instrument, the operator can easily confirm the position of the surgical instrument.

[0009] A surgical system according to a second aspect of this invention comprises a first manipulator for supporting an endoscope, a second manipulator for supporting a surgical instrument, and a control device including one or more processors, wherein the control device is configured to control the first manipulator so that the endoscope moves with a first pivot position as its pivot, and to control the second manipulator so that the surgical instrument moves with a second pivot position as its pivot, and the control device acquires the respective tip positions of the endoscope and the surgical instrument, and provides a prompt to pull the endoscope forward if the distance between the tip position of the surgical instrument and the first pivot position is less than the distance between the tip position of the endoscope and the first pivot position.

[0010] In the surgical system according to the second aspect of this invention, as described above, the control device acquires the tip positions of the endoscope and the surgical instrument, and displays a prompt to pull the endoscope forward if the distance between the tip position of the surgical instrument and the first pivot position is smaller than the distance between the tip position of the endoscope and the first pivot position. As a result, if the distance between the tip position of the surgical instrument and the first pivot position is smaller than the distance between the tip position of the endoscope and the first pivot position, and even if the orientation of the endoscope is changed, the prompt to pull the endoscope forward is displayed, allowing the operator to easily find the tip position of the surgical instrument that is outside the field of view by pulling the endoscope forward. Consequently, even if there is a misalignment of the surgical instrument, the operator can easily confirm its position. [Effects of the Invention]

[0011] According to the present invention, even if there is a displacement of the surgical instrument, the operator can easily confirm the position of the surgical instrument. [Brief explanation of the drawing]

[0012] [Figure 1] This is a diagram showing the configuration of a surgical system according to an embodiment. [Figure 2] This figure shows the configuration of a medical manipulator according to an embodiment. [Figure 3] This diagram shows the configuration of the operating handle according to the embodiment. [Figure 4] This figure shows the configuration of the foot pedal according to the embodiment. [Figure 5] This figure shows the configuration of the arm of a medical manipulator according to an embodiment. [Figure 6] This is a diagram of forceps. [Figure 7] This is a perspective view showing the configuration of the operating section of a medical manipulator according to an embodiment. [Figure 8] This is a diagram showing an endoscope. [Figure 9] This is a diagram showing a pivot position teaching device. [Figure 10]This is a diagram for explaining the translational movement of the arm. [Figure 11] This is a diagram for explaining the rotational movement of the arm. [Figure 12] This is a block diagram showing the configuration of the control unit of the medical manipulator according to the embodiment. [Figure 13] This is a diagram showing an image captured by an endoscope and a graphical user interface. [Figure 14] This is a diagram showing an area of the graphical user interface. [Figure 15] This is a diagram for explaining the display of the clutch area. [Figure 16] This is a diagram for explaining the display of the medical device status information area. [Figure 17] This is a diagram for explaining the display of the left pop-up area. [Figure 18] This is a diagram for explaining the display of the right pop-up area. [Figure 19] This is a diagram for explaining the display of the touch panel of the remote operation device. [Figure 20] This is a diagram for explaining the label according to the embodiment. [Figure 21] This is a diagram for explaining the reflection of interference correction between the arms. [Figure 22] This is a diagram for explaining the misalignment of the forceps and the endoscope. [Figure 23] This is a table showing an example of the insertion amount and deviation angle of the forceps and the endoscope. [Figure 24] This is Figure 1 showing an example of the relationship of the deviation direction between the forceps and the endoscope. [Figure 25] This is Figure 2 showing an example of the relationship of the deviation direction between the forceps and the endoscope. [Figure 26] This is a diagram showing an example of the relationship between the imaging range of the endoscope and the deviation. [Figure 27] This is a diagram showing the first imaging range and the second imaging range of the endoscope. [Figure 28] This is a diagram showing an example when the tip position of the forceps and the pivot position of the endoscope are close. [Figure 29] This diagram illustrates an example where the tip of the forceps and the pivot position of the endoscope are far apart. [Figure 30] This is a flowchart illustrating the out-of-field detection process according to the embodiment. [Modes for carrying out the invention]

[0013] The following describes embodiments of the present invention based on the drawings.

[0014] Referring to Figures 1 to 30, the configuration of the surgical system 100 according to the embodiment will be described. The surgical system 100 comprises a medical manipulator 1, which is a patient P-side device, and a remote control device 2, which is an operator-side device for operating the medical manipulator 1. The medical manipulator 1 is equipped with a medical trolley 3 and is configured to be movable. The remote control device 2 is positioned at a distance from the medical manipulator 1, and the medical manipulator 1 is configured to be remotely controlled by the remote control device 2. The operator (such as a doctor) inputs a command to the remote control device 2 to cause the medical manipulator 1 to perform a desired action. The remote control device 2 transmits the input command to the medical manipulator 1. The medical manipulator 1 operates based on the received command. The medical manipulator 1 is also positioned in an operating room, which is a sterile field. Note that the surgical system 100 is an example of a "surgical system" in the claims.

[0015] The remote control device 2 is located, for example, inside or outside the operating room. The remote control device 2 includes an operating handle 21, a foot pedal 22, a touch panel 23, a monitor 24, a support arm 25, and a support bar 26. The operating handle 21 constitutes an operating handle for the operator (such as a doctor) to input commands.

[0016] The operating handle 21 is configured to operate the medical device 4. The operating handle 21 also receives input for the amount of manipulation applied to the medical device 4. The operating handle 21 includes an operating handle 21L located on the left side as viewed from the operator (such as a doctor) and operated by the operator's left hand, and an operating handle 21R located on the right side and operated by the operator's right hand.

[0017] Furthermore, as shown in Figure 3, the operating handle 21 includes a link section 21a, a link section 21b, a link section 21c, and a link section 21d operated by the operator (such as a doctor). The link section 21a rotates around the A4 axis. When the link section 21a rotates around the A4 axis, the arm section 61, which will be described later, rotates around the JT4 axis. The link section 21b rotates around the A5 axis relative to the link section 21a. When the link section 21b rotates around the A5 axis, the arm section 61, which will be described later, rotates around the JT5 axis. The link section 21c rotates around the A6 axis relative to the link section 21b. When the link section 21c rotates around the A6 axis, the arm section 61 rotates around the JT6 axis. The link section 21d rotates around the A7 axis relative to the link section 21c. As the link portion 21d rotates around the A7 axis, the arm portion 61 rotates around the JT7 axis. The medical instrument 4 is an example of a "surgical instrument" as defined in the claims.

[0018] Furthermore, the operating handle 21 modifies (scales) the amount of movement of the arm 60 (medical instrument 4) in relation to the amount of manipulation received by the operating handle 21. For example, if the multiplier for the amount of movement is set to 1 / 2, the medical instrument 4 is controlled to move a distance that is half the distance of movement of the operating handle 21. This allows for precise and detailed surgical procedures to be performed.

[0019] As shown in Figure 4, the foot pedals 22 are provided in multiple configurations to perform functions related to the medical device 4. The multiple foot pedals 22 are arranged on a base 28. The foot pedals 22 include a switching pedal 22a, a clutch pedal 22b, a camera pedal 22c, an incision pedal 22d, and a coagulation pedal 22e. The switching pedal 22a, clutch pedal 22b, camera pedal 22c, incision pedal 22d, and coagulation pedal 22e are operated by the operator's feet. The incision pedal 22d includes an incision pedal 22dR for the right arm 60 and an incision pedal 22dL for the left arm 60. The coagulation pedal 22e includes a coagulation pedal 22eR for the right arm 60 and a coagulation pedal 22eL for the left arm 60. The camera pedal 22c is an example of the "input device" and "third switch" in the claims.

[0020] The selector pedal 22a is configured to switch the arm 60 operated by the operating handle 21. In this embodiment, the clutch pedal 22b is configured to perform a clutch operation that temporarily disconnects the operating connection between the arm 60 and the operating handle 21. While the clutch pedal 22b is pressed down by the operator, the operation by the operating handle 21 is no longer transmitted to the arm 60.

[0021] The camera pedal 22c is operated to cause the two operating handles 21 to move the endoscope 6. Specifically, the camera pedal 22c is provided for inputting commands to move the endoscope 6. Specifically, when the camera pedal 22c is pressed down by the operator, a command to move the endoscope 6 is input. Furthermore, when a command enabling the movement of the endoscope 6 is input by the camera pedal 22c (i.e., while the camera pedal 22c is pressed down by the operator), the endoscope 6 is configured to move by operating both the operating handle 21R and the operating handle 21L.

[0022] While the cutting pedal 22d (coagulation pedal 22e) is pressed down by the operator, a high-frequency current flows from an electrosurgical device (not shown) to the medical instrument 4 (electrosurgical unit) to cut or coagulate the tissue.

[0023] As shown in Figure 1, the monitor 24 is a scope-type display device for displaying endoscopic images (see Figure 13) captured by the endoscope 6. The support arm 25 supports the monitor 24 so that its height is at the same height as the operator's (such as a doctor's) face. The touch panel 23 is located on the support bar 26. The medical manipulator 1 can be operated by the remote control device 2 by detecting the operator's head using a sensor (not shown) located near the monitor 24. The operator operates the operating handle 21 and foot pedal 22 while viewing the affected area on the monitor 24. This inputs a command to the remote control device 2. The command input to the remote control device 2 is transmitted to the medical manipulator 1.

[0024] The medical trolley 3 is equipped with a control unit 31 that controls the operation of the medical manipulator 1, and a storage unit 32 that stores programs for controlling the operation of the medical manipulator 1. Based on commands input to the remote control device 2, the control unit 31 of the medical trolley 3 controls the operation of the medical manipulator 1.

[0025] Furthermore, the medical trolley 3 is equipped with an input device 33. The input device 33 is configured to accept commands to move or change the posture of the positioner 40, arm base 50, and multiple arms 60, primarily for preparing for surgery before the procedure.

[0026] The medical manipulator 1 shown in Figures 1 and 2 is positioned in an operating room. The medical manipulator 1 comprises a medical trolley 3, a positioner 40, an arm base 50, and multiple arms 60. The arm base 50 is attached to the tip of the positioner 40. The arm base 50 has a relatively long rod shape (elongated shape). The base of each of the multiple arms 60 is attached to the arm base 50. The multiple arms 60 are configured to be able to assume a folded position (storage position). The arm base 50 and the multiple arms 60 are covered with a sterile drape (not shown) during use. The arms 60 also support medical instruments 4.

[0027] The positioner 40 is composed of, for example, a 7-axis articulated robot. The positioner 40 is also positioned on a medical trolley 3. The positioner 40 moves the arm base 50. Specifically, the positioner 40 is configured to move the position of the arm base 50 in three dimensions.

[0028] The positioner 40 also includes a base portion 41 and a plurality of link portions 42 connected to the base portion 41. The plurality of link portions 42 are connected to each other by joint portions 43.

[0029] As shown in Figure 1, a medical instrument 4 is attached to the tip of each of the multiple arms 60. The medical instrument 4 includes, for example, interchangeable instruments and an endoscope 6 (see Figure 8) for imaging the surgical site.

[0030] Furthermore, the surgical system 100 is equipped with a monitor cart 8, as shown in Figure 1. The monitor cart 8 is equipped with a display unit 8a. The display unit 8a is provided separately from the monitor 24 of the remote control device 2. The monitor cart 8 is also equipped with an image processing device 8b. The display unit 8a of the monitor cart 8 displays the same image as the image displayed on the monitor 24 of the remote control device 2. In other words, the image displayed on the monitor 24 and viewed by the surgeon can be viewed by workers (nurses, assistants, etc.) around the medical manipulator 1 and patient P via the display unit 8a of the monitor cart 8. Note that the image processing device 8b is an example of a "control device" as defined in the claims.

[0031] As shown in Figure 5, the instrument is equipped with a driven unit 4a, which is driven by a servo motor M2 located on the holder 71 of the arm 60. A forceps 4b is also provided at the tip of the instrument.

[0032] Furthermore, as shown in Figure 6, the instrument includes a first support 4e that rotatably supports the proximal ends of the end effector members 104a and 104b at the tip side around the JT11 axis, a second support 4f that rotatably supports the proximal end of the first support 4e at the tip side around the JT10 axis, and a shaft 4c connected to the proximal end of the second support 4f. The driven unit 4a, the shaft 4c, the second support 4f, the first support 4e, and the forceps 4b are arranged along the Z direction. The JT11 axis is perpendicular to the direction in which the shaft 4c extends (Z direction). The JT10 axis is spaced apart from the JT11 axis in the direction in which the shaft 4c extends and is perpendicular to the direction in which the shaft 4c extends and to the JT11 axis.

[0033] The first support 4e has a forceps 4b attached to it so as to rotate around the axis of the JT11 axis. The second support 4f supports the first support 4e so as to rotate around the JT10 axis. In other words, the first support 4e is attached to the second support 4f so as to rotate around the axis of the JT10 axis. The tip side (Z1 direction side) of the first support 4e has a U shape. A tool center point (TCP1, clevis) is set in the center of the U-shaped tip side of the first support 4e on the axis of the JT11 axis.

[0034] Furthermore, the medical instrument 4 (forceps 4b) is equipped with a JT9 axis as the rotation axis of the shaft 4c (the axis along the direction in which the shaft 4c extends) and a JT12 axis as the opening and closing axis of the forceps 4b. Multiple servo motors (for example, four) are provided on the holder 71 of the arm 60, and the rotating body of the driven unit 4a is driven by these multiple servo motors M2. This drives the medical instrument 4 around the J9 to J12 axes.

[0035] Furthermore, as shown in Figure 8, TCP2 of the endoscope 6 is set at the tip of the endoscope 6.

[0036] Next, we will describe the configuration of arm 60 in detail.

[0037] As shown in Figure 5, the arm 60 includes an arm section 61 (base section 62, link section 63, joint section 64) and a translational movement mechanism section 70 provided at the tip of the arm section 61. The arm 60 is configured to move its tip in three dimensions relative to its root side (arm base 50). The arm section 61 is composed of a 7-axis articulated robot arm. Multiple arms 60 have similar configurations.

[0038] As shown in Figure 5, the arm 60 is equipped with JT1 to JT7 axes as rotation axes and JT8 axis as a linear motion axis. The JT1 to JT7 axes correspond to the rotation axes of the joint 64 of the arm 61. The JT7 axis corresponds to the base end link portion 72 of the translational movement mechanism 70. The JT8 axis corresponds to the axis that moves the tip end link portion 73 of the translational movement mechanism 70 relative to the base end link portion 72 along the Z direction. That is, the servo motor M1 shown in Figure 12 is provided to correspond to the JT1 to JT7 axes of the arm 60. The servo motor M3 is provided to correspond to the JT8 axis.

[0039] The translational movement mechanism 70 is provided at the tip of the arm 61 and the medical instrument 4 is attached to it. The translational movement mechanism 70 translates the medical instrument 4 in the direction of insertion into the patient P. The translational movement mechanism 70 is configured to translate the medical instrument 4 relative to the arm 61. Specifically, the translational movement mechanism 70 is provided with a holder 71 for holding the medical instrument 4. A servo motor M2 (see Figure 12) is housed in the holder 71.

[0040] As shown in Figure 7, the medical manipulator 1 is equipped with an arm operating unit 80 attached to the arm 60 for operating the arm 60. The arm operating unit 80 includes an enable switch 81 and a joystick 82 and a switch unit 83. The enable switch 81 permits or dispermits the movement of the arm 60 by the joystick 82 and the switch unit 83. The enable switch 81 is also used to permit the movement of the medical instrument 4 by the arm 60 when an operator (nurse, assistant, etc.) grasps and presses the arm operating unit 80. The joystick 82 and the switch unit 83 are used to operate the arm 60. The enable switch 81 is an example of the "input device" and "second switch" in the claims.

[0041] Furthermore, the switch unit 83 includes a switch unit 83a that moves the medical device 4 toward the direction of insertion into the patient P along the longitudinal direction of the medical device 4, and a switch unit 83b that moves the medical device 4 toward the opposite direction from the direction of insertion into the patient P. Both the switch unit 83a and the switch unit 83b are composed of push-button switches.

[0042] Furthermore, as shown in Figure 7, the arm operating unit 80 includes a pivot button 85 that teaches the pivot position PP, which serves as the fulcrum (see Figure 11) for the movement of the medical instrument 4 attached to the arm 60. The pivot button 85 is provided on the surface 80b of the arm operating unit 80, adjacent to the enable switch 81. When the tip of the endoscope 6 (see Figure 8) or the pivot position teaching device 7 (Figure 9) is moved to a position corresponding to the insertion position of the trocar T inserted into the patient P's body surface S, the pivot position PP is taught when the pivot button 85 is pressed and stored in the memory unit 32. Note that in teaching the pivot position PP, the pivot position PP is set as a single point (coordinate), and teaching the pivot position PP does not set the direction of the medical instrument 4.

[0043] As shown in Figure 1, an endoscope 6 is attached to one of the multiple arms 60 (for example, arm 60c), and other medical instruments 4 are attached to the remaining arms 60 (for example, arms 60a, 60b, and 60d). Specifically, during surgery, an endoscope 6 is attached to one of the four arms 60, and other medical instruments 4 (such as forceps 4b) are attached to the other three arms 60. The pivot position PP is then taught to the arm 60 to which the endoscope 6 is attached, with the endoscope 6 attached. The pivot position PP is also taught to the arms 60 to which the other medical instruments 4 are attached, with the pivot position teaching device 7 attached. The endoscope 6 is attached to one of the two centrally located arms 60 (arms 60b and 60c) of the four arms 60 which are arranged adjacent to each other. In other words, the pivot position PP is set individually for each of the multiple arms 60. Arm 60c is an example of the "first manipulator" in the claims. Arms 60a, 60b, and 60d are examples of the "second manipulator" in the claims.

[0044] Furthermore, as shown in Figure 7, an adjustment button 86 for optimizing the position of the arm 60 is provided on the surface 80b of the arm operating section 80. After teaching the pivot position PP for the arm 60 to which the endoscope 6 is attached, pressing the adjustment button 86 optimizes the position of the other arm 60 (arm base 50).

[0045] Furthermore, as shown in Figure 7, the arm operation unit 80 includes a mode switching button 84 that switches between a mode for translating (see Figure 10) and a mode for rotating (see Figure 11) the medical instrument 4 attached to the arm 60. A mode indicator 84a is also provided near the mode switching button 84. The mode indicator 84a displays the switched mode. Specifically, the current mode (translation mode or rotation mode) is displayed by whether the mode indicator 84a lights up (rotation mode) or turns off (translation mode).

[0046] Furthermore, the mode indicator 84a also functions as a pivot position indicator, showing that the pivot position PP has been taught.

[0047] As shown in Figure 10, in the mode in which the arm 60 is translated, the arm 60 is moved so that the tip 4d of the medical instrument 4 moves on the XY plane. Also, as shown in Figure 11, in the mode in which the arm 60 is rotated, when the pivot position PP is not taught, the arm 60 rotates around the forceps 4b, and when the pivot position PP is taught, the arm 60 is moved so that the medical instrument 4 rotates with the pivot position PP as the fulcrum. Note that the medical instrument 4 is rotated while its shaft 4c is inserted into the trocar T.

[0048] Furthermore, as shown in Figure 12, the arm 60 is equipped with multiple servo motors M1, an encoder E1, and a reduction gear (not shown) corresponding to the multiple joints 64 of the arm section 61. The encoder E1 is configured to detect the rotation angle of the servo motor M1. The reduction gear is configured to reduce the rotation of the servo motor M1 and increase the torque.

[0049] Furthermore, as shown in Figure 12, the translational movement mechanism 70 is equipped with a servo motor M2 for rotating a rotating body provided on the driven unit 4a of the medical instrument 4, a servo motor M3 for translating the medical instrument 4, encoders E2 and E3, and a reduction gear (not shown). Encoders E2 and E3 are configured to detect the rotation angles of servo motors M2 and M3, respectively. The reduction gear is configured to reduce the rotation of servo motors M2 and M3 to increase torque.

[0050] Furthermore, the positioner 40 is equipped with multiple servo motors M4, an encoder E4, and a reduction gear (not shown) to correspond to the multiple joints 43 of the positioner 40. The encoder E4 is configured to detect the rotation angle of the servo motors M4. The reduction gear is configured to reduce the rotation of the servo motors M4 and increase the torque.

[0051] The medical cart 3 is equipped with a servo motor M5, an encoder E5, a reduction gear (not shown), and a brake, each of which drives one of the multiple front wheels (not shown). The reduction gear is configured to reduce the rotation of the servo motor M5 and increase the torque. The throttle section 34a of the medical cart 3 is equipped with a potentiometer P1 (see Figure 1), and the servo motors M5 of the front wheels are driven based on the rotation angle detected by the potentiometer P1 in response to the twisting of the throttle section 34a. The rear wheels (not shown) of the medical cart 3 are of the twin-wheel type, and the rear wheels are steered based on the left and right rotation of the operating handle 34. The operating handle 34 of the medical cart 3 is equipped with a potentiometer P2 (see Figure 2), and the rear wheels of the medical cart 3 are equipped with a servo motor M6, an encoder E6, and a reduction gear (not shown). The reduction gear is configured to reduce the rotation of the servo motor M6 and increase the torque. The servo motor M6 is driven based on the rotation angle detected by the potentiometer P2 in response to the left and right rotation of the control handle 34. In other words, the steering of the rear wheels caused by the left and right rotation of the control handle 34 is power-assisted by the servo motor M6.

[0052] Furthermore, the medical trolley 3 moves in the forward and backward direction when its front wheels are driven. Also, when the operating handle 34 of the medical trolley 3 is rotated, the rear wheels are steered, causing the medical trolley 3 to rotate in the left and right direction.

[0053] The control unit 31 of the medical trolley 3 includes an arm control unit 31a that controls the movement of a plurality of arms 60 based on a command, and a positioner control unit 31b that controls the movement of the positioner 40 and the driving of the front wheels (not shown) of the medical trolley 3 based on a command. The arm control unit 31a is electrically connected to a servo control unit C1 for controlling a servo motor M1 for driving the arms 60. The servo control unit C1 is also electrically connected to an encoder E1 for detecting the rotation angle of the servo motor M1. Note that the control unit 31 is an example of a "control device" as defined in the claims.

[0054] Furthermore, a servo control unit C2 for controlling a servo motor M2 for driving the medical device 4 is electrically connected to the arm control unit 31a. An encoder E2 for detecting the rotation angle of the servo motor M2 is electrically connected to the servo control unit C2. Furthermore, a servo control unit C3 for controlling a servo motor M3 for translating the translational movement mechanism 70 is electrically connected to the arm control unit 31a. An encoder E3 for detecting the rotation angle of the servo motor M3 is electrically connected to the servo control unit C3.

[0055] The motion command input to the remote control device 2 is then input to the arm control unit 31a. The arm control unit 31a generates a position command based on the input motion command and the rotation angle detected by the encoder E1 (E2, E3), and outputs the position command to the servo control unit C1 (C2, C3). The servo control unit C1 (C2, C3) generates a torque command based on the position command input from the arm control unit 31a and the rotation angle detected by the encoder E1 (E2, E3), and outputs the torque command to the servo motor M1 (M2, M3). As a result, the arm 60 moves in accordance with the motion command input to the remote control device 2. The control unit 31 also controls the arm 60c so that the endoscope 6 moves with the pivot position PP as the pivot point. The control unit 31 also controls the arm 60a (60b, 60d) so that the medical instrument 4 moves with the pivot position PP as the pivot point.

[0056] Furthermore, as shown in Figure 12, the control unit 31 (arm control unit 31a) is configured to operate the arm 60 based on input signals from the joystick 82 of the arm operation unit 80. Specifically, the arm control unit 31a generates a position command based on the input signal (operation command) received from the joystick 82 and the rotation angle detected by the encoder E1, and outputs the position command to the servo control unit C1. The servo control unit C1 generates a torque command based on the position command received from the arm control unit 31a and the rotation angle detected by the encoder E1, and outputs the torque command to the servo motor M1. As a result, the arm 60 is moved in accordance with the operation command input to the joystick 82.

[0057] The control unit 31 (arm control unit 31a) is configured to operate the arm 60 based on input signals from the switch unit 83 of the arm operation unit 80. Specifically, the arm control unit 31a generates a position command based on the input signal (operation command) received from the switch unit 83 and the rotation angle detected by the encoder E1 or E3, and outputs the position command to the servo control unit C1 or C3. The servo control unit C1 or C3 generates a torque command based on the position command received from the arm control unit 31a and the rotation angle detected by the encoder E1 or E3, and outputs the torque command to the servo motor M1 or M3. As a result, the arm 60 is moved in accordance with the operation command input to the switch unit 83.

[0058] Furthermore, as shown in Figure 12, a servo control unit C4 for controlling the servo motor M4 that moves the positioner 40 is electrically connected to the positioner control unit 31b. An encoder E4 for detecting the rotation angle of the servo motor M4 is electrically connected to the servo control unit C4. In addition, a servo control unit C5 for controlling the servo motor M5 that drives the front wheels (not shown) of the medical trolley 3 is electrically connected to the positioner control unit 31b. An encoder E5 for detecting the rotation angle of the servo motor M5 is electrically connected to the servo control unit C5.

[0059] Furthermore, operation commands related to setting the preparation position are input from the input device 33 to the positioner control unit 31b. The positioner control unit 31b generates a position command based on the operation command input from the input device 33 and the rotation angle detected by the encoder E4, and outputs the position command to the servo control unit C4. The servo control unit C4 generates a torque command based on the position command input from the positioner control unit 31b and the rotation angle detected by the encoder E4, and outputs the torque command to the servo motor M4. As a result, the positioner 40 is moved in accordance with the operation command input to the input device 33. Similarly, based on the operation command from the input device 33, the positioner control unit 31b moves the medical trolley 3.

[0060] The surgical system 100 includes an image processing device 8b. The image processing device 8b performs processing based on a predetermined program. The image processing device 8b is composed of a computer. The image processing device 8b includes a processing unit such as a CPU that executes the program, and a storage unit such as memory where the program is stored. The image processing device 8b is configured to generate a graphical user interface G (see Figure 14) and display the graphical user interface G superimposed on the endoscopic image (see Figure 13) captured by the endoscope 6 on the monitor 24 of the remote control device 2. The image processing device 8b is also configured to display the graphical user interface G superimposed on the endoscopic image (see Figure 13) captured by the endoscope 6 on the display unit 8a. The image processing device 8b is also configured to acquire images from the endoscope 6. The image processing device 8b is also configured to communicate with the control unit 31.

[0061] As shown in Figure 14, the graphical user interface G includes a clutch area G1. As shown in Figure 15, the clutch area G1 displays the state of the clutch pedal 22b. Figure 15(a) shows the state when the clutch pedal 22b is not pressed (OFF state). Figure 15(b) shows the state when the operator has their foot on the clutch pedal 22b (hover state). Figure 15(c) shows the state when the clutch pedal 22b is pressed (ON state).

[0062] As shown in Figure 14, in this embodiment, the graphical user interface G includes a camera area G2 that displays information about the endoscope 6. The camera area G2 is displayed in the area near the lower edge ed (see Figure 13) of the screen gr of the monitor 24.

[0063] As shown in Figure 14, the graphical user interface G includes a hand area G3. The hand area G3 displays information about each medical instrument 4, each arm 60, and the status of the coagulation pedal 22e and the incision pedal 22d. The hand area G3 includes hand area G3a, which shows information about the medical instrument 4 and arm 60a (number "4" in hand area G3) operated by the left-hand operating handle 21L; hand area G3b, which shows information about the replacement medical instrument 4 and arm 60b (number "3" in hand area G3); and hand area G3c, which shows information about the medical instrument 4 and arm 60d (number "1" in hand area G3) operated by the left-hand operating handle 21L. The hand area G3 is displayed in the area near the lower edge ed of the screen gr of the monitor 24. The clutch area G1 is also displayed in the area near the lower edge ed of the screen gr. Specifically, as shown in Figure 14, the clutch area G1, camera area G2, and hand area G3 are displayed between the lower edge ed of the screen gr and a position above the edge ed by a length of 1 / 10 (L11) of the vertical length of the screen gr.

[0064] Furthermore, information regarding arm 60 includes the arm number (such as "1" or "2") and an arrow icon displayed when it is set as the replacement target for arm 60 to which a replacement medical device 4 is attached; information regarding medical device 4 includes the name of medical device 4; and information regarding the status of the coagulation pedal 22e and incision pedal 22d includes the operating status of the clutch pedal 22b, the operating status of the incision pedal 22d, and the operating status of the coagulation pedal 22e.

[0065] Furthermore, as shown in Figure 13, the hand areas of the arm 60 to be operated (hand area G3a and hand area G3c) are displayed in dark gray, with a white oval outlined within them, and the arm 60 numbers (1 and 4) displayed within the white oval outline. The hand area of ​​the arm 60 not to be operated (hand area G3b) is displayed in light gray, and the arm 60 number (3) is displayed in an even lighter gray.

[0066] As shown in Figure 14, the graphical user interface G includes a pop-up area, the medical device status information area G4. The medical device status information area G4 displays the current number of uses / maximum number of uses (see Figure 16) of the medical device 4 attached to each arm 60 as a pop-up. When the current number of uses equals the maximum number of uses, the current number of uses is displayed in red. In addition, if an error occurs in the medical device 4 attached to each arm 60, error information is displayed as a pop-up. If no medical device 4 is attached to the arm 60, nothing is displayed in the medical device status information area G4. The medical device status information area G4 is displayed on the monitor 24 in an adjacent area above the clutch area G1, camera area G2, and hand area G3.

[0067] As shown in Figure 14, the graphical user interface G includes a level indicator area G5. The level indicator area G5 displays the angle information of the endoscope 6. The level indicator area G5 is also displayed only while the camera pedal 22c is pressed down. That is, when the image processing device 8b receives a command to enable movement of the endoscope 6, it displays the level indicator LV of the endoscope 6 in the level indicator area G5.

[0068] As shown in Figure 14, the graphical user interface G includes a left pop-up area G6. When the foot is on the foot pedal 22 in a hover state, the icons shown in Figures 17(a) to (c) are displayed in the left pop-up area G6. Figure 17(a) shows the icons displayed when the foot is on the coagulation pedal 22eL and the incision pedal 22dL. Figure 17(b) shows the icons displayed when the foot is on the clutch pedal 22b. Figure 17(c) shows the icons displayed when the foot is on the camera pedal 22c. The left pop-up area G6 is displayed on the left side of the monitor 24.

[0069] As shown in Figure 14, the graphical user interface G includes a right pop-up area G7. The right pop-up area G7 displays icons (Figure 18) when the feet are placed on the coagulation pedal 22eR and the incision pedal 22dR. The right pop-up area G7 is displayed on the right side of the monitor 24.

[0070] Furthermore, as shown in Figure 14, the graphical user interface G includes a first area G8 that displays the range of motion of the arm 60 and the range of motion of the arm 60 that can be operated by the operating handle 21. The graphical user interface G also includes a second area G9 that displays the direction of operation of the operating handle 21 necessary to return the operating handle 21 to the operating range and / or (in this embodiment, "and") return the arm 60 to the operating range.

[0071] Furthermore, there are two arms 60 that can be operated by the operating handle 21. For example, the left arm 60L (e.g., arm 60a, see Figure 1) for supporting the medical instrument 4 is operated by the operating handle 21L. The right arm 60R (e.g., arm 60d, see Figure 1) for supporting the medical instrument 4 is operated by the operating handle 21R. The first area G8 (first area G8L, first area G8R) and the second area G9 (second area G9L, second area G9R) are provided separately for the left arm 60L and the right arm 60R, respectively.

[0072] Furthermore, as shown in Figure 14, the graphical user interface G includes error notification areas G15 (G15a, G15b). Error notification area G15a is displayed as a pop-up to show warning or error information when a warning or error occurs. Error notification area G15b is displayed as a pop-up to show details of the notes related to the warning or error displayed in error notification area G15a.

[0073] The Around Display AR, shown in Figure 13, which indicates whether a medical device 4 is within or outside the field of view of the endoscope 6, is configured to be switchable between being displayed and not displayed. Specifically, as shown in Figure 19, the "Around Display" button is displayed when the touch panel 23 of the remote control device 2 is operated. When "On" is selected for "Around Display," the Around Display AR is displayed when a predetermined input operation is performed on the camera pedal 22c by the operator, if a medical device 4 is located within or outside the field of view of the endoscope 6. When "Off" is selected for "Around Display," the Around Display AR is not displayed even if a predetermined input operation is performed on the camera pedal 22c by the operator. This allows the "Around Display" setting to be easily switched according to the operator's skill level and needs. The marker MK1, which indicates that a medical device 4 is outside the field of view of the endoscope 6, may also be changed to not be displayed according to the operator's settings.

[0074] As shown in Figure 14, the graphical user interface G includes a status area G10. The status area G10 displays information such as the remaining battery level of the medical manipulator 1, the brightness / contrast of the monitor 24, lap time, and elapsed time of surgery. Specifically, as shown in Figure 14, the status area G10 is displayed between the upper edge eu of the screen gr and a position below the edge eu by a length of 1 / 10 (L11) of the vertical length of the screen gr.

[0075] Here, the control unit 31 is configured to determine whether the multiple medical instruments 4, each supported by a plurality of arms 60, are located outside the field of view of the endoscope 6, based on the imaging range information of the endoscope 6 and the position information of the tip of each of the multiple medical instruments 4. In other words, the control unit 31 acquires the position of the medical instruments 4 based on the posture and position of the arms 60. The control unit 31 also acquires the imaging direction of the endoscope 6 based on the posture and position of the arms 60. The control unit 31 also acquires the angle of view (field of view) of the endoscope 6 based on the zoom state of the endoscope 6. The control unit 31 acquires the angle of view (field of view) of the endoscope 6 using a value set as the mechanical mechanism (lens, etc.) of the endoscope 6. Then, the control unit 31 acquires the coordinates of the tip of the medical instrument 4 relative to the field of view of the endoscope 6 from the field of view of the endoscope 6, the posture and position of the endoscope 6, and the position information of the arms 60. As a result, the control unit 31 determines whether the medical instruments 4 are located outside the field of view of the endoscope 6. Furthermore, the image processing device 8b acquires information from the control unit 31 regarding the determination result of whether or not the medical instrument 4 is located outside the field of view of the endoscope 6, and generates a graphical user interface G based on the acquired information.

[0076] As shown in Figures 13 and 14, when at least one of the multiple medical instruments 4 is located outside the field of view of the endoscope 6, the image processing device 8b responds to a predetermined input from the camera pedal 22c or enable switch 81 and displays a marker MK1 indicating the medical instrument 4 located outside the field of view in the inner area of ​​the graphical user interface G, which does not include the vicinity of the edge of the display screen. In other words, the image processing device 8b acquires information from the control unit 31 that at least one of the multiple medical instruments 4 is located outside the field of view of the endoscope 6, generates a graphical user interface G including a marker MK1 indicating the medical instrument 4 located outside the field of view, and displays it. Furthermore, the image processing device 8b is configured to display the marker MK1 in the area G11 near the outer edge of the level indicator area G5 of the graphical user interface G. In other words, the marker MK1 (graphical user interface G) is displayed in the area G11 (marker display area) near the outer edge of the level indicator area G5, which includes the central part CN1 but does not include the vicinity of the edge e of the screen gr of the monitor 24. The outer edge vicinity area G11 is the area located outside and adjacent to the spirit level LV, which will be described later. Marker MK1 is an example of the "first marking" in the claims.

[0077] Furthermore, the image processing device 8b is configured to display the indicator MK1 on the graphical user interface G in response to the operation of the enable switch 81 attached to the arm 60. In other words, when an operator such as an assistant or nurse presses the enable switch 81 to move the arm 60, the indicator MK1 is displayed on the graphical user interface G.

[0078] Furthermore, the marker MK1 includes an arrow indicating the direction in which a medical device 4 located outside the field of view is located. Specifically, a number (for example, "3") is displayed inside the arrow to identify the medical device 4 located outside the field of view. The number may also be located outside the arrow. The inside of the arrow is displayed transparent or semi-transparent so that the image is visible through it. For medical devices 4 that are to be operated, the number on the marker MK1 is shown as the number of the black arm 60 inside a white circle. For medical devices 4 that are not to be operated, the number on the marker MK1 is shown as the number of the gray arm 60 inside a black circle.

[0079] In this embodiment, as shown in Figure 27, the control unit 31 acquires the tip position of the medical instrument 4 and acquires a second imaging range A2 that reflects the positional displacement related to the tip position of the medical instrument 4 and the diameter of the shaft of the medical instrument 4, relative to the first imaging range A1 corresponding to the field of view of the endoscope 6. The control unit 31 then determines whether the tip position of the medical instrument 4 is within the second imaging range A2. In other words, the control unit 31 determines whether the tip of the medical instrument 4 is within the second imaging range A2, which is wider than the first imaging range A1 corresponding to the field of view of the endoscope. Here, the tip position of the medical instrument 4 is the tool center point position (TCP) (see Figure 6), which is the control center when moving the medical instrument 4.

[0080] As shown in Figure 13, if the control unit 31 determines that the tip position of the medical instrument 4 is outside the second imaging range A2, the image processing device 8b displays a marker MK1 indicating that the medical instrument 4 is outside the field of view of the endoscope 6.

[0081] Furthermore, in this embodiment, if the control unit 31 determines that the tip position of the medical instrument 4 is within the second imaging range A2, the image processing device 8b displays an around view AR indicating that the medical instrument 4 is within the field of view of the endoscope 6 or in the peripheral area outside the field of view, where the tip position of the medical instrument 4 would be within the field of view if the endoscope 6 were pulled forward. Note that the around view AR is an example of the "second display" in the claims.

[0082] Furthermore, if the control unit 31 determines that the tip of the medical instrument 4 is within the second imaging range A2, the image processing device 8b displays a prompt to pull the endoscope 6 forward. In other words, when the operator makes a predetermined input using the camera pedal 22c or the enable switch 81, and the around view display AR is displayed, if the corresponding medical instrument 4 is not in the image of the endoscope 6, the operator will recognize that the corresponding medical instrument 4 is located in one of the directions if the endoscope 6 is pulled forward. As a result, the around view display AR prompts the operator to pull the endoscope 6 forward.

[0083] Furthermore, the Around Display AR includes a number that identifies the arm 60 supporting the medical device 4 that is determined to be within the second imaging range A2. In the example shown in Figure 13, the Around Display AR includes the number "4" indicating that the medical device 4 supported by arm 60a (number "4" in hand area G3) is located within the second imaging range A2, and the number "1" indicating that the medical device 4 supported by arm 60d (number "1" in hand area G3) is located within the second imaging range A2.

[0084] Furthermore, as shown in Figure 20, the image processing device 8b is configured to display the around view AR in an area adjacent to the outer edge vicinity area G11 of the level indicator area G5 of the graphical user interface G. For example, the image processing device 8b displays the around view AR above the outer edge vicinity area G11. The image processing device 8b also displays the around view AR in an adjacent area outside the outer edge vicinity area G11 in the left-right direction. For example, the image processing device 8b displays the around view AR to the left of the outer edge vicinity area G11 in the left-right direction.

[0085] Furthermore, if the control unit 31 determines that the tip position of the medical instrument 4 is outside the second imaging range A2, the image processing device 8b displays a marker MK1 indicating that the medical instrument 4 is outside the field of view of the endoscope 6 in response to a predetermined input to the camera pedal 22c or the enable switch 81. Also, if the control unit 31 determines that the tip position of the medical instrument 4 is within the second imaging range A2, the image processing device 8b displays an around indicator AR indicating that the medical instrument 4 is within the field of view of the endoscope 6 or in the peripheral area outside the field of view that would be within the field of view if the endoscope 6 were pulled forward.

[0086] As shown in Figure 28, the control unit 31 acquires the tip positions of both the endoscope 6 and the medical instrument 4. The control unit 31 also determines whether the distance D2 between the tip position of the medical instrument 4 and the pivot position PP1 is greater than the distance D1 between the tip position of the endoscope 6 and the pivot position PP1. If the control unit 31 determines that the distance D2 between the tip position of the medical instrument 4 and the pivot position PP1 is less than the distance D1 between the tip position of the endoscope 6 and the pivot position PP1, the image processing device 8b displays a message prompting the user to pull the endoscope 6 towards them.

[0087] Furthermore, the image processing device 8b displays an around view (AR) to encourage the user to pull the endoscope 6 forward, indicating that when the endoscope 6 is pulled forward, the tip of the medical instrument 4 will be in the peripheral area outside the field of view.

[0088] Furthermore, as shown in Figure 29, when the tip of the medical instrument 4 is in front of the tip of the endoscope 6, and the control unit 31 determines that the distance between the tip of the medical instrument 4 and the pivot position PP1 is greater than the distance between the tip of the endoscope 6 and the pivot position PP1, the image processing device 8b displays an indication of the direction of the tip of the medical instrument 4 relative to the imaging range of the endoscope 6 (display of marker MK1).

[0089] Furthermore, the control unit 31 corrects for positional misalignment caused by errors in the spacing between arm 60c and arms 60a, 60b, or 60d to obtain the respective tip positions of the endoscope 6 and the medical instrument 4. For example, as shown in Figure 21, the control unit 31 obtains a corrected arm-base matrix based on the interference correction value and the arm-base matrix. The control unit 31 also obtains a TCP matrix as seen from the arm base based on the arm-base matrix and the TCP matrix. The control unit 31 also obtains a flange surface matrix as seen from the arm base based on the arm-base matrix and the flange surface matrix. Then, the control unit 31 obtains a corrected TCP matrix based on the corrected arm-base matrix and the TCP matrix as seen from the arm base. The control unit 31 also obtains a corrected flange surface matrix based on the corrected arm-base matrix and the flange surface matrix as seen from the arm base.

[0090] The positional displacement related to the tip position of the medical instrument 4 includes a first positional displacement caused by the positional displacement of the field of view of the endoscope 6, and a second positional displacement which is the positional displacement of the tip of the medical instrument 4. The control unit 31 calculates the first positional displacement based on the positional displacement at the pivot position PP1 of the endoscope 6 and the distance of the tip of the endoscope 6 from the pivot position PP1. The control unit 31 also calculates the second positional displacement based on the positional displacement at the pivot position PP2 of the medical instrument 4 and the distance of the tip of the medical instrument 4 from the pivot position PP2.

[0091] Specifically, as shown in Figure 22, the control unit 31 calculates the amount of displacement based on the length L1 of the shaft of the medical instrument 4 (endoscope 6), the distance L2 from the pivot position PP at the tip of the medical instrument 4 (endoscope 6) (pivot depth), and the displacement radius L3. Here, since the taught pivot position PP is the inner circumferential surface of the trocar T, there is a displacement from the actual pivot position PP by the radius of the medical instrument 4 (endoscope 6). Therefore, the displacement radius L3 is the length corresponding to the radius of the medical instrument 4 (endoscope 6). Then, the displacement angle θ is calculated based on L1, L2, and L3. The displacement angle θ is calculated by θ = atan(L3 / (L1-L2)). Since L1 and L3 are constant values ​​for each medical instrument 4 (endoscope 6), the displacement angle θ changes as the distance L2 from the pivot position PP at the tip of the medical instrument 4 (endoscope 6) changes.

[0092] For example, the displacement angle θ changes with the change in pivot depth L2, as shown in Figure 23. In the example shown in Figure 23, the shaft length L1 of medical instrument 4 (forceps) is 507 mm and the displacement radius L3 is 4 mm. Also, the shaft length L1 of endoscope 6 is 498 mm and the displacement radius L3 is 6 mm.

[0093] The displacement L4 of the endoscope 6 in the field of view is calculated by L4 = (L1 + depth from endoscope 6 to medical instrument 4) × sinθ. Also, as shown in Figure 22, the displacement L4 of the tip position (TCP position) of medical instrument 4 is obtained by L4 = L1 × sinθ.

[0094] Furthermore, the effect of the tip displacement of the medical instrument 4 on the field of view of the endoscope 6 changes depending on the insertion direction of the medical instrument 4 relative to the field of view, so a correction is performed. As shown in Figure 24, the direction of displacement differs depending on whether the medical instrument 4 is inserted diagonally to the field of view of the endoscope 6 or laterally to the field of view of the endoscope 6, as shown in Figure 25. For example, if the medical instrument 4 is inserted perpendicularly from the side to the field of view of the endoscope 6, the position of the medical instrument 4 will not shift in the lateral direction of the field of view (the insertion direction of the medical instrument 4). The insertion direction of the medical instrument 4 is calculated based on two points: the TCP position of the medical instrument 4 and the position obtained by shifting the flange surface by the shaft offset amount.

[0095] Furthermore, the amount of displacement applied to the field of view is determined based on the amount of displacement in the insertion direction of medical device 4, the amount of displacement 90 degrees from the insertion direction of medical device 4, the amount of displacement 45 degrees from the insertion direction of medical device 4, and the amount of displacement -45 degrees from the insertion direction of medical device 4. The maximum values ​​in the vertical and horizontal directions are then corrected and used as the respective displacement amounts.

[0096] As shown in Figure 26, a third imaging range A3 is calculated that takes into account the positional displacement related to the tip position of the medical instrument 4, including a first positional displacement caused by the positional displacement of the endoscope 6's field of view and a second positional displacement, which is the positional displacement of the tip of the medical instrument 4, relative to the first imaging range A1 corresponding to the field of view of the endoscope 6. If the amount of displacement is large, it may not be possible to determine the direction of the out-of-field view (the direction of the out-of-field view may change depending on the direction of the displacement).

[0097] Furthermore, the calculation of the third imaging range A3 relative to the first imaging range A1 is performed separately for each of the multiple medical devices 4. In other words, the amount of displacement of each medical device 4 is calculated from the insertion distance and insertion angle of each medical device 4.

[0098] Furthermore, as shown in Figure 27, a third imaging range A3 is acquired that takes into account the positional displacement of the endoscope 6's field of view and the positional displacement of the medical instrument 4's tip relative to the first imaging range A1, and a second imaging range A2 is acquired that reflects the diameter of the medical instrument 4's shaft. The second imaging range A2 is larger vertically and horizontally than the third imaging range A3 by the diameter of the medical instrument 4's shaft. If the point of the medical instrument 4's tip position (TCP position) falls within the second imaging range A2, it is considered to be within the range.

[0099] Furthermore, in the examples shown in Figures 13 and 14, among arms 60a, 60b, 60c, and 60d, arm 60a (number "4" in hand area G3a), arm 60b (number "3" in hand area G3b), and arm 60d (number "1" in hand area G3c) have medical instruments 4 (medical instruments 4 other than the endoscope 6, such as forceps 4b). In addition, the endoscope 6 is attached to arm 60c (number "2" in camera area G2). Arm 60a (number "4" in hand area G3a) and arm 60d (number "1" in hand area G3) are in an operable state (active) by the operating handle 21, and hand areas G3a and G3c are displayed in dark gray. Arm 60b is in an inactive state (not operable by the operating handle 21), and hand area G3b is displayed in light gray.

[0100] In the example shown in Figure 13, the medical instrument 4 supported by arm 60a (number "4" in hand area G3a) and arm 60d (number "1" in hand area G3c) is located within the field of view of the endoscope 6. On the other hand, the medical instrument 4 supported by arm 60b (number "3" in hand area G3b) is located outside the field of view of the endoscope 6.

[0101] As shown in Figure 13, the image processing device 8b is configured to display a marker MK1 in the direction in which the medical instrument 4 outside the field of view is located within the outer edge vicinity area G11 adjacent to the level indicator area G5. In other words, the graphical user interface G displays the marker MK1 indicating the medical instrument 4 outside the field of view in the outer edge vicinity area G11, and in area G12 (see Figure 20) which corresponds to the direction in which the medical instrument 4 outside the field of view is located relative to the center CN2 of the level indicator area G5. In Figure 13, the marker MK1 (an arrow and the letter "3") is displayed.

[0102] Furthermore, as shown in Figure 20, the area G12 where the marker MK1 indicating the medical device 4 outside the field of view is displayed is one of the areas G12 obtained by dividing the area near the outer edge G11 into multiple areas radially from the central CN2 of the level area G5. The area near the outer edge G11 is divided into eight areas radially from the central CN2. In Figure 13, the marker MK1 (arrow and the letter "3") is displayed, for example, in the upper area G12c (see Figure 20). Note that Figure 20 shows all patterns in which the medical device 4 is located outside the field of view.

[0103] Furthermore, as shown in Figure 20, the image processing device 8b is configured to display a marker MK1 indicating the direction corresponding to the area where the medical device 4 is located, which is one of eight divisions of the out-of-field area according to the shape of the rectangular display unit, where the vertical length and horizontal length are not equal.

[0104] Furthermore, as shown in Figure 20, the display unit has a horizontal length greater than its vertical length. The area outside the field of view is divided into eight regions according to the shape of the display unit, such that the angles of the upper region G12c and the lower region G12g are greater than the angles of the right region G12a, the left region G12e, the upper right region G12b, the upper left region G12d, the lower right region G12h, and the lower left region G12f.

[0105] Furthermore, the image processing device 8b is configured to display the marker MK1 at a predetermined position within each of the eight divided regions of the outer edge vicinity area G11.

[0106] Furthermore, as shown in Figure 14, when the image processing device 8b receives a command via the control unit 31 to enable movement of the endoscope 6, it displays the level indicator LV of the endoscope 6 in the level indicator area G5. In other words, while the camera movement operation is being performed on the camera pedal 22c, the image processing device 8b displays the level indicator of the endoscope 6 in the level indicator area G5 of the graphical user interface G. The image processing device 8b then displays the graphical user interface G on the monitor 24 for displaying the marker MK1 in the area G11 near the outer edge outside the level indicator area G5. The level indicator LV represents the tilt of the endoscope 6's field of view relative to the patient P.

[0107] Furthermore, as shown in Figure 14, the level indicator area G5 is a rectangular region (a horizontally elongated rectangle) that includes the upper edge G5a, left edge G5b, right edge G5c, and lower edge G5d. By displaying the indicator MK1 in the area G11 near the outer edge of the level indicator area G5, which is of a predetermined size and includes the central part of the screen gr, the operator's visibility can be improved. The upper edge G5a is positioned between a location 1 / 10 (L11) above the central part CN1 of the screen gr of the monitor 24 by a length of 1 / 10 (L11) of the vertical length of the screen gr, and a location 1 / 10 (L11) below the upper edge eu of the screen gr by a length of 1 / 10 (L11) of the vertical length of the screen gr. Preferably, the upper edge portion G5a is positioned between a position 1 / 8 the length of the vertical length of the screen gr from the central portion CN1 of the screen gr of the monitor 24 and a position 1 / 8 the length of the vertical length of the screen gr from the upper edge eu of the screen gr. More preferably, the upper edge portion G5a is positioned between a position 1 / 6 the length of the vertical length of the screen gr from the central portion CN1 of the screen gr of the monitor 24 and a position 1 / 6 the length of the vertical length of the screen gr from the upper edge eu of the screen gr.

[0108] Furthermore, the upper part G5a is located below the status area G10, which displays information such as the remaining battery level of the medical manipulator 1.

[0109] Furthermore, the left-hand portion G5b is positioned between a position 1 / 10 (L12) to the left of the central portion CN1 of the screen gr of the monitor 24, and a position 1 / 10 (L12) to the right of the left edge el of the screen gr. Preferably, the left-hand portion G5b is positioned between a position 1 / 8 of the horizontal length of the screen gr to the left of the central portion CN1 of the screen gr of the monitor 24, and a position 1 / 8 of the horizontal length of the screen gr to the right of the left edge el of the screen gr. More preferably, the left-hand portion G5b is positioned between a position 1 / 6 of the horizontal length of the screen gr to the left of the central portion CN1 of the screen gr of the monitor 24, and a position 1 / 6 of the horizontal length of the screen gr to the right of the left edge el of the screen gr.

[0110] Furthermore, the right-hand portion G5c is positioned between a position to the right of the central portion CN1 of the screen gr of the monitor 24 by a length of 1 / 10 (L12) of the horizontal length of the screen gr, and a position to the left of the right edge er of the screen gr by a length of 1 / 10 (L12) of the horizontal length of the screen gr. Preferably, the right-hand portion G5c is positioned between a position to the right of the central portion CN1 of the screen gr of the monitor 24 by a length of 1 / 8 of the horizontal length of the screen gr, and a position to the left of the right edge er of the screen gr by a length of 1 / 8 of the horizontal length of the screen gr. More preferably, the right-hand portion G5c is positioned between a position to the right of the central portion CN1 of the screen gr of the monitor 24 by a length of 1 / 6 of the horizontal length of the screen gr, and a position to the left of the right edge er of the screen gr by a length of 1 / 6 of the horizontal length of the screen gr.

[0111] Furthermore, the lower edge portion G5d is positioned between a position 1 / 10 (L11) below the central portion CN1 of the screen gr of the monitor 24 by a length of 1 / 10 (L11) of the vertical length of the screen gr and a position 1 / 10 (L11) above the lower edge ed of the screen gr by a length of 1 / 10 of the vertical length of the screen gr. Preferably, the lower edge portion G5d is positioned between a position 1 / 8 of the vertical length of the screen gr of the monitor 24 by a length of 1 / 8 of the vertical length of the screen gr and a position 1 / 8 of the vertical length of the screen gr above the lower edge ed of the screen gr. More preferably, the lower edge portion G5d is positioned between a position 1 / 6 of the vertical length of the screen gr of the monitor 24 by a length of 1 / 6 of the vertical length of the screen gr and a position 1 / 6 of the vertical length of the screen gr above the lower edge ed of the screen gr.

[0112] The lower edge G5d of the level indicator area G5 is positioned above the hand area G3a, hand area G3b, hand area G3c, and camera area G2. Furthermore, the lower edge G5d of the level indicator area G5 is positioned above the medical device status information area G4.

[0113] Note that the central part CN1 of the screen gr of monitor 24 and the central part CN2 of the level indicator area G5 are in approximately the same position.

[0114] Furthermore, the marker MK1 indicating the medical device 4 located outside the field of view is positioned between the level area G5 and at least one (all of these in this embodiment) of the medical device status information area G4, the left pop-up area G6, the right pop-up area G7, and the status area G10. The medical device status information area G4 indicates the status of the medical device 4. The left pop-up area G6 is displayed when the foot pedal 22 is operated (when the foot is on the foot pedal 22 in a hover state). The right pop-up area G7 is displayed when the coagulation pedal 22eR and the incision pedal 22dR are operated (when the operator's foot is on them). The status area G10 indicates the status of the surgical system 100.

[0115] Furthermore, as shown in Figure 13, the label MK1 includes a number that identifies the arm 60 supporting the medical instrument 4 located outside the field of view. In the example shown in Figure 13, the label MK1 displays the number "3", indicating that the medical instrument 4 supported by arm 60b (number "3" in hand area G3) is located outside the field of view of the endoscope 6.

[0116] (Out-of-field detection processing) Next, with reference to Figure 30, the out-of-field detection process in the surgical system 100 will be described. The out-of-field detection process is performed by the control unit 31.

[0117] When a predetermined input is received via the camera pedal 22c or the enable switch 81, in step S1, the control unit 31 acquires the tip position of the medical instrument 4 (forceps). In step S2, the control unit 31 acquires the second imaging range A2.

[0118] In step S3, the control unit 31 determines whether the medical instrument 4 (forceps) is in front of the endoscope 6. If the medical instrument 4 is in front, the process proceeds to step S7; if the endoscope 6 is in front, the process proceeds to step S4. In step S4, the control unit 31 determines whether the medical instrument 4 (forceps) is within the range of the second imaging range A2. If the medical instrument 4 (forceps) is within the range of the second imaging range A2, the process proceeds to step S7; if the medical instrument 4 (forceps) is outside the range of the second imaging range A2, the process proceeds to step S5.

[0119] In step S5, the control unit 31 acquires the direction of the area outside the field of view. In step S6, the image processing device 8b displays the direction of the area outside the field of view using the indicator MK1. Then, the process returns to step S1. In step S7, the image processing device 8b displays the around view (AR). Then, the process returns to step S1.

[0120] [Effects of this embodiment] In this embodiment, the following effects can be obtained.

[0121] In this embodiment, as described above, the control unit 31 acquires the tip position of the medical instrument 4, acquires a second imaging range A2 that reflects the positional displacement related to the tip position of the medical instrument 4 and the diameter of the shaft of the medical instrument 4, relative to the first imaging range A1 corresponding to the field of view of the endoscope 6, and determines whether the tip position of the medical instrument 4 is within the second imaging range A2. This makes it possible to determine whether the tip position of the medical instrument 4 is outside the field of view of the endoscope 6 based on the second imaging range A2 that reflects the positional displacement related to the tip position of the medical instrument 4. As a result, when the tip position of the medical instrument 4 relative to the field of view of the endoscope 6 is uncertain due to a displacement of the tip position of the medical instrument 4 relative to the field of view of the endoscope 6, it is possible to suppress the indication of an uncertain direction for the tip position of the medical instrument 4. Furthermore, since it is possible to determine whether the tip position of the medical instrument 4 is outside the field of view of the endoscope 6 based on the second imaging range A2 that reflects the diameter of the shaft of the medical instrument 4, it is possible to determine whether the tip position of the medical instrument 4, which has a certain size, is within the field of view of the endoscope 6, not just a specific point at the tip of the medical instrument 4. As a result, even if there is a misalignment of the medical device 4, the operator can easily confirm its position.

[0122] Furthermore, in this embodiment, as described above, if the control unit 31 determines that the tip position of the medical instrument 4 is outside the second imaging range A2, the image processing device 8b displays a marker MK1 indicating that the medical instrument 4 is outside the field of view of the endoscope 6. As a result, the marker MK1 is displayed only when the tip position of the medical instrument 4 is definitely outside the field of view of the endoscope 6, so the operator can easily find the tip position of the medical instrument 4 by pointing the endoscope 6 in the direction indicated by the display of the marker MK1.

[0123] Furthermore, in this embodiment, as described above, if the control unit 31 determines that the tip position of the medical instrument 4 is within the second imaging range A2, the image processing device 8b displays an around-view AR (Around Display) indicating that the medical instrument 4 is within the field of view of the endoscope 6 or is in the peripheral area outside the field of view, where the tip position of the medical instrument 4 would be within the field of view if the endoscope 6 were pulled forward. As a result, the around-view AR is displayed when the tip position of the medical instrument 4 is in the peripheral area of ​​the field of view of the endoscope 6, so the operator can easily recognize that the tip position of the medical instrument 4 is located in the peripheral area of ​​the field of view. Also, if the tip of the medical instrument 4 is not visible in the field of view of the endoscope 6, the operator can easily find the tip position of the medical instrument 4 by pulling the endoscope 6 forward.

[0124] Furthermore, in this embodiment, as described above, if the control unit 31 determines that the tip of the medical instrument 4 is within the second imaging range A2, the image processing device 8b displays a prompt to pull the endoscope 6 forward. This allows the operator to be prompted to pull the endoscope 6 forward if the tip of the medical instrument 4 is not visible in the field of view of the endoscope 6.

[0125] Furthermore, in this embodiment, as described above, the around-view display (AR) includes a number that identifies the arm 60 supporting the medical instrument 4 that is determined to be within the second imaging range A2. This allows the operator to easily identify and recognize the medical instrument 4 located at the periphery of the endoscope 6's field of view.

[0126] Furthermore, in this embodiment, as described above, if the control unit 31 determines that the tip position of the medical instrument 4 is outside the second imaging range A2, the image processing device 8b displays a marker MK1 indicating that the medical instrument 4 is outside the field of view of the endoscope 6 in response to a predetermined input to the camera pedal 22c or the enable switch 81. Also, if the control unit 31 determines that the tip position of the medical instrument 4 is within the second imaging range A2, the image processing device 8b displays an around indicator AR indicating that the medical instrument 4 is within the field of view of the endoscope 6 or in the peripheral area outside the field of view that would be within the field of view if the endoscope 6 were pulled forward. As a result, the operator can easily confirm the tip position of the medical instrument 4 by making a predetermined input to the camera pedal 22c or the enable switch 81, thereby enabling the display of either the marker MK1 or the around indicator AR.

[0127] Furthermore, in this embodiment, as described above, the positional misalignment related to the tip position of the medical instrument 4 includes a first positional misalignment caused by a positional misalignment of the field of view of the endoscope 6, and a second positional misalignment which is a positional misalignment of the tip of the medical instrument 4. This allows both the positional misalignment of the field of view of the endoscope 6 and the positional misalignment of the tip of the medical instrument 4 to be treated together as a positional misalignment related to the tip position of the medical instrument 4.

[0128] Furthermore, in this embodiment, as described above, the control unit 31 calculates the first positional displacement based on the positional displacement of the endoscope 6 at the pivot position PP1 and the distance of the tip of the endoscope 6 from the pivot position PP1. This makes it easy to calculate the first positional displacement based on the insertion distance of the endoscope 6. The control unit 31 also calculates the second positional displacement based on the positional displacement of the medical instrument 4 at the pivot position PP2 and the distance of the tip of the medical instrument 4 from the pivot position PP2. This makes it easy to calculate the second positional displacement based on the insertion distance of the medical instrument 4.

[0129] Furthermore, in this embodiment, as described above, the tip position of the medical device 4 is the tool center point (TCP), which is the control center when the medical device 4 is operated. This allows the operator to easily grasp the tool center point, which is the control center when the medical device 4 is operated by the operator.

[0130] Furthermore, in this embodiment, as described above, if the control unit 31 determines that the distance between the tip position of the medical instrument 4 and the pivot position PP1 is smaller than the distance between the tip position of the endoscope 6 and the pivot position PP1, the image processing device 8b displays a prompt to pull the endoscope 6 forward. As a result, if the distance between the tip position of the medical instrument 4 and the pivot position PP1 is smaller than the distance between the tip position of the endoscope 6 and the pivot position PP1, and even if the orientation of the endoscope 6 is changed, the prompt to pull the endoscope 6 forward is displayed, allowing the operator to easily locate the tip position of the medical instrument 4 which is outside the field of view. Consequently, even if there is a misalignment of the medical instrument 4, the operator can easily confirm its position.

[0131] Furthermore, in this embodiment, as described above, the image processing device 8b displays an around view (AR) to prompt the user to pull the endoscope 6 forward, indicating that the tip of the medical instrument 4 is in the peripheral area outside the field of view and will be within the field of view when the endoscope 6 is pulled forward. This allows the operator to easily recognize that, if the tip of the medical instrument 4 is not visible in the field of view of the endoscope 6, it is in the peripheral area outside the field of view and will be within the field of view when the endoscope 6 is pulled forward.

[0132] Furthermore, in this embodiment, as described above, when the tip position of the medical instrument 4 is in front of the tip position of the endoscope 6, if the control unit 31 determines that the distance between the tip position of the medical instrument 4 and the pivot position PP1 is greater than the distance between the tip position of the endoscope 6 and the pivot position PP1, the image processing device 8b displays the direction of the tip position of the medical instrument 4 relative to the imaging range of the endoscope 6. This allows the operator to easily recognize the position of the tip position of the medical instrument 4 when changing the orientation of the endoscope 6 brings the tip position of the medical instrument 4 into the imaging range of the endoscope 6.

[0133] Furthermore, in this embodiment, as described above, the control unit 31 corrects for positional misalignment caused by errors in the spacing between arm 60c and arms 60a, 60b, or 60d, and acquires the respective tip positions of the endoscope 6 and medical instrument 4. This makes it possible to accurately acquire the respective tip positions of the endoscope 6 and medical instrument 4 by correcting for positional misalignment caused by assembly errors when assembling arms 60a, 60b, 60c, and 60d.

[0134] [Differentiation] It should be noted that the embodiments disclosed herein are illustrative and not restrictive in all respects. The scope of the present invention is defined by the claims rather than by the description of the embodiments above, and further includes all modifications (exceptions) within the meaning and scope equivalent to the claims.

[0135] For example, the above embodiment shows an example in which the image processing device 8b acquires an image from the endoscope 6 and generates a graphical user interface G, but the present invention is not limited thereto. For example, the control unit 31 of the medical manipulator 1 may generate the graphical user interface G, or the control unit (not shown) of the remote control device 2 may generate the graphical user interface G. Furthermore, an image processing device that acquires an image from the endoscope 6 and performs image processing, and an image processing device that generates the graphical user interface G and overlays it on the image from the endoscope 6 may be provided separately.

[0136] Furthermore, in the above embodiment, the control unit 31 of the medical manipulator 1 acquires the tip position of the medical instrument 4, acquires a second imaging range A2 that reflects the positional displacement related to the tip position of the medical instrument 4 and the diameter of the shaft of the medical instrument 4 relative to the first imaging range A1 corresponding to the field of view of the endoscope 6, and determines whether the tip position of the medical instrument 4 is within the second imaging range A2. However, the present invention is not limited to this. In the present invention, for example, the tip position of the medical instrument 4 may be acquired by an image processing device 8b or a control unit (not shown) of a remote control device 2, acquires a second imaging range A2 that reflects the positional displacement related to the tip position of the medical instrument 4 and the diameter of the shaft of the medical instrument 4 relative to the first imaging range A1, and determines whether the tip position of the medical instrument 4 is within the second imaging range A2. Alternatively, the above processing may be performed by multiple control units (control unit 31, image processing device 8b, and remote control device 2, etc.).

[0137] Furthermore, although the above embodiment shows an example in which the image processing device 8b determines whether or not input has been received from the camera pedal 22c or the enable switch 81, the present invention is not limited thereto. For example, the control unit 31 may determine whether or not input has been received from the camera pedal 22c or the enable switch 81 and transmit the determination result to the image processing device 8b.

[0138] Furthermore, in the above embodiment, when it is determined that the tip position of the medical instrument 4 is within the second imaging range A2, an example of around display AR is shown indicating that the medical instrument 4 is within the field of view of the endoscope 6 or in the peripheral area outside the field of view where the tip position of the medical instrument 4 will be within the field of view if the endoscope 6 is pulled forward is shown, but the present invention is not limited to this. In the present invention, the indication that the medical instrument 4 is within the field of view of the endoscope 6 or in the peripheral area outside the field of view where the tip position of the medical instrument 4 will be within the field of view if the endoscope 6 is pulled forward may be indicated by a display that directly instructs to pull the endoscope 6, such as "PULL" or "ZOOM OUT". Alternatively, the display may be indicated by a graphic showing that the endoscope 6 is being pulled forward.

[0139] Furthermore, in the above embodiment, an example was shown in which, when it is determined that the tip position of the medical instrument 4 is outside the second imaging range A2, a sign MK1 including an arrow indicating that the medical instrument 4 is outside the field of view of the endoscope 6 is displayed. However, the present invention is not limited to this. In the present invention, the sign indicating that the medical instrument 4 is outside the field of view of the endoscope 6 may be a sign that does not include an arrow.

[0140] Furthermore, although the above embodiment shows an example of a configuration in which an enable switch 81, a joystick 82, and a switch unit 83 are provided on multiple arms 60, the present invention is not limited thereto. For example, an enable switch 81, a joystick 82, and a switch unit 83 may be provided on at least one of the multiple arms 60.

[0141] Furthermore, while the above embodiment shows an example where the around display AR and indicator MK1 are displayed on the graphical user interface G in response to the operation of the enable switch 81 of the arm 60, the present invention is not limited thereto. For example, the around display AR and indicator MK1 may be displayed on the graphical user interface G in response to the operation of other switches of the arm 60 (joystick 82 and switch unit 83).

[0142] Furthermore, while the above embodiment shows an example where the around view display AR and the indicator MK1 are displayed on the graphical user interface G when a predetermined input is made by the camera pedal 22c or the enable switch 81, the present invention is not limited thereto. For example, the around view display AR and the indicator MK1 may also be displayed on the graphical user interface G when a predetermined input is made by another input device. For example, the around view display AR and the indicator MK1 may be displayed on the graphical user interface G when a predetermined input is made by a switch (first switch) provided on the operating handle 21.

[0143] Furthermore, in the above embodiment, an example was shown in which the marker MK1 is positioned towards the center of the screen gr when one or two medical instruments 4 are located outside the field of view of the endoscope 6, but the present invention is not limited to this. For example, the position in which the marker MK1 corresponding to each of the multiple medical instruments 4 is positioned may be fixed.

[0144] Furthermore, in the above embodiment, an example was shown where the tip position of the medical instrument 4 used to determine whether or not the image is within the second imaging range A2 is the tool center point, but the present invention is not limited to this. In the present invention, the tip position of the medical instrument 4 may be the tip positions of the end effector members 104a and 104b.

[0145] Furthermore, although the above embodiment shows an example in which the arm portion 61 and the positioner 40 are composed of a 7-axis articulated robot, the present invention is not limited to this. For example, the arm portion 61 and the positioner 40 may be composed of an articulated robot with an axis configuration other than a 7-axis articulated robot (for example, a 6-axis or 8-axis robot).

[0146] Furthermore, although the above embodiment shows an example in which the medical manipulator 1 includes a medical trolley 3, a positioner 40, and an arm base 50, the present invention is not limited to this. For example, the medical trolley 3, the positioner 40, and the arm base 50 are not necessarily required, and the medical manipulator 1 may consist only of an arm 60.

[0147] The functions of the elements disclosed herein can be performed using circuits or processing circuits, including general-purpose processors, dedicated processors, integrated circuits, ASICs (Application Specific Integrated Circuits), conventional circuits, and / or combinations thereof, configured or programmed to perform the disclosed functions. A processor is considered a processing circuit or circuit because it includes transistors and other circuits. In this disclosure, a circuit, unit, or means is hardware that performs the enumerated functions, or hardware programmed to perform the enumerated functions. The hardware may be hardware disclosed herein, or other known hardware that is programmed or configured to perform the enumerated functions. If the hardware is a processor, which is considered a type of circuit, then the circuit, means, or unit is a combination of hardware and software, and the software is used to configure the hardware and / or the processor.

[0148] [Aspect] Those skilled in the art will understand that the exemplary embodiments described above are specific examples of the following embodiments.

[0149] (Item 1) A first manipulator for supporting the endoscope, A second manipulator for supporting surgical instruments, A control unit including one or more processors, The control device acquires the tip position of the surgical instrument, acquires a second imaging range that reflects the positional displacement related to the tip position of the surgical instrument and the diameter of the shaft of the surgical instrument with respect to a first imaging range corresponding to the field of view of the endoscope, and determines whether the tip position of the surgical instrument is within the second imaging range.

[0150] (Item 2) The surgical system according to item 1, wherein the control device determines that the tip position of the surgical instrument is outside the second imaging range, and then displays a first indication that the surgical instrument is outside the field of view of the endoscope.

[0151] (Item 3) The surgical system according to item 1 or item 2, wherein the control device determines that the tip position of the surgical instrument is within the second imaging range, and then displays a second indication that the surgical instrument is within the field of view of the endoscope or in the peripheral area outside the field of view, where the tip position of the surgical instrument would be within the field of view if the endoscope were pulled forward.

[0152] (Item 4) The surgical system according to item 3, wherein the control device determines that the tip of the surgical instrument is within the second imaging range, and as the second display, displays a message prompting the user to pull the endoscope forward.

[0153] (Item 5) The surgical system according to item 4, wherein the second indicator includes a number that identifies the second manipulator supporting the surgical instrument determined to be within the second imaging range.

[0154] (Item 6) The surgical system according to any one of items 1 to 5, wherein the control device, in response to a predetermined input to an input device, performs a first display indicating that the surgical instrument is outside the field of view of the endoscope when it determines that the tip position of the surgical instrument is outside the second imaging range, and performs a second display indicating that the surgical instrument is within the field of view of the endoscope or in the peripheral area outside the field of view that would be within the field of view if the endoscope were pulled forward.

[0155] (Item 7) The surgical system according to item 6, wherein the input device includes at least one of a first switch provided on an operating handle for operating the surgical instrument, a second switch provided on the second manipulator, and a third switch for performing movement operations of the endoscope using the operating handle.

[0156] (Item 8) The surgical system according to any one of items 1 to 7, wherein the positional displacement related to the tip position of the surgical instrument includes a first positional displacement resulting from a positional displacement of the field of view of the endoscope and a second positional displacement which is a positional displacement of the tip of the surgical instrument.

[0157] (Item 9) The surgical system according to item 8, wherein the control device calculates the first positional displacement based on the positional displacement of the endoscope at the pivot position and the distance of the tip of the endoscope from the pivot position, and calculates the second positional displacement based on the positional displacement of the surgical instrument at the pivot position and the distance of the tip of the surgical instrument from the pivot position.

[0158] (Item 10) The surgical system described in any one of items 1 to 9, wherein the tip position of the surgical instrument is the tool center point position.

[0159] (Item 11) A first manipulator for supporting the endoscope, A second manipulator for supporting surgical instruments, A control unit including one or more processors, The control device is configured to control the first manipulator so that the endoscope moves with the first pivot position as the pivot point, and to control the second manipulator so that the surgical instrument moves with the second pivot position as the pivot point. A surgical system comprising: a control device that acquires the tip positions of the endoscope and the surgical instrument, and when the distance between the tip position of the surgical instrument and the first pivot position is smaller than the distance between the tip position of the endoscope and the first pivot position, displays a prompt to pull the endoscope forward.

[0160] (Item 12) The surgical system according to item 11, wherein the control device displays a message prompting the user to pull the endoscope forward, indicating that when the endoscope is pulled forward, the tip of the surgical instrument is in the peripheral area outside the field of view, which would then be within the field of view.

[0161] (Item 13) The surgical system according to item 11 or 12, wherein the control device displays the direction of the tip of the surgical instrument relative to the imaging range of the endoscope when the tip of the surgical instrument is in front of the tip of the endoscope, and the distance between the tip of the surgical instrument and the first pivot position is greater than the distance between the tip of the endoscope and the first pivot position.

[0162] (Item 14) The surgical system according to any one of items 11 to 13, wherein the control device corrects for positional misalignment caused by an error in the distance between the first manipulator and the second manipulator to obtain the respective tip positions of the endoscope and the surgical instrument. [Explanation of Symbols]

[0163] 4. Medical Instruments (Surgical Instruments) 6 Endoscope 8b Image processing device (control device) 21, 21L, 21R Operating handle 22c Camera Pedal (Input Device, 3rd Switch) 31 Control Unit (Control Device) 60c Arm (First Manipulator) 60a, 60b, 60d Arm (Second Manipulator) 81 Enable switch (input device, second switch) 100 Surgical Operating Systems (Surgical Systems) A1 First imaging range A2 Second imaging range AR Around View (Second View) MK1 sign (1st display) PP1 Pivot position (First pivot position) PP2 Pivot Position (Second Pivot Position)

Claims

1. a first manipulator for supporting an endoscope; a second manipulator for supporting a surgical instrument; a control unit including one or more processors; The control device acquires the tip position of the surgical instrument, acquires a second imaging range that reflects a positional deviation related to the tip position of the surgical instrument and a diameter of the shaft of the surgical instrument relative to a first imaging range that corresponds to the angle of view of the endoscope, and determines whether the tip position of the surgical instrument is within the second imaging range.

2. The surgical system according to claim 1 , wherein the control device, when determining that the tip position of the surgical instrument is outside the second imaging range, displays a first indication indicating that the surgical instrument is outside the field of view of the endoscope.

3. 2. The surgical system of claim 1, wherein when the control device determines that the tip position of the surgical instrument is within the second imaging range, it displays a second indication indicating that the surgical instrument is within the field of view of the endoscope or is outside the field of view in a manner that would bring the tip position of the surgical instrument within the field of view if the endoscope were pulled toward the user.

4. The surgical system according to claim 3 , wherein when the control device determines that the tip position of the surgical instrument is within the second imaging range, the second display prompts the user to pull the endoscope toward themselves.

5. The surgical system of claim 4 , wherein the second display includes a number identifying the second manipulator that supports the surgical instrument determined to be within the second imaging range.

6. 2. The surgical system according to claim 1, wherein, in response to a predetermined input to an input device, the control device, when determining that the tip position of the surgical instrument is outside the second imaging range, displays a first display indicating that the surgical instrument is outside the field of view of the endoscope, and, when determining that the tip position of the surgical instrument is within the second imaging range, displays a second display indicating that the surgical instrument is within the field of view of the endoscope or is in a periphery outside the field of view that would be within the field of view if the endoscope were pulled toward the user.

7. 7. The surgical system according to claim 6, wherein the input device includes at least one of a first switch provided on an operating handle for operating the surgical instrument, a second switch provided on the second manipulator, and a third switch for operating the movement of the endoscope using the operating handle.

8. The surgical system of claim 1, wherein the positional deviation associated with the tip position of the surgical instrument includes a first positional deviation caused by a positional deviation of the field of view of the endoscope and a second positional deviation that is a positional deviation of the tip of the surgical instrument.

9. 9. The surgical system of claim 8, wherein the control device calculates the first positional deviation based on a positional deviation of the endoscope at a pivot position and a distance from the pivot position of the tip of the endoscope, and calculates the second positional deviation based on a positional deviation of the surgical instrument at a pivot position and a distance from the pivot position of the tip of the surgical instrument.

10. The surgical system of claim 1 , wherein the tip position of the surgical instrument is a tool center point position.

11. a first manipulator for supporting an endoscope; a second manipulator for supporting a surgical instrument; a control unit including one or more processors; the control device is configured to control the first manipulator to move the endoscope to a first pivot position as a fulcrum, and to control the second manipulator to move the surgical instrument to a second pivot position as a fulcrum; The control device acquires the tip positions of the endoscope and the surgical instrument, and if the distance between the tip position of the surgical instrument and the first pivot position is smaller than the distance between the tip position of the endoscope and the first pivot position, displays a message urging the user to pull the endoscope toward themselves.

12. The surgical system according to claim 11, wherein the control device displays a display to prompt the user to pull the endoscope toward the user, indicating that the tip position of the surgical instrument is outside the field of view and will be within the field of view if the endoscope is pulled toward the user.

13. The surgical system according to claim 11, wherein the control device displays a direction of the tip position of the surgical instrument relative to the imaging range of the endoscope when the tip position of the surgical instrument is located closer to the tip position of the endoscope and the distance between the tip position of the surgical instrument and the first pivot position is greater than the distance between the tip position of the endoscope and the first pivot position.

14. The surgical system according to claim 11 , wherein the control device corrects positional deviations caused by errors in the distance between the first manipulator and the second manipulator to obtain the tip positions of the endoscope and the surgical instrument.

Images