Surgical robot system, computer for surgical robot, and method for providing surgical information for surgical robot

Abstract

The present invention provides a surgical robot system that allows for easy verification of how an operator manipulated the surgical robot during surgery, a computer for the surgical robot, and a method for providing surgical information for the surgical robot. [Solution] The computer displays a graphical user interface related to surgical procedures on the display unit. The graphical user interface displays a video showing the operation of the tip of a surgical instrument during surgery, multiple timeline areas corresponding to multiple arms, and numerical values ​​generated based on sensor signals that correspond to the force applied to the patient's tissue by the tip. The computer displays the numerical values ​​in association with the operation of the tip shown in the video.

Description

Technical Field

[0001] The present invention relates to a surgical robot system, a computer for a surgical robot, and a method for providing surgical information for a surgical robot.

Background Art

[0002] Conventionally, surgical operations using surgical robots have been performed. The surgical robot includes a console (operating device) and a surgical robot arm (arm device). A surgical instrument is attached to the surgical robot arm, and the operator operates the console to move the surgical robot arm and the surgical instrument. It is desirable that any operator can improve their proficiency in operating the console so that patients can undergo surgery with confidence.

[0003] The following Patent Document 1 describes a system that captures and stores the state of a surgical operation using a surgical robot performed in an operating room with a depth camera, and enables the reproduction of a patterned image of the captured image after the surgery. Further, the system of Patent Document 1 stores histories such as the state of the surgical robot arm and the state of the handheld input device (handheld UID) of the console as system data. The stored system data is used to reconstruct the position, orientation, and movement of the robot arm and surgical instrument and to pattern the captured image.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, the system described in Patent Document 1 only allows users to view a schematic image of the surgical procedure performed by the surgical robot, making it difficult to see how the operator was operating the console during the surgery. [Means for solving the problem]

[0006] A first aspect of the present invention is a surgical robot system comprising a surgical robot configured to perform surgical procedures on a patient using surgical instruments, and a computer, wherein the surgical robot includes an arm device having a plurality of arms to which a plurality of types of surgical instruments can be detachably attached, an endoscope configured to capture images showing the surgical site and the tip of a surgical instrument, and a hand controller configured to remotely control the arms, the hand controller having a pair of movable members used to open and close the tip, and a sensor configured to generate signals in response to the operation of the tip. The computer acquires a plurality of images captured by the endoscope during the surgical procedure and a plurality of signals generated by the sensor during the surgical procedure, and displays a graphical user interface related to the surgical procedure on a display unit. The graphical user interface displays a video generated from the plurality of images showing the operation of the tip during the surgical procedure, a plurality of timeline regions corresponding to each of the plurality of arms, and a numerical value generated based on at least one of the plurality of signals corresponding to the force applied to the patient's tissue by the tip. Each of the multiple timeline regions extends linearly in accordance with the surgical timeline and visually distinguishes between a first period in which a first type of surgical instrument is attached to the arm and a second period in which a second type of surgical instrument is attached to the arm. The computer displays the numerical values ​​in a graphical user interface in association with the operation of the tip shown in the video.

[0007] A second aspect of the present invention is a computer for a surgical robot comprising a plurality of arms and a hand controller having a pair of movable members for opening and closing the tips of surgical instruments attached to the arms, the computer comprising a memory for storing a program and a processor for executing the program. The surgical robot comprises an endoscope configured to capture images showing the surgical site and the tips, and a sensor configured to generate signals in response to the operation of the tips. By executing the program, the processor functions as means for acquiring a plurality of images captured by the endoscope during a surgical operation performed on a patient using the surgical robot, means for acquiring a plurality of signals generated by the sensor during the surgical operation, and means for displaying a graphical user interface relating to the surgical operation on a display unit. The graphical user interface displays a video generated from the plurality of images showing the operation of the tips during the surgical operation, a plurality of timeline regions corresponding to each of the plurality of arms, and a numerical value generated based on at least one of the plurality of signals corresponding to the force applied to the patient's tissue by the tips. Each of the multiple timeline regions extends linearly in accordance with the surgical timeline and visually distinguishes between a first period in which a first type of surgical instrument is attached to the arm and a second period in which a second type of surgical instrument is attached to the arm. The processor displays the numerical values ​​in the graphical user interface in association with the operation of the tip shown in the video.

[0008] A third aspect of the present invention is a method for providing surgical information for a surgical robot comprising a plurality of arms and a hand controller having a pair of movable members for opening and closing the tips of surgical instruments attached to the arms, wherein the surgical robot comprises an endoscope configured to capture images showing a surgical site and the tips, and a sensor configured to generate signals in response to the operation of the tips. The method includes the steps of acquiring a plurality of images captured by the endoscope during a surgical operation performed on a patient using the surgical robot, acquiring a plurality of signals generated by the sensor during the surgical operation, and displaying a graphical user interface relating to the surgical operation on a display unit. The graphical user interface displays a video generated from the plurality of images showing the operation of the tips during the surgical operation, a plurality of timeline regions corresponding to each of the plurality of arms, and a numerical value generated based on at least one of the plurality of signals corresponding to the force applied to the patient's tissue by the tips. Each of the plurality of timeline regions extends linearly in accordance with the timeline of the surgical operation and visually distinguishes between a first period in which a first type of surgical instrument is attached to the arm and a second period in which a second type of surgical instrument is attached to the arm. The graphical user interface displays the numerical values ​​in association with the operation of the tip shown in the video. [Effects of the Invention]

[0009] According to the present invention, it is possible to easily verify how the operator was manipulating the surgical robot during surgery. [Brief explanation of the drawing]

[0010] [Figure 1] Figure 1 is a block diagram showing the configuration of a surgical robot according to an embodiment. [Figure 2] Figure 2 is a perspective view showing the external appearance of the arm device according to this embodiment. [Figure 3] Figure 3 is a perspective view showing the external appearance of the operating device according to this embodiment. [Figure 4] Figure 4 is a perspective view showing the external appearance of a hand controller according to this embodiment. [Figure 5] Figure 5 is a perspective view showing the external appearance of a foot unit according to an embodiment. [Figure 6] Figure 6 is a perspective view showing an operator using the control device according to the embodiment. [Figure 7] Figure 7 is a block diagram showing the configuration of a control device according to an embodiment. [Figure 8] Figure 8 is a block diagram showing the configuration of an information processing system according to an embodiment. [Figure 9] Figure 9 is a diagram illustrating the connection between a surgical robot and an information processing system, and an overview of the information transmitted and received between each device, according to an embodiment. [Figure 10] Figure 10 is a block diagram showing the usage configuration of the information processing device according to the embodiment. [Figure 11] Figure 11 is a block diagram showing the configuration of an arm device according to an embodiment. [Figure 12] Figure 12 is a block diagram showing the configuration of the operating device according to the embodiment. [Figure 13] Figure 13 is a diagram illustrating the current values ​​that constitute the status log transmitted from the operating device to the control device according to the embodiment. [Figure 14] Figure 14 shows an example of a status log according to this embodiment. [Figure 15] Figure 15 is a diagram illustrating the current values ​​that constitute the operation log transmitted from the arm device to the control device according to the embodiment. [Figure 16] Figure 16 shows an example of an operation log according to this embodiment. [Figure 17] Figure 17 is a schematic diagram showing the configuration of a surgical image according to an embodiment. [Figure 18] Figure 18 is a schematic diagram showing the configuration of a surgical image according to an embodiment. [Figure 19] Figure 19 is a schematic enlarged view showing the configuration of an endoscopic image according to an embodiment. [Figure 20] FIG. 20 is a diagram schematically showing the configuration of an image or the like indicating the usage record of a surgical instrument according to an embodiment. [Figure 21] FIG. 21 is an enlarged view schematically showing the configuration of an information display area according to an embodiment. [Figure 22] FIG. 22 is a diagram schematically showing the configuration and transition of an image related to the state of a foot pedal according to an embodiment. [Figure 23] FIG. 23 is a diagram schematically showing the configuration and transition of an image related to the state of a foot pedal according to an embodiment. [Figure 24] FIG. 24 is an enlarged view schematically showing the configuration of an information display area according to an embodiment. [Figure 25] FIG. 25 is a diagram schematically showing the configuration of an edited image displayed when a bookmark insertion button is operated in a surgical image according to an embodiment. [Figure 26] FIG. 26 is a diagram schematically showing the configuration of a surgical image when a hand-drawn image is inserted first according to an embodiment. [Figure 27] FIG. 27 is a flowchart showing a process of receiving information necessary for generating a surgical image according to an embodiment. [Figure 28] FIG. 28 is a flowchart showing a process of generating a surgical image according to an embodiment. [Figure 29] FIG. 29 is a flowchart showing details of an additional process of additional information according to an embodiment. [Figure 30] FIG. 30 is a flowchart showing a process related to storing a screenshot by adding a bookmark according to an embodiment. [Figure 31] FIG. 31 is a flowchart showing a process related to displaying a screenshot and playing back before and after the bookmark position according to an embodiment. [Figure 32] FIG. 32 is a diagram schematically showing an image of a photographed operating device, an image of a photographed patient's abdomen, an image of a photographed instrument storage, and an image of a photographed corridor connecting to an operating room according to a modified example of an embodiment. [Figure 33]Figure 33 is a schematic diagram showing the configuration of a surgical image according to a modified embodiment. [Modes for carrying out the invention]

[0011] Figure 1 is a block diagram showing the configuration of the surgical robot 4.

[0012] The surgical robot 4 is a device used in endoscopic surgery. The surgical robot 4 comprises an arm device 1, an operating device 2, and a control device 3 that controls the arm device 1 and the operating device 2. The control device 3 is built into the arm device 1. The arm device 1, which houses the control device 3, and the operating device 2 may be installed in the same facility, or they may be installed in different facilities via a network.

[0013] Figure 2 is a perspective view showing the external appearance of the arm device 1.

[0014] Arm device 1 is a patient-side device equipped with multiple arms to which an endoscope and forceps can be attached. Arm device 1 operates according to drive instructions transmitted from control device 3 when operating device 2 is operated by an operator, who is a physician.

[0015] The arm device 1 comprises a base 101, an operating unit 102, a base arm 103, a support unit 104, and arms 111 to 114. Surgical instruments 121 to 124 are detachably attached to each of the arms 111 to 114.

[0016] The control unit 102 is installed on the base 101 and includes a display unit and an input unit. The operator sets the movement speed of the arms 111 to 114 of the arm device 1 by operating the input unit of the control unit 102 while looking at the display unit of the control unit 102. The base arm 103 is an arm with multiple joints, the rear end of which is installed on the base 101. The support unit 104 has its upper surface rotatably connected to the tip of the base arm 103, and moves in conjunction with the movement of the tip of the base arm 103, as well as rotating around the axis of the tip of the base arm 103.

[0017] Arms 111 to 114 have their upper ends mounted on the lower surface of the support part 104. Each of the arms 111 to 114 is equipped with 12 axes and joints between axes. The lower ends of arms 111 to 114 are provided with supports configured to allow the attachment and detachment of surgical instruments, and surgical instruments 121 to 124 are attached to these supports.

[0018] Surgical instruments 121-124 are equipped with elongated shafts 121a-124a, and during endoscopic surgery, the shafts 121a-124a of surgical instruments 121-124 are inserted into the patient's body via a guide tube (trocar) inserted into the patient's abdomen. In this embodiment, surgical instruments 121, 123, and 124 are forceps, and surgical instruments (forceps) 121 and 124 are so-called grasping forceps. Surgical instrument (forceps) 123 is a so-called electrosurgical unit, and is equipped with a hand at its tip capable of cutting and coagulation. Surgical instrument 122 is an endoscope. Surgical instrument (endoscope) 122 is, for example, a 3D videoscope. Hereinafter, surgical instrument 122 will also be referred to as "endoscope 122".

[0019] Note that the forceps are not limited to grasping forceps; other types of forceps such as hemostatic forceps or dissection / ligation forceps may also be used, as may electric instruments such as electrosurgical units. Surgical instruments 121-124 may be replaced with different types of surgical instruments as needed during the endoscopic surgery.

[0020] Figure 3 is a perspective view showing the external appearance of the operating device 2.

[0021] Operating device 2 is an operator-side device used by a physician operator to drive arm device 1 during endoscopic surgery.

[0022] The operating device 2 comprises a base 201, a support part 202, a frame member 203, an operation panel 204, a viewer unit 210, two hand controllers 220, and a foot unit 230.

[0023] The support section 202 is mounted on the upper part of the base 201 so as to be movable up and down. The frame member 203 is mounted on the support section 202. The operation panel 204 is mounted on the upper surface of the front center of the frame member 203. Armrests 203a are formed on the left and right sides of the operation panel 204 for the operator to rest their elbows on when operating the hand controller 220. The height of the armrests 203a can be changed by moving the support section 202 up and down relative to the base 201.

[0024] The viewer unit 210 is supported at the tip of an arm 213 mounted on the upper surface of a support unit 202, and its height can be changed by rotating the joint of the arm 213. The viewer unit 210 includes a display unit (hereinafter referred to as viewer) 211 and a head sensor 212. The viewer 211 displays images (endoscopic images) captured by the endoscope 122. The head sensor 212 is positioned to straddle the area in front of the viewer 211 and is a transmissive photoelectric sensor. When an operator looks into the viewer 211, the operator's head approaches the viewer 211, the area between the head sensors 212 is shielded from light, and the state in which the operator is looking into the viewer 211 is detected.

[0025] The two hand controllers 220 each have seven axes and inter-axis joints and are mounted on the support section 202. The left and right hand controllers 220 are input devices for the operator to operate the arms 111-114 of the arm device 1 using their left and right hands, respectively. The configuration of the hand controllers 220 will be described later with reference to Figure 4.

[0026] The foot unit 230 is installed on the lower front side of the base 201 so as to be movable forward and backward. The foot unit 230 is an input device for the operator to operate the arm device 1 using their left and right feet. The configuration of the foot unit 230 will be described later with reference to Figure 5. By operating the control panel 204, the operator can change the settings of the control device 2, such as the height of the armrest 203a, the height of the viewer unit 210, and the forward and backward position of the foot unit 230.

[0027] Figure 4 is a perspective view showing the external appearance of the hand controller 220.

[0028] The two hand controllers 220 are configured symmetrically in the left-right direction. Each hand controller 220 comprises an operating section 221 and six movable parts 222 to 226.

[0029] The operating section 221 comprises a support shaft 221a, a pair of movable plates 221b, and a pair of hook-and-loop fasteners 221c. The operating section 221 is the part that the operator directly operates with their fingers, and is the object that the operator operates by contacting it.

[0030] The support shaft 221a is a cylindrical member installed on the movable part 222. The pair of movable plates 221b are plate-shaped members provided so as to sandwich the support shaft 221a. The movable plates 221b are installed on the support shaft 221a such that the end of each plate that is closer to the movable part 222 moves closer to and further away from the support shaft 221a. The hook-and-loop fastener 221c is installed on the movable plates 221b. The operating unit 221 includes an encoder 261 (see Figure 12) for detecting the opening and closing angles of the pair of movable plates 221b, and an encoder 271 (see Figure 12) for detecting the amount of rotation of the operating unit 221 relative to the movable part 222.

[0031] The movable parts 222 to 226 constitute an arm connecting the operating unit 221 and the support unit 202. The movable parts 222 to 226 are parts that the operator does not directly touch or operate, but rather parts that the operator operates indirectly through the operating unit 221. Each joint of the arm made up of the movable parts 222 to 226 is formed by the axial rotation of two adjacent movable parts. Each of the movable parts 222 to 226 is equipped with encoders 272 to 276 (see Figure 12) for detecting the amount of rotation of the movable part relative to the movable part adjacent to it on the opposite side from the operating unit 221. That is, encoders 271 to 276 are provided one at each end of the seven axes of the hand controller 220.

[0032] Furthermore, an encoder is not necessarily required to detect the amount of rotation of a movable part. Various condition detectors that detect the position, amount of rotation, angle, presence, etc. of the object to be detected, such as encoders or sensors, can be used.

[0033] Before starting the operation, the operator presses their thumb and index finger against the pair of movable plates 221b so as to sandwich them. In this position, the hook-and-loop fastener 221c is secured to the thumb and index finger that are holding the pair of movable plates 221b. This fixes the thumb and index finger of the left hand to the operating section 221 of the left hand controller 220, and the thumb and index finger of the right hand to the operating section 221 of the right hand controller 220. Note that the fingers used to hold the pair of movable plates 221b are not limited to the thumb and index finger; the operator may choose fingers that are easiest for them to use. For example, the thumb and middle finger may also be used to hold the pair of movable plates 221b.

[0034] When the operator moves the control unit 221 during surgery, each joint of the hand controller 220 rotates axially in accordance with the movement of the control unit 221. Then, based on the output values ​​of encoders 271 to 276, the target arm of the arm device 1 is driven, and the surgical instrument attached to that arm moves. Furthermore, if forceps are attached to the target arm, when the operator opens or closes the pair of movable plates 221b of the control unit 221 during surgery, the tip of the forceps attached to that arm is driven to open or close based on the output value of encoder 261 of the control unit 221.

[0035] Figure 5 is a perspective view showing the external appearance of the foot unit 230.

[0036] The foot unit 230 includes foot pedals 231-237 and foot sensors 241-248.

[0037] The operator provides input to operate the arm device 1 by pressing foot pedals 231 to 237 with their feet. In other words, foot pedals 231 to 237 are controls for making predetermined inputs. Foot pedals 234 to 237 are controls for switching the power on and off of the electrosurgical unit when an electrosurgical unit is attached to the target arm.

[0038] Foot pedal 231, when pressed, switches the surgical instrument operated by the right hand controller 220 to either surgical instrument 123 or 124. Foot pedal 232 is a clutch pedal. While foot pedal 232 is pressed, the operation of the hand controller 220 is not transmitted to the arm device 1. Foot pedal 233 is a camera pedal. While foot pedal 233 is pressed, both hand controllers 220 can be used to operate the arm equipped with the endoscope 122.

[0039] Foot pedals 234 and 236 are incision pedals. While foot pedals 234 and 236 are pressed, high-frequency current is applied to the tip of the forceps (electrosurgical unit) to enable incision. Foot pedals 235 and 237 are coagulation pedals. While foot pedals 235 and 237 are pressed, high-frequency current is applied to the tip of the forceps (electrosurgical unit) to enable coagulation.

[0040] Furthermore, each of the foot pedals 231 to 237 is equipped with a limit sensor 281 to 287 (see Figure 12) for detecting whether or not the foot pedal is pressed. This allows for the detection of whether or not the foot pedals 231 to 237 are being pressed.

[0041] Foot sensors 241-248 are reflective photoelectric sensors for detecting the position of the operator's feet. Foot sensors 241-244 detect whether the left foot is positioned on foot pedals 231-233, and foot sensors 245-248 detect whether the right foot is positioned on foot pedals 234-237. Specifically, foot sensors 241-248 detect hover states, including when the foot is in front of the foot pedal, above the foot pedal, and when the foot pedal is being pressed, as well as non-hover states, where the foot is not inside the foot unit 230.

[0042] Figure 6 is a perspective view showing the operator using the control device 2.

[0043] Before the start of the surgery, the patient is positioned on the operating table below the surgical instruments 121-124, a guide tube (trocar) is inserted into the patient's abdomen, and the surgical instruments 121-124 are inserted into the patient's body through the guide tube (trocar). The operator sits in a chair, placing both arms on the armrests 203a and both feet inside or in front of the foot unit 230. The operator secures their thumb and index finger to the movable plate 221b using hook-and-loop fasteners 221c (see Figure 4). The operator then places their head inside the viewer unit 210 and looks into the viewer 211. When the head sensor 212 detects that the operator's head is positioned inside the viewer unit 210, the operating device 2 becomes capable of operating the arm device 1.

[0044] Once surgery begins, the operator operates the left and right control units 221 (see Figure 4) to input grasping operations for the forceps and movement operations for moving the tips of the forceps. In response to the operations input by the operator to the control device 2, the arms 111-114 of the arm device 1 and the surgical instruments 121-124 are driven. Various surgeries are then performed in this manner.

[0045] The operator moves their head away from the viewer unit 210 when giving instructions to assistants in the operating room, taking a break, or ending surgery. When the head sensor 212 detects that the operator's head is not positioned inside the viewer unit 210, the operating device 2 becomes unable to operate the arm device 1. This prevents malfunction of the arm device 1.

[0046] Figure 7 is a block diagram showing the configuration of the control device 3.

[0047] The control device 3 comprises a control unit 31, a storage unit 32, and a communication unit 33. The control unit 31 is configured, for example, by a CPU. The control unit 31 controls each part of the hardware of the control device 3 and performs various processes by executing computer programs stored in the storage unit 32. The storage unit 32 is configured, for example, by an SSD or HDD. The communication unit 33 is equipped with a communication interface that can communicate with the arm device 1, the operating device 2, and the storage device 310 (see Figure 8) based on a predetermined communication standard.

[0048] Figure 8 is a block diagram showing the configuration of the information processing system 5.

[0049] The information processing system 5 comprises a storage device 310 and an information processing device 320. The network 330 is, for example, the internet.

[0050] The storage device 310 comprises a control unit 311, a storage unit 312, and a communication unit 313. The control unit 311 is configured, for example, by a CPU. The control unit 311 controls various parts of the storage device 310's hardware and performs various processes by executing computer programs stored in the storage unit 312. The storage unit 312 is configured, for example, by an SSD or HDD. The storage device 310 is connected to the information processing device 320 via a network 330 so as to be able to communicate with it. The communication unit 313 comprises a communication interface that can communicate with the information processing device 320, the control device 3, the video processing device 302 (described later), and the operating room camera 301 (see Figure 9) based on a predetermined communication standard such as Ethernet or Wi-Fi.

[0051] The information processing device 320 comprises a control unit 321, a storage unit 322, and a communication unit 323. The control unit 321 is configured, for example, by a CPU. The control unit 321 controls various parts of the hardware of the information processing device 320 and performs various processes by executing computer programs stored in the storage unit 322. The storage unit 322 is configured, for example, by an SSD or HDD. The communication unit 323 is configured with a communication interface that can communicate with the storage device 310 and the observation terminal 340 (see Figure 10) based on a predetermined communication standard such as Ethernet or Wi-Fi.

[0052] Figure 9 shows an overview of the connections between the surgical robot 4 and the information processing system 5, as well as the information transmitted and received between each device.

[0053] The operating room is equipped with an arm device 1, an operating device 2, a control device 3, an endoscope 122 attached to the arm device 1, an image processing device 302 for processing images obtained by the endoscope 122, an operating room camera 301 for imaging the entire operating room, and a storage device 310. The control device 3, the image processing device 302, and the operating room camera 301 are communicated with the storage device 310. An information processing device 320 is installed outside the operating room, for example in a cloud environment, and is communicated with the storage device 310.

[0054] Furthermore, the operating device 2 and the storage device 310 do not necessarily have to be installed in the operating room; they may be installed in other rooms within the facility.

[0055] The operating room camera 301 transmits the captured video (operating room image) to the storage device 310 based on a request from the storage device 310. The endoscope 122 is connected to the image processing device 302. The image processing device 302 transmits the video (endoscopic image) captured by the endoscope 122 to the operating device 2. The image processing device 302 transmits the endoscopic image captured by the endoscope 122 to the storage device 310 based on a request from the storage device 310. The operating device 2 displays the endoscopic image received from the image processing device 302 on the viewer 211 (see Figure 3).

[0056] As described above, the operator, who is a physician, operates the control device 2 to drive the arm device 1 and perform surgery.

[0057] At this time, the operating device 2 transmits a drive instruction to the control device 3 in real time in response to the operator's operation, that is, each time the encoders 261, 271-276 and limit sensors 281-287 (see Figure 12) generate an output, based on the output generated. The operating device 2 also transmits a current value indicating the state of the object operated by the operating device 2, which changes as a result of the operator's operation, to the control device 3 at predetermined time intervals (e.g., every second). Based on a request from the storage device 310, the control device 3 transmits a state log generated from multiple current values ​​received from the operating device 2 to the storage device 310. The control device 3 converts the drive instruction received from the operating device 2 for use with the arm device 1 and transmits the converted drive instruction to the arm device 1 in real time, that is, each time it receives a drive instruction from the operating device 2. As a result, the arm device 1 operates in response to the operator's operation of the operating device 2. The arm device 1 transmits a current value indicating the operation of arms 111-114, etc., to the control device 3 at predetermined time intervals (e.g., every second). Based on a request from the storage device 310, the control device 3 transmits an operation log generated from the current value received from the arm device 1 to the storage device 310.

[0058] The storage device 310 stores the status log and operation log received from the control device 3 in the storage unit 312. The storage device 310 also stores the video (operating room image) received from the operating room camera 301 and the video (endoscopic image) received from the video processing device 302 in the storage unit 312. The reception of the operating room image, endoscopic image, status log, and operation log continues for a predetermined period, for example, from the start to the end of use of the operating room where surgery is performed. The storage device 310 transmits the operating room image, endoscopic image, status log, and operation log stored in the storage unit 312 to the information processing device 320 in real time.

[0059] Furthermore, the control unit 311 of the storage device 310 may process the operating room images stored in the memory unit 312 as needed. For example, the control unit 311 may blur information that could lead to the estimation of personal information of people or patients contained in the operating room images, so that people and personal information cannot be identified.

[0060] The information processing device 320 stores the operating room images, endoscopic images, status logs, and operation logs received from the storage device 310 in the storage unit 322. In response to a request from the observation terminal 340 (see Figure 10), the information processing device 320 generates a surgical image including the operating room image, the endoscopic image, a reconstructed image based on the status log, and a reconstructed image based on the operation log, and transmits it to the observation terminal 340.

[0061] For an observer attempting to learn how to operate the control device 2, simply watching a skilled operator (expert) operate the device from the side will not be sufficient to fully learn its operation. For example, if the view of the object being operated (operating unit 221 or foot pedals 231-237) is obstructed by part of the control device 2, the observer will not be able to grasp what operations are actually being performed. Furthermore, if multiple objects are being operated simultaneously, the observer must keep multiple objects in their field of view, making it difficult to grasp multiple operations.

[0062] Furthermore, if the pair of movable plates 221b of the control unit 221 are opened or closed only slightly, it is difficult to determine the extent of the operation by looking at the control unit 221. Also, if the feet are positioned in front of or above the foot pedals 231-237, it is not possible to grasp this preparation state by simply looking at the operating room images or endoscopic images.

[0063] In contrast, in this embodiment, surgical images, including operating room images, endoscopic images, reconstructed images based on a state log, and reconstructed images based on an operation log, are transmitted to an observation terminal 340 (see Figure 10) used by the observer. The reconstructed images based on the state log include images visualizing the operator's actions on the target being operated by the operating device 2. The display unit 343 of the observation terminal 340 displays images visualizing the operator's actions on the target being operated by the operating device 2. By referring to these images together with images showing the operation of the surgical robot 4 (arm device 1 and / or operating device 2) and / or the operation of the surgical instruments 121-124, the observer can learn the operation of the surgical robot 4 or surgical instruments 121-124 and the operations of the target being operated for that purpose.

[0064] Figure 10 is a block diagram showing the usage configuration of the information processing device 320.

[0065] The observation terminal 340 is a computer operated by an observer who is learning how to operate the operating device 2, which is operated by an expert. The observation terminal 340 comprises a control unit 341, a storage unit 342, a display unit 343, an input unit 344, and a communication unit 345.

[0066] The control unit 341 is configured, for example, by a CPU. The control unit 341 controls various parts of the hardware of the observation terminal 340 and performs various processes by executing computer programs stored in the storage unit 342. The storage unit 342 is configured, for example, by an SSD or HDD. The display unit 343 is configured, for example, by a liquid crystal display. The input unit 344 is configured, for example, by a keyboard or mouse. The display unit 343 and the input unit 344 may be configured by touch panel displays. The communication unit 345 is equipped with a communication interface that can communicate with the information processing device 320 based on a predetermined communication standard such as Ethernet or Wi-Fi.

[0067] The control unit 341 of the observation terminal 340 acquires surgical images from the information processing device 320 for learning how to operate the operating device 2 by running a web browser or a predetermined application. The control unit 341 then displays the acquired surgical images on the display unit 343. The observer refers to the surgical images displayed on the display unit 343 and learns how to operate the operating device 2 by the operator, who is a physician. The surgical images displayed on the display unit 343 will be explained later with reference to Figure 17 and subsequent figures.

[0068] Figure 11 is a block diagram showing the configuration of the arm device 1.

[0069] The arm device 1 comprises a control unit 131, a storage unit 132, a communication unit 133, a plurality of operating units 140, and sensors 151 to 154.

[0070] The control unit 131 is composed of, for example, an FPGA or a CPU. The storage unit 132 is composed of, for example, ROM and RAM. The communication unit 133 has a communication interface that can communicate with the control device 3 based on a predetermined communication standard.

[0071] Each operating unit 140 corresponds to the operating parts of the base arm 103 and arms 111-114 of the arm device 1. Each operating unit 140 corresponds, for example, to one joint of an arm. Each operating unit 140 includes a motor 141 and an encoder 142. The motor 141 is a stepping motor. The encoder 142 outputs the amount of drive of the motor 141. One encoder 142 is provided at one end of each of the 12 axes of each arm 111-114.

[0072] Sensors 151 to 154 are installed on arms 111 to 114, respectively, and detect the attachment and detachment of surgical instruments to and from arms 111 to 114. Sensors 151 to 154 may be, for example, sensors that detect when a surgical instrument is electrically connected to an arm, or photoelectric sensors that detect when a surgical instrument is physically attached or detached.

[0073] The surgical instruments attached to the arm device 1 are marked with their name and serial number. When a surgical instrument is attached to arms 111 to 114, the control unit 131 stores the name and serial number of the surgical instrument in the storage unit 132.

[0074] The control unit 131 drives the motor 141 of the corresponding operating unit 140 based on the drive instruction (see Figure 9) received from the control device 3 via the communication unit 133. The control unit 131 also transmits the current value that constitutes the operation log based on the output value of the encoder 142, the detection signals of sensors 151 to 154, and the names and serial numbers of surgical instruments 121 to 124 to the control device 3 via the communication unit 133. The operation log, the detection signals of sensors 151 to 154, and the names and serial numbers of surgical instruments 121 to 124 are transmitted from the control device 3 to the information processing device 320 via the storage device 310 and stored in the storage unit 322 of the information processing device 320.

[0075] Figure 12 is a block diagram showing the configuration of the operating device 2.

[0076] The operating device 2 comprises a control unit 251, a storage unit 252, a communication unit 253, a viewer 211, a head sensor 212, an operating unit 221, movable parts 222-226, foot pedals 231-237, and foot sensors 241-248. For convenience, Figure 12 shows the configuration of one of the left or right hand controllers 220.

[0077] The control unit 251 is configured, for example, by an FPGA or CPU. The storage unit 252 is configured, for example, by ROM and RAM. The communication unit 253 is equipped with a communication interface that can communicate with the endoscope 122 and the control device 3 based on a predetermined communication standard.

[0078] The control unit 221 is equipped with encoders 261 and 271. The movable parts 222 to 226 are each equipped with encoders 272 to 276. The foot pedals 231 to 237 are each equipped with limit sensors 281 to 287. Encoders 261, 271 to 276 output the amount of movement or rotation of the corresponding part.

[0079] Specifically, encoder 261 of the operation unit 221 outputs the amount by which the pair of movable plates 221b (see Figure 4) of the operation unit 221 are pressed. Encoders 271 to 276 output the amount of rotation between the operation unit 221 and the movable part 222, the amount of rotation between the movable parts 222 and 223, the amount of rotation between the movable parts 223 and 224, the amount of rotation between the movable parts 224 and 225, the amount of rotation between the movable parts 225 and 226, and the amount of rotation between the movable part 226 and the support unit 202 (see Figure 3), respectively. Limit sensors 281 to 287 detect whether the foot pedals 231 to 237 are pressed or not. The detection signals from limit sensors 284 to 287 are information indicating the switching of power supply and de-power supply by the foot pedals 234 to 237.

[0080] The control unit 251 displays the endoscopic image received from the image processing device 302 on the viewer 211. The control unit 251 transmits the current values ​​that constitute the state log, based on the output values ​​of encoders 261, 271-276 and detection signals from limit sensors 281-287, to the control device 3 via the communication unit 253. The state log is transmitted from the control device 3 to the information processing device 320 via the storage device 310 and stored in the storage unit 322 of the information processing device 320.

[0081] Figure 13 is an illustrative diagram showing the current values ​​that constitute the status log transmitted from the operating device 2 to the control device 3. The information, such as the current values ​​shown in Figure 13, stored together with the current time for a predetermined period and at predetermined intervals, constitutes the status log.

[0082] In the example shown in Figure 13, the current values ​​indicating the current state of each part include the output values ​​of encoders 261, 271-276 corresponding to the left and right hand controllers 220, the angle between the pair of movable plates 221b calculated from the output value of encoder 261 corresponding to the left operating unit 221, and the angle between the pair of movable plates 221b calculated from the output value of encoder 261 corresponding to the right operating unit 221. The angle between the pair of movable plates 221b is calculated by the control unit 251 of the operating device 2.

[0083] In Figure 13, the current values ​​for axes 1_1 to 1_7 of the hand controller (right) represent the output values ​​of encoders 271 to 276, each located at one end of the seven axes of the right hand controller 220. The current value of 1_grip angle of the hand controller (right) represents the angle between the pair of movable plates 221b, calculated from the output value of encoder 261, which corresponds to the right operating unit 221. Similarly, the current values ​​for axes 1_1 to 1_7 of the hand controller (left) represent the output values ​​of encoders 271 to 276, each located at one end of the seven axes of the left hand controller 220. The current value of 1_grip angle of the hand controller (left) represents the angle between the pair of movable plates 221b, calculated from the output value of encoder 261, which corresponds to the left operating unit 221. The current values ​​that make up the state log also include the detection signals of limit sensors 281 to 287, which correspond to foot pedals 231 to 237, the detection signals of foot sensors 241 to 248, and the detection signal of head sensor 212.

[0084] The control unit 251 of the operating device 2 transmits such current values ​​to the control device 3, for example, at 1-second intervals. The control device 3 stores the received set of multiple current values ​​and multiple current times corresponding to each current value as a state log in the storage unit 32.

[0085] Figure 14 shows an example of a state log. Figure 14 shows the state log of the angle between a pair of movable plates 221b calculated from the output value of the encoder 261 corresponding to the left operating unit 221, and includes the current time every second within a predetermined period and the current value (angle) at that time.

[0086] Figure 15 is an illustrative diagram showing the current values ​​that constitute the operation log transmitted from the arm device 1 to the control device 3. The operation log is information that stores current values, as shown in Figure 15, along with the current time, at predetermined intervals for a predetermined period. For convenience, Figure 15 shows current values ​​corresponding to various parts of arm 111 and parts of arm 112.

[0087] In Figure 15, the current values ​​for axes 1_1 to 1_12 of operation arm 1 represent the output values ​​of encoders 142, each located at one end of the 12 axes of arm 111. Similarly, the current values ​​for axes 2_1 to 2_12 of operation arm 2 represent the output values ​​of encoders 142, each located at one end of the 12 axes of arm 112 (the current values ​​for axes 2_5 to 2_12 are omitted in Figure 15).

[0088] When the control device 3 receives a drive instruction from the operating device 2, it processes the received drive instruction for use with the arm device 1 and transmits it to the arm device 1. The arm device 1 drives each motor 141 based on the received drive instruction.

[0089] The current values ​​that make up the operation log include the output values ​​of each encoder 142, which indicate the amount of movement of each motor 141 that operated based on the drive instruction, as the current values ​​of each part.

[0090] The control unit 131 of the arm device 1 transmits such current values ​​to the control device 3, for example, at 1-second intervals. The control device 3 stores the set of multiple received current values ​​and multiple current times corresponding to each current value as an operation log in the storage unit 132.

[0091] Figure 16 shows an example of an operation log. Figure 16 shows the operation log of the amount of motor movement at a predetermined movable part of arm 111, and includes the current time every second within a predetermined period and the current value (amount of movement) at that time.

[0092] Next, we will explain the image displayed on the display unit 343 of the observation terminal 340 (see Figure 10).

[0093] When the control unit 321 of the information processing device 320 receives instruction information from the observation terminal 340, it transmits the endoscopic images and operating room images stored in the storage unit 322 to the observation terminal 340, and also generates various images and information based on the state log and operation log stored in the storage unit 322, and transmits them to the observation terminal 340. The observation terminal 340 displays the surgical image 400, which includes the endoscopic images, operating room images, and various information based on the state log and operation log, on the display unit 343. The display of the surgical image 400 on the display unit 343 of the observation terminal 340 may be so-called download playback, where the display starts after receiving images and information for the entire period from the information processing device 320, or it may be so-called streaming playback, where the display is performed while receiving images and information from the information processing device 320.

[0094] The images displayed on the display unit 343 are not limited to images transmitted from the information processing device 320 to the observation terminal 340. The images displayed on the display unit 343 may also be images generated by the control unit 341 of the observation terminal 340 based on status logs and operation logs received from the information processing device 320.

[0095] Figure 17 is a schematic diagram showing the composition of surgical image 400.

[0096] The surgical image 400 includes an endoscopic image 401, an operating room image 402, schematic images 403 and 404, display switching buttons 411 and 412, a video control area 420, and an information display area 430. In addition, the surgical image 400 includes an image 440 showing information from the head sensor 212, an image 450 showing the usage record of surgical instruments, and an image 460 showing the number of views. Images 440, 450, and 460 will be explained later with reference to Figure 20.

[0097] As shown in Figure 17, the endoscopic image 401, the operating room image 402, and the schematic images 403 and 404 are all videos. Schematic image 403 is a video that three-dimensionally represents the operating state of each part of the arm device 1, based on the operation log of the arm device 1. Schematic image 404 is a video that three-dimensionally represents the operating state of the hand controller 220 of the operating device 2, based on the state log of the operating device 2. These four videos are synchronized with each other and show the state at the same time.

[0098] The display switching buttons 411 and 412 are used to switch between a mode in which four videos (endoscopic image 401, operating room image 402, and schematic images 403 and 404) are displayed side by side, and a mode in which one of the four videos is displayed in an enlarged view. When the display switching button 411 is operated, the four videos are displayed side by side as shown in Figure 17, and when the display switching button 412 is operated, one video is displayed in an enlarged view as shown in Figure 18.

[0099] When the display switching button 411 is operated, a screen is displayed for setting the positions of the four videos, and by setting on this screen, the four videos can be displayed in the desired positions, as shown in Figure 17. When the display switching button 412 is operated, a screen is displayed for setting which of the four videos to display, and by setting on this screen, one of the four videos can be displayed, as shown in Figure 18.

[0100] Figure 19 is a schematic enlarged view showing the composition of endoscopic image 401.

[0101] The endoscopic image 401 displays display areas 401a to 401e, which show information about surgical instruments 121 to 124 attached to arms 111 to 114 of the arm device 1, along with the image acquired by the endoscope 122. The endoscopic image 401 is the same image as the image displayed on the viewer 211 of the operating device 2.

[0102] Display area 401a indicates the operating status of the clutch pedal, foot pedal 232. When a foot is positioned relative to foot pedal 232, the outer edge of display area 401a is colored. When foot pedal 232 is pressed, the inside of display area 401a is colored. Display area 401b indicates the operating status of foot pedal 233 for operating the arm to which the endoscope 122 is attached. When a foot is positioned relative to foot pedal 233, the outer edge of display area 401b is colored, and when foot pedal 233 is pressed, the inside of display area 401b is colored.

[0103] The display area 401c indicates the operating status of the foot pedals 234 and 235 for operating the forceps (electrosurgical unit). When a foot is positioned relative to either the foot pedal 234 or 235, the outer perimeter of the display area 401c is colored. When the foot pedal 234 is pressed, the interior of the display area 401c is colored with a first color. When the foot pedal 235 is pressed, the interior of the display area 401c is colored with a second color. The first color is, for example, light blue, and the second color is, for example, yellow.

[0104] Display area 401d shows the operating status of foot pedals 236 and 237 for operating forceps (electrosurgical units), and display area 401e shows the operating status of foot pedals 236 and 237 for operating other forceps (electrosurgical units). As described above, each time foot pedal 231 is pressed, the target of foot pedals 236 and 237 is switched between forceps 123 and 124. When a foot is positioned relative to foot pedal 236 or 237, the outer perimeter of the target display area in display areas 401d and 401e is colored, when foot pedal 236 is pressed, the interior of the target display area in display areas 401d and 401e is colored with a first color, and when foot pedal 237 is pressed, the interior of the target display area in display areas 401d and 401e is colored with a second color.

[0105] Furthermore, display areas 401c to 401e display information about surgical instruments 121, 123, and 124 attached to arms 111, 113, and 114, such as their names. The presence or absence of surgical instruments and the names of the surgical instruments are generated based on the detection signals from sensors 151 to 154 and the names and serial numbers of the surgical instruments 121 to 124 acquired when they were attached to arms 111 to 114.

[0106] By referring to display areas 401a to 401e within the endoscopic image 401, the observer can understand the operating status of the foot pedals 232 to 237 and determine what surgical instruments are attached to the arms 111 to 114.

[0107] Furthermore, as shown in Figure 19, dividing lines 401f can be displayed within the endoscopic image 401, dividing the region of the endoscopic image 401 into four sections. This allows the observer to use the four regions divided by the dividing lines 401f as a guide to refer to the region in which each surgical instrument is located, and to learn how to position and move each surgical instrument according to the surgical situation.

[0108] Returning to Figure 17, the video control area 420 includes a play button 421, a playback position marker 422, a playback speed setting button 423, a warning marker 424, a bookmark insertion button 425, a bookmark marker 426, and an editing start button 427.

[0109] When the play button 421 is operated, playback and pause of the endoscopic image 401, the operating room image 402, and the schematic images 403 and 404 are switched.

[0110] The playback position mark 422 indicates the playback position of the endoscopic image 401, the operating room image 402, and the schematic images 403 and 404 on the timeline 420a. The playback position can be changed by operating the playback position mark 422. When the playback speed setting button 423 is operated, a submenu opens, and the playback speed can be set within the submenu. The warning mark 424 indicates the position on the timeline 420a where an event requiring the observer's attention has occurred, such as the position on the timeline 420a where an error occurred in the surgical robot 4.

[0111] When the bookmark insertion button 425 is operated, a bookmark mark 426 is added at the position of the playback position mark 422. In addition, a screenshot of the surgical image 400 at the time the bookmark insertion button 425 is operated is stored in the storage unit 322 of the information processing device 320. The storage of the screenshot of the surgical image 400 will be explained later with reference to Figure 25.

[0112] When the editing start button 427 is pressed, the observer can write text and shapes on the surgical image 400 by clicking and dragging. If the bookmark insertion button 425 is pressed in this state, a screenshot of the surgical image 400, along with the text and shapes written on it, is stored in the storage unit 322 of the information processing device 320. The fact that the editing start button 427 enables the writing of text and shapes on the surgical image 400 will be explained later with reference to Figure 26.

[0113] Figure 20 is a schematic diagram showing the composition of images 440, 450, and 460 displayed within surgical image 400.

[0114] The band areas in images 440, 450, and 460 correspond to the timeline 420a within the video control area 420 shown in Figure 17.

[0115] The band area in image 440, which shows information from the head sensor 212, includes parts displayed in black and parts displayed in white. The parts displayed in black indicate that the head sensor 212 has detected that the operator is looking into the viewer 211, that is, that the operator's approach to the viewer 211 has been detected. The parts displayed in white indicate that the head sensor 212 has not detected that the operator is looking into the viewer 211, that is, that the operator's approach to the viewer 211 has not been detected.

[0116] Image 450, which shows the record of surgical instrument usage, includes band areas numbered 1 to 4. Band areas 1 to 4 correspond to arms 111 to 114, respectively. The hatched areas within the band areas indicate the type of forceps or endoscope attached to arms 111 to 114 and the duration of attachment. In the example shown in Figure 20, band areas 1 and 3 show that the forceps attached to arms 111 and 113 were replaced during the surgery. Band area 2 shows that the endoscope 122 attached to arm 112 was removed several times towards the end of the surgery. This was because the endoscope 122 was removed from arm 112 to wipe off condensation from the head of the endoscope 122. Band area 4 shows that the forceps 124 attached to arm 114 was never removed.

[0117] Furthermore, when a cursor such as a mouse is placed over a band area in image 450, surgical instrument information 451 is displayed, including the name of the forceps or endoscope that was attached at the time and position where the cursor was placed, and the time period during which the forceps or endoscope was used, as shown in band area 4.

[0118] The band in image 460, which shows the number of views, contains areas of varying shades. These shades represent the number of views by the observer at each location. For example, the white areas indicate that the observer has viewed this area 0 times, and each increase in density indicates that the number of views has increased by a predetermined number.

[0119] Furthermore, when a mouse cursor or other cursor is placed over the band area within image 460, view count information 461, indicating the view count at the time the cursor was placed, will be displayed.

[0120] Figure 21 is a schematic enlarged view showing the configuration of the information display area 430 when the operation information tab is selected.

[0121] The information display area 430 displays content synchronized with the playback position of the endoscopic image 401, the operating room image 402, and the schematic images 403 and 404.

[0122] When the operation information tab is selected, the information display area 430 comprises a display selection area 431, two images 432a, two images 432b, two images 433a, two images 433b, two images 434, and images 435-437. Images 432a, 432b, 433a, 433b, and 434 are arranged on the left and right sides within the information display area 430.

[0123] Image 432a on the left schematically shows the pressed state of the pair of movable plates 221b of the left hand controller 220, and image 432a on the right schematically shows the pressed state of the pair of movable plates 221b of the right hand controller 220. Image 432b on the left shows the angle corresponding to the amount of pressure applied to the pair of movable plates 221b of the left hand controller 220, and image 432b on the right shows the angle corresponding to the amount of pressure applied to the pair of movable plates 221b of the right hand controller 220. Images 432a and 432b are generated based on the state log. By referring to images 432a and 432b, the observer can understand to what extent the pair of movable plates 221b of the operating unit 221 are open or closed.

[0124] Images 433a on the left and right show the clamping state of the forceps tips operated by the left and right hand controllers 220, respectively. Images 433b on the left and right show the angles corresponding to the amount of clamping of the forceps operated by the left and right hand controllers 220, respectively. Images 433a and 433b are generated based on the operation log. By referring to images 433a and 433b, the observer can understand how much the corresponding forceps tips are open and closed.

[0125] The left image 434 shows a graph corresponding to the pressing state of foot pedals 234 and 235, and the right image 434 shows a graph corresponding to the pressing state of foot pedals 236 and 237. In image 434, ON indicates that the foot pedal is pressed, and OFF indicates that the foot pedal is not pressed. Image 434 shows the pressing state over a 30-second period in the past. Image 434 is generated based on the state log. By referring to the left image 434, the observer can understand the timing and duration when foot pedals 234 and 235 were pressed, and by referring to the right image 434, the observer can understand the timing and duration when foot pedals 236 and 237 were pressed.

[0126] Image 435 schematically shows the position of the operator's foot on the foot unit 230 and the pressing state of the foot pedals 231-237. Image 435 is generated based on the state log.

[0127] Figures 22 and 23 schematically illustrate the structure and transitions of image 435.

[0128] As shown in Figure 22, Image 435 includes pedal images 471 to 477 corresponding to foot pedals 231 to 237, and a foot image 481 displayed according to the standby state of the foot.

[0129] The control unit 321 of the information processing device 320 determines the hover state and non-hover state for each foot pedal 231 to 237 based on the detection signals from the foot sensors 241 to 248. As described above, the hover state includes the state in front of the foot pedal, the state above the foot pedal, and the state in which the foot pedal is pressed down, while the non-hover state includes the state in which the foot is not inside the foot unit 230. Based on the detection signals from the limit sensors 281 to 287, the control unit 321 determines whether each foot pedal 231 to 237 is pressed down or not. The control unit 321 then generates an image 435 based on the determination result.

[0130] As shown in the upper part of Figure 22, if the left and right feet are not positioned within the foot unit 230, the pedal images 471-477 are displayed with the normal border thickness, and the foot image 481 is not displayed. This indicates that both feet are not in a position to operate or prepare to operate the foot pedals 231-237.

[0131] From the state shown in the upper part of Figure 22, when the feet are positioned in front of or above the foot pedals, the foot image 481 is displayed as shown in the lower part of Figure 22. In the example shown in the lower part of Figure 22, the two foot images 481 are positioned in front of the pedal images 473 and 475, respectively. This indicates that the left foot is positioned in front of or above the foot pedal 233, and the right foot is positioned in front of or above the foot pedal 235. It also indicates that both feet are ready to operate the foot pedals 233 and 235.

[0132] From the state shown in the lower part of Figure 22, when the foot pedal is pressed down, the outer edge of the corresponding pedal image is colored, as shown in the upper part of Figure 23. In the example shown in the upper part of Figure 23, the outer edge of pedal image 477 is colored. This indicates that foot pedal 237 is pressed down.

[0133] From the upper state of Figure 23, when the foot pedal 233 is pressed, the outer edge of the pedal image 473 is colored, as shown in the lower state of Figure 23. Also, in the lower state of Figure 23, the right foot has moved outside the foot unit 230, so there is no foot image 481 corresponding to the right foot. This indicates that the right foot is not in a ready state for operation.

[0134] The observer can understand the operation and preparation status of the foot pedals 231-237 by referring to the pedal images 471-477 and foot image 481 in image 435.

[0135] Returning to Figure 21, Image 436 shows the detection results based on the head sensor 212 and the cumulative time the operator has looked into the viewer 211 as detected by the head sensor 212. Image 436 is generated based on the status log. By referring to Image 436, the observer can determine whether the operator is looking into the viewer 211 and how long they have looked into the viewer 211 cumulatively during the surgery.

[0136] Image 437 shows the names and serial numbers of surgical instruments 121-124 attached to arms 111-114 of arm device 1. Image 437 is generated based on the detection signals from sensors 151-154 and the names and serial numbers of surgical instruments 121-124 acquired when they were attached to arms 111-114. By referring to Image 437, the observer can determine the current names and serial numbers of surgical instruments 121-124.

[0137] Figure 24 is a schematic enlarged view showing the configuration of the information display area 430 when the settings information tab is selected.

[0138] When the settings information tab is selected, the information display area 430 includes a display selection area 431 and images 438 and 439.

[0139] Image 438 schematically shows the pivot positions of surgical instruments 121-124 attached to arms 111-114. A pivot position is the position on a surgical instrument where the relative position between the instrument and the arm to which it is attached remains unchanged as the instrument moves in accordance with the movement of arms 111-114. Surgical instruments 121-124 are moved by arms 111-114 using the pivot positions as fulcrums. Image 438 shows the distances between each pivot position.

[0140] Image 439 shows the settings for the operating device 2. Image 439 shows the load (weight of the hand controller 220) when the operator operates the operating unit 221, the ratio (scaling) of the amount of movement of the hand controller 220 to the amount of movement of the actual surgical instrument, the height of the armrest 203a, and the front-to-back position (depth) of the foot unit 230.

[0141] The observer can understand the pivot positions of surgical instruments 121-124 and the settings of the operating device 2 by referring to images 438 and 439.

[0142] Furthermore, the display selection area 431 may include tabs other than operation information and setting information, and various information may be displayed in the information display area 430 when other tabs are selected. For example, information about other devices related to surgery may be displayed.

[0143] Figure 25 schematically shows the configuration of the edited image 500 that is displayed when the bookmark insertion button 425 is operated in the surgical image 400.

[0144] When the bookmark insertion button 425 is pressed on the surgical image 400, the surgical image 400 at that point in time is automatically saved as a screenshot 502 in the storage unit 322 of the information processing device 320.

[0145] The edited image 500 includes a bookmark display area 501, a screenshot 502, an editing tool area 503, a text input area 504, a close button 511, and a loop playback button 512.

[0146] The bookmark display area 501 displays the bookmark item 501a corresponding to the set bookmark. When bookmark item 501a is manipulated, it becomes selected, and the screenshot 502, image 502a, and text 504a corresponding to the selected bookmark item 501a are displayed. If the bookmark insertion button 425 is operated in the surgical image 400 and the edited image 500 is displayed, bookmark item 501a becomes selected. Also, if the trash can icon within bookmark item 501a is manipulated, the corresponding bookmark is deleted.

[0147] Screenshot 502 is a screenshot of the surgical image 400 corresponding to the selected bookmark item 501a. The observer can insert a hand-drawn image 502a into Screenshot 502 by dragging the mouse over it. The observer can also change the color and line thickness of the shapes drawn on Screenshot 502 by operating the various buttons and sliders in the editing tool area 503. The observer can add text 504a to the bookmark item 501a by selecting the text input area 504 and typing characters via the keyboard.

[0148] When the close button 511 is pressed, the image 502a and text 504a set within the edited image 500 are stored in the storage unit 322 of the information processing device 320, associated with the screenshot 502. Subsequently, the edited image 500 is closed, and the surgical image 400 is displayed again. Because the screenshot 502, image 502a, and text 504a are saved, the observer can then refer to the saved content to smoothly learn the operator's operations.

[0149] When the loop playback button 512 is pressed, the image 502a and text 504a set within the edited image 500 are stored in the storage unit 322 of the information processing device 320, in association with the screenshot 502, similar to the close button 511. The edited image 500 is then closed and the surgical image 400 is displayed again. In this case, the video corresponding to the bookmark item 501a that was selected when the edited image 500 was closed (for example, ±5 seconds) is repeatedly played within the surgical image 400. This allows the observer to repeatedly refer to each image for scenes they wish to focus on, facilitating smooth learning.

[0150] The number of times the video is repeated when the loop playback button 512 is operated is not particularly limited; playback may continue until the observer instructs the system to stop playback, or it may end after a predetermined number of repetitions. Furthermore, the video does not necessarily need to be played repeatedly; it may end after being played once.

[0151] When a lecture is conducted using the observation terminal 340, the instructor can insert a hand-drawn image 502a to draw the attention of the lecture participants to the part indicated by the image 502a. This allows the participants to smoothly learn the operations performed by the operator.

[0152] Figure 25 illustrates the case where the bookmark insertion button 425 is operated in the surgical image 400, and then a hand-drawn image 502a is inserted into the screenshot 502. However, it is also possible to insert a hand-drawn image in the surgical image 400 first.

[0153] Figure 26 schematically shows the configuration of the surgical image 400 when the hand-drawn image 491 is inserted first.

[0154] In the surgical images 400 shown in Figures 17 and 18, when the edit start button 427 is operated, the editing tool area 428 is displayed in the video control area 420 of the surgical image 400 in place of the edit start button 427, as shown in Figure 26. The editing tool area 428 includes an icon for changing the color of hand-drawn lines and an end icon for ending drawing.

[0155] The observer inserts a hand-drawn image 491 into the surgical image 400 shown in Figure 26 by operating the mouse and then operates the bookmark insertion button 425. As a result, a screenshot 502 of the surgical image 400 and the hand-drawn image 491 are automatically saved in the storage unit 322 of the information processing device 320. Then, the edited image 500, including the screenshot 502 with the hand-drawn image 491 inserted, is displayed. Note that the insertion of the hand-drawn image 491 may be performed while the surgical image 400 is being played back, or it may be performed after the playback of the surgical image 400 has been temporarily stopped.

[0156] Next, with reference to Figures 27-30, the processing performed by the control unit 321 of the information processing device 320 will be explained.

[0157] Figure 27 is a flowchart showing the process of receiving the information necessary to generate surgical images 400.

[0158] In step S1, the control unit 321 of the information processing device 320 receives endoscopic images, operating room images, status logs, and operation logs transmitted in real time from the storage device 310 and stores them in the storage unit 322. The status logs include logs showing the status (grip angle) of the operating unit 221 based on the output of the encoder 261, logs showing the status (amount of movement) of the movable parts 222 to 226 based on the respective outputs of the encoders 272 to 276, logs showing the status (e.g., powered on or not powered on) of the foot pedals 231 to 237 based on the respective outputs of the limit sensors 281 to 287, and logs showing the status of the viewer 211 (whether or not the operator is approaching the viewer) based on the output of the head sensor 212. The operation logs include logs showing the operation (amount of movement) of the operating unit 140 of the arm device 1 based on the output of the encoder 142.

[0159] Figure 28 is a flowchart showing the process for generating surgical images 400. Each step in Figure 28 is performed in response to a request from the observation terminal 340.

[0160] In step S2, the control unit 321 generates first to third reconstructed images based on the state log and the operation log. The first reconstructed image includes images 432a, 432b, 434, 435, and 440 that visualize the operation on the operating object (operating unit 221, foot pedals 231 to 237 and viewer 211) that the operator touches and operates. The second reconstructed image includes image 435 that schematically shows the state of preparation for operation on the operating object (foot pedals 231 to 237) (a state in which the foot is positioned in front of or above the foot pedal). The third reconstructed image includes images 433a and 433b that visualize the operation of the forceps attached to the arm of the arm device 1.

[0161] Furthermore, in step S2, the control unit 321 generates a schematic image 404 based on the state log, specifically the log showing the state of the operation unit 221 (grip angle; in Figure 13, the 1_grip angle of the right hand controller and the 1_grip angle of the left hand controller) and the log showing the state of the movable parts 222 to 226 (amount of movement; in Figure 13, the 1_1 axis to 1_7 axis of the right hand controller and the 1_1 axis to 1_7 axis of the left hand controller), and generates a schematic image 403 based on the operation log.

[0162] In step S3, the control unit 321 generates a surgical image 400 based on the endoscopic image 401, the operating room image 402, the schematic images 403 and 404 generated in step S2, and the first to third reconstructed images generated in step S2. In step S4, the control unit 321 adds additional information to the surgical image 400 generated in step S3.

[0163] Figure 29 is a flowchart detailing the additional information addition process in step S4 of Figure 28.

[0164] In step S11, the control unit 321 of the information processing device 320 adds to the surgical image 400 information indicating the portion of the endoscopic image 401 that has already been viewed from the total period of endoscopic images 401 and the number of times it has been viewed (image 460 showing the number of views in Figure 20). In step S12, the control unit 321 adds to the surgical image 400 information indicating whether or not surgical instruments 121 to 124 were attached in chronological order (image 450 showing the usage record of surgical instruments in Figure 20), and information indicating the type of surgical instrument (surgical instrument information 451 in Figure 20 and image 437 in Figure 21).

[0165] Returning to Figure 28, in step S5, the control unit 321 provides the surgical image 400, to which additional information was added in step S4, to the observation terminal 340. The control unit 321 performs the processing in steps S1 to S4 for endoscopic images, operating room images, status logs, and operation logs for a predetermined period, and then performs the processing in step S5. The control unit 321 repeatedly performs the processing in steps S1 to S5 until it receives an instruction from the observation terminal 340 to terminate the display of the surgical image 400, and upon receiving the termination instruction, it terminates the processing shown in Figure 28.

[0166] The control unit 321 may perform the processing in steps S1 to S4 for the endoscopic images, operating room images, status logs, and operation logs for the entire period, and then perform the processing in step S5. In this case, repeating the processing in steps S1 to S5 is unnecessary.

[0167] In the process shown in Figure 28, the control unit 321, in the step of providing the surgical image 400 in step S5, provides the display unit 343 of the observation terminal 340 with an image (video) in which the endoscopic image 401, the operating room image 402, the schematic images 403 and 404, the video control area 420, each image in the information display area 430, and images 440, 450, and 460 change over time and are displayed in sync with each other.

[0168] Figure 30 is a flowchart showing the process related to the storage of screenshot 502 due to the addition of a bookmark.

[0169] In step S21, the control unit 321 of the information processing device 320 determines whether the bookmark insertion button 425 has been operated. If the bookmark insertion button 425 has been operated, in step S22, the control unit 321 stores the playback position of the endoscopic image 401 and a screenshot of the surgical image 400 at the time the bookmark insertion button 425 was operated. In step S23, the control unit 321 determines whether it has received an instruction from the observation terminal 340 to end the playback of the surgical image 400. If the control unit 321 has not received an instruction to end, it returns to step S21; if it has received an instruction to end, it terminates the process shown in Figure 30.

[0170] Figure 31 is a flowchart showing the process related to displaying screenshot 502 and playback before and after the bookmark position.

[0171] In step S31, the control unit 321 of the information processing device 320 determines whether a bookmark has been selected. Bookmark selection is performed by manipulating the bookmark item 501a in the editing image 500, manipulating the bookmark mark 426 in the surgical image 400, and manipulating the bookmark insertion button 425 in the surgical image 400.

[0172] If a bookmark is selected, in step S32, the control unit 321 provides the target screenshot 502 and the handwritten images 502a, 491 and text 504a associated with the screenshot 502 to the observation terminal 340. As a result, as shown in Figure 25, the edited image 500, including the screenshot 502, images 502a, 491 and text 504a, is displayed on the display unit 343 of the observation terminal 340.

[0173] In step S33, the control unit 321 determines whether the loop playback button 512 of the edited image 500 has been operated. If the loop playback button 512 has been operated, the edited image 500 is closed and the surgical image 400 is displayed. Then, in step S34, the control unit 321 sends information to the observation terminal 340 to repeatedly play back the selected bookmark position for a predetermined period before and after (for example, from -5 seconds to +5 seconds). That is, for example, the selected bookmark position ±5 seconds is loop-played on the observation terminal 340. If the loop playback button 512 has not been operated, the process proceeds to step S35.

[0174] In step S35, the control unit 321 determines whether the close button 511 of the edited image 500 has been operated. If the close button 511 has not been operated, the control unit 321 returns to step S33. If the close button 511 has been operated, the edited image 500 is closed and the surgical image 400 is displayed. Thus, the process shown in Figure 31 is completed.

[0175] <Effects of the Embodiment> The control unit 321 of the information processing device 320 acquires a state log (see Figure 13) indicating the state of the target being operated as the operator operates the operating unit 221 (see Figure 4), foot pedals 231-237 (see Figure 5), and viewer 211 (see Figure 3) (target being operated) of the operating device 2 (step S1 in Figure 27). Based on the state log, it generates images 432a, 432b, 434, 435 (see Figure 21) and image 440 (see Figure 20) (reconstructed image) that visualize the operator's operation on the target being operated (step S2). It then provides a surgical image 400 (see Figures 17, 18) that associates the endoscopic image 401, operating room image 402, and schematic images 403, 404 (see Figure 17) (operation images) showing the operation of the surgical robot 4 or surgical instruments 121-124 (see Figure 2) with the reconstructed image (step S5 in Figure 28).

[0176] By referring to the motion images and reconstructed images, the observer can easily and accurately confirm the operation of the surgical robot 4 or surgical instruments 121-124, and the operation of the target being manipulated.

[0177] The control unit 221 (operated object) is moved according to the operator's operation, and the status log includes information (position, amount of rotation, angle, presence or absence, etc.) based on the output of the encoder 261 and sensors (status detectors) that detect the status of the control unit 221 (operated object). With this configuration, by using the output of the encoder 261 and other sensors that detect the status of the operated object of the control device 2, images 432a and 432b (reconstructed images) can be generated that visualize how the operator actually moved the operated object.

[0178] The control unit 221 (operated object) is moved in response to the operator's operation, and the reconstructed image includes image 432a (see Figure 21) (schematic image) which schematically shows the degree of movement of the control unit 221 (operated object) due to the operator's operation. With this configuration, subtle adjustments made by the operator can be schematically visualized, and the observer can intuitively grasp how far the operator moved the control unit 221. As a result, even an inexperienced person can smoothly acquire the skills to operate an expert's controlled object by confirming the subtle operations of an expert using image 432a.

[0179] Image 432a (schematic image) is a schematic image of the object being operated on the plane by which the pair of movable plates 221b (objects being operated) of the operating unit 221 move. With this configuration, by referring to image 432a, the observer can accurately confirm the amount of movement of the pair of movable plates 221b on the plane of movement, and thus grasp the movement of the pair of movable plates 221b more accurately.

[0180] The reconstructed image includes image 432b (see Figure 21) which numerically shows the degree of movement of the operating unit 221 (operated object) due to the operator's actions. With this configuration, the extent to which the operator moved the operated object can be quantitatively shown, for example, by the distance or angle of movement, and the observer can quantitatively grasp the degree of movement of the operated object. As a result, the observer can learn how to operate the operated object efficiently and accurately, and it becomes easier to reproduce the techniques of an expert.

[0181] The controllable elements include foot pedals 234-237 (operators) that switch the power on and off to the forceps (electrical instruments) attached to the arm device 1. The status log includes information indicating the switching of power on and off by the foot pedals 234-237 (operators). The reconstructed image includes image 434 (see Figure 21) schematically showing the period during which the foot pedals 234-237 (operators) were operated. With this configuration, an observer using the observation terminal 340 can understand whether the operator sporadically powered the electrical instruments or continuously powered them over a certain period of time during the treatment of the affected area with the forceps (electrical instruments) shown in the endoscopic image 401. This allows the observer to learn what kind of power supply operations to perform on the electrical instruments when cutting or coagulating the affected area using electrical instruments such as electrosurgical units.

[0182] The operating object includes a viewer 211 (display unit) for the operator to observe the endoscopic image 401, the status log includes information based on the output of a head sensor 212 (sensor) that detects the operator's approach to the viewer 211, and the reconstructed image includes an image 440 (see Figure 20) that schematically shows the duration of the operator's approach to the viewer 211 and the interruption period. The duration of the operator's approach to the viewer 211 suggests that the operator is viewing the image on the viewer 211 and performing the procedure, while the interruption period suggests that the operator is not viewing the image on the viewer 211 and has interrupted the procedure. With this configuration, the observer can understand the interval of the procedure performed by the operator by referring to image 440. This allows the observer to learn what state the procedure shown in the endoscopic image 401 should be in when it is appropriate to interrupt the procedure and give instructions to the assistant, change forceps, take a rest, etc.

[0183] The reconstructed images include images 434 and 435 that schematically show the timing of operations on the foot pedals 231-237 (operated objects). The timing of operations is indicated by the timing when the signal value turns ON in image 434, and by the timing when the outer edges of the pedal images 471-477 (see Figure 23) are colored in image 435. With this configuration, the observer can easily understand at what timing the operator operated on the operated objects in the procedure shown in the endoscopic image 401.

[0184] Step S2 in Figure 28 includes the process of generating an image 435 (second reconstructed image) that schematically shows the state of readiness for operation based on the output from foot sensors 241-248 (sensors) that detect the state of readiness for operation with respect to foot pedals 231-237 (operated object). In the surgical image 400, the operation image and the second reconstructed image are associated. As shown in the lower part of Figure 22, if a foot image 481 is displayed and the outer edge of the pedal image corresponding to the foot image 481 is not colored, it can be seen that the foot is ready to operate with respect to the corresponding foot pedal. Through this process, the observer can easily understand at what timing the operator prepared to operate with respect to foot pedals 231-237 in the procedure shown in the endoscopic image 401.

[0185] Step S1 in Figure 27 includes the step of acquiring an operation log of the operating unit 140 of the arm device 1, which is driven by an operation on the operating unit 221 (the object being operated on), and Step S2 in Figure 28 includes the step of generating images 433a and 433b (third reconstructed images) that visualize the driving state of the operating unit 140 based on the operation log. In the surgical image 400, the endoscopic image 401, the operating room image 402, and the schematic images 403 and 404 (operation images) are associated with the third reconstructed images. This process allows the observer to compare the operation of the operating unit 221 by the operator with how the operating unit 140 of the arm device 1 is actually driven in response to this operation.

[0186] The operating device 2 includes an operating unit 221 (operated object) and movable parts 222-226 (second operated objects) that move in conjunction with the operator contacting and operating the operating unit 221 (operated object). The operation image includes a schematic image 404 that visualizes the overall movement of the operating unit 221 and movable parts 222-226 based on the state log. This process allows the observer to smoothly confirm how the hand controller 220 actually operated by the operator by referring to the schematic image 404 in conjunction with, for example, images 432a and 432b (reconstructed images).

[0187] The control unit 321 of the information processing device 320 adds image 460, which relates to the number of views shown in Figure 20, to the surgical image 400 as information indicating the portion of the operation images that have already been viewed out of the total operation images and the number of views (step S11 in Figure 29), and provides the surgical image 400 with image 460 added (step S5 in Figure 28). Through this process, the observer can grasp the steps of the surgery that have been frequently referenced so far, such as steps in which complex techniques or operations were performed. As a result, the observer can efficiently learn the techniques and operations on the object being operated on.

[0188] The object that the operator touches and manipulates is the operating part 221 for opening and closing the tip of the forceps attached to the arm device 1. With this configuration, the observer can grasp the manipulation of the object by the skilled operator's hand and easily reproduce the skilled operator's technique.

[0189] The control unit 321 of the information processing device 320 adds image 450, which is an image related to the usage record of surgical instruments (Figure 20), to the surgical image 400 as information indicating the presence or absence of the surgical instruments 121 to 124 in a time series, based on the output of sensors 151 to 154 that detect the attachment and detachment of surgical instruments 121 to 124 to the arm device 1 (step S12 in Figure 29), and provides the surgical image 400 with image 450 added (step S5 in Figure 28). This process allows the observer to understand the timing of attachment, detachment, and replacement of surgical instruments 121 to 124 to the arm device 1 in the procedure shown in the endoscopic image 401. This makes it easier for the observer to reproduce the techniques of a skilled surgeon using surgical instruments 121 to 124.

[0190] The control unit 321 of the information processing device 320 adds the surgical instrument information 451 from Figure 20 and the image 437 from Figure 21 to the surgical image 400 as information indicating the types of surgical instruments 121 to 124 attached to the arm device 1 (step S12 in Figure 29), and provides the surgical image 400 with the surgical instrument information 451 and image 437 added (step S5 in Figure 28). This process allows the observer to understand which surgical instruments were used according to the progress of the procedure. This makes it easier for the observer to reproduce the techniques of a skilled surgeon using the surgical instruments 121 to 124.

[0191] The objects that the operator touches and manipulates are the foot pedals 231-237 of the operating device 2. With this configuration, the observer can understand how the skilled operator manipulates the objects with their feet and easily reproduce the skilled operator's foot movements. For example, when switching the operation of the hand controller 220 to angle adjustment of the endoscope 122 using foot pedal 233, the observer can refer to image 435 (reconstructed image) to understand how frequently the angle of the endoscope 122 is being adjusted.

[0192] The foot pedals include foot pedals 234-237 for the right foot and foot pedals 231-233 for the left foot. The observer can observe the operation of foot pedals 231-237 by both the left and right feet.

[0193] The objects that the operator touches and manipulates include the operating section 221 and foot pedals 231-237 of the hand controller 220 for operating surgical instruments 121-124 attached to the arm device 1. The reconstructed image includes images 432a and 432b (first image) visualizing the operation of the operating section 221, and images 434 and 435 (second image) visualizing the operation of the foot pedals 231-237. In the surgical image 400, as shown in Figure 21, images 432a and 432b (first image) are positioned above image 435 (second image). With this configuration, the vertical position of the operator's hands and feet coincides with the vertical position of the objects manipulated in the surgical image 400, allowing the observer to learn the operator's operations more smoothly.

[0194] In the process of providing surgical images (step S5 in Figure 28), the control unit 321 of the information processing device 320 changes the endoscopic image 401, the operating room image 402, and schematic images 403, 404 (motion images) and images 432a, 432b, 434, 435 (reconstructed images) over time. Through this process, the observer can smoothly grasp the operation of the object being operated on as the surgery progresses by seeing the motion images and reconstructed images that change over time, i.e., by seeing the moving images.

[0195] In the process of providing surgical images 400 (step S5 in Figure 28), the control unit 321 of the information processing device 320 synchronizes the endoscopic image 401, the operating room image 402, and schematic images 403, 404 (action images) with images 432a, 432b, 434, 435 (reconstructed images). This process allows the observer to smoothly confirm the operation of the target of the procedure shown in the action image.

[0196] The control unit 321 of the information processing device 320 accepts the specification of an arbitrary point in time via the bookmark insertion button 425 while the surgical image 400 is being displayed as it changes over time (step S21 in Figure 30), and automatically saves a screenshot 502 (surgical image 400) of the specified point in time (step S22). Because the observer can save the surgical image 400 at any point in time while the surgical image 400 is being displayed as it changes over time, they can smoothly learn the operations during the procedure.

[0197] For example, if the operator's movements are too fast and it is not possible to smoothly compare the endoscopic image 401, the operating room image 402, and the schematic images 403 and 404 (movement images) with images 432a, 432b, 434, and 435 (reconstructed images), the observer can save a screenshot 502 corresponding to the surgical image 400 at that time, as shown in Figure 25. Furthermore, after the surgery, if the observer receives instruction from a skilled worker while referring to the surgical image 400, and the skilled worker makes annotations on the surgical image 400 as shown in Figures 25 and 26, the observer can save images 502a and 491 corresponding to the annotations, along with the screenshot 502 corresponding to the surgical image 400. This allows the observer to smoothly learn the procedures by referring to the saved screenshots 502 afterward.

[0198] The control unit 321 of the information processing device 320 accepts the designation of an arbitrary point in time via the bookmark insertion button 425 while the surgical image 400 changes over time is being displayed (step S21 in Figure 30), accepts a playback instruction for the designated point in time via the loop playback button 512 (step S33 in Figure 31), and plays back the surgical image 400 for a predetermined period before and after the designated point in time based on the playback instruction (step S34). With this process, the observer can designate an arbitrary point in time while the surgical image 400 changes over time is being displayed, and then confirm the motion image and reconstructed image around that point in time through playback. For example, if the operator's actions are fast and it is not possible to smoothly compare the motion image and the reconstructed image, the observer can designate that point in time and then confirm the motion image and reconstructed image around that point in time through playback. This allows the observer to smoothly learn the operations for the procedure.

[0199] As shown in Figure 17, in the surgical image 400, the endoscopic image 401 and schematic images 403 and 404 (operational images) are associated with the operating room image 402 obtained from the operating room camera 301 that photographs the operating room where the arm device 1 is installed. As shown in Figure 32, images 405 of the operating device 2, 406 of the patient's abdomen, 407 of the equipment room, and 408 of the corridor leading to the operating room may also be photographed by the camera, and images 405 to 408 may be displayed in the surgical image 400 together with the operational images. In this case, images 405 to 408 are videos that were photographed simultaneously with the operational images.

[0200] As described above, in the surgical image 400, the endoscopic image 401 and schematic images 403, 404 (motion images) are associated with images obtained from a camera that photographs at least one of the operating room where the arm device 1 is installed, the operating device 2, the patient's abdomen, the equipment room, and the corridor leading to the operating room. This allows the observer to confirm the status of the operating room, the operating device 2, the patient's abdomen, the equipment room, and the corridor in accordance with the progress of the procedure shown in the motion images. For example, the observer can confirm the status of assistance and movements from the assistant in the operating room, the angles of each joint of the hand controller 220 of the operating device 2, the angle of the guide tube (trocar) in the patient's abdomen, and the entry, exit, and movement of the assistant in the equipment room or corridor in accordance with the progress of the procedure. This allows the observer to further learn about their own and the team's actions during the procedure.

[0201] The information processing system 5 shown in Figure 8 comprises a storage device 310 and an information processing device 320. The storage device 310 stores a state log indicating the state of the target of operation as it changes when the operator operates the target of operation of the operating device 2, which controls the movement of the arm device 1 of the surgical robot 4 to which surgical instruments 121-124 are attached. The target of operation is, for example, the operating unit 221 (see Figure 4), the foot pedals 231-237 (see Figure 5), and the viewer 211 (see Figure 3). Based on the state log stored in the storage device 310, the information processing device 320 generates images 432a, 432b, 434, 435, and 440 (reconstructed images) that visualize the operator's operation on the target of operation, and provides a surgical image 400 that associates the reconstructed images with endoscopic images 401, operating room images 402, and schematic images 403, 404 (movement images) that show the movement of the surgical robot 4 or surgical instruments 121-124.

[0202] With this configuration, the observer can easily and accurately confirm the operation of the surgical robot 4 or surgical instruments 121-124, and the operation of the target being manipulated, by referring to the motion image and the reconstructed image.

[0203] <Example of changes> In the above embodiment, image 432a (see Figure 21), which shows the degree of movement of the operating unit 221 due to the operator's operation, shows a schematic diagram of a pair of movable plates 221b, and the degree to which the pair of movable plates 221b in this schematic diagram are drawn according to the operation of the operating unit 221. However, the embodiment is not limited to this, and instead of a schematic diagram of the pair of movable plates 221b, a meter that schematically shows the degree to which the pair of movable plates 221b are opened may be displayed in image 432a.

[0204] Image 432a was a two-dimensional and schematic representation of the pair of movable plates 221b (operated objects) of the operating unit 221, but instead, an image showing the movable plates 221b (operated objects) in three dimensions and schematic may be used.

[0205] In the above embodiment, the image 434 shown in Figure 21 indicates the ON or OFF state of the foot pedals corresponding to the left and right hand controllers 220. However, it is not limited to this, and an output value indicating the strength of the energy applied to the corresponding electrosurgical unit when the foot pedal is ON may also be displayed. In this case, the strength of the energy applied to the electrosurgical unit when the foot pedal is ON may be changed during surgery by setting the energy generator. For example, if the strength of the energy applied to the electrosurgical unit when the foot pedal is ON is set to V1, the height of the graph in image 434 will be V1 while the foot pedal is pressed. If the strength of the energy applied to the electrosurgical unit when the foot pedal is ON is set to V2 (>V1), the height of the graph in image 434 will be V2 while the foot pedal is pressed, and the height of the graph in image 434 will be 0 when the foot pedal is not pressed.

[0206] In the above embodiment, the image 434 shown in Figure 21 shows the pressing state of the foot pedals 234 to 237 based on the state log. In addition, the ON or OFF state of the voltage actually applied to the forceps may also be shown based on the operation log.

[0207] In the above embodiment, the endoscopic image 401, the operating room image 402, and the schematic images 403 and 404 are images that change over time and continuously, i.e., moving images, and the display of other parts of the surgical image 400 also changes over time and continuously. However, the embodiment is not limited to this, and each part of the surgical image 400 may be a series of still images (frame-by-frame video) that change at predetermined time intervals (for example, intervals of a few seconds). Also, when repeated playback is performed by the loop playback button 512 in Figure 25, each part of the surgical image 400 may be a frame-by-frame video.

[0208] In the above embodiment, the endoscopic image 401, the operating room image 402, and the schematic images 403, 404 (action images) and the images 432a, 432b, 434, 435 (reconstructed images) were synchronized with each other. However, they do not need to be synchronized with each other as long as the action images and reconstructed images are associated with each other. For example, the reconstructed images may be displayed with a slight delay compared to the action images. That is, the time at which the reconstructed images are displayed may be later than the time at which the action images are displayed. The delay time of the reconstructed images may be adjusted to make it easier for the observer to learn. This allows the observer to move their gaze to the reconstructed image after viewing the action image and confirm the operation they just saw in the action image using the reconstructed image.

[0209] Similarly, in the modified example of the above embodiment, images 405-408 (see Figure 32) were synchronized with the motion image and the reconstructed image, but they do not need to be synchronized with the motion image and / or the reconstructed image. Also, images 405-408 do not need to be a video of the entire duration of the motion image and / or the reconstructed image; only a predetermined period may be displayed.

[0210] In the above embodiment, the screenshot 502 corresponding to the surgical image 400 was saved in the storage unit 322 of the information processing device 320 by the bookmark insertion button 425. However, it is not limited to this, and may be saved in the storage unit 342 of the observation terminal 340.

[0211] In the above embodiment, the operating room images, endoscopic images, operation logs, and status logs stored in the storage device 310 were transmitted to the information processing device 320 in real time during the surgery, but they may also be transmitted after the surgery.

[0212] In the above embodiment, the surgical image 400 included images 401-404, images 432a-435, etc. However, the surgical image 400 only needs to include at least one of images 401-404 (action images) and at least one of images 432a, 432b, 434, 435, 440 (reconstructed images visualizing the operator's actions). In this case as well, the observer can easily and accurately confirm the actions of the surgical robot 4 or surgical instruments 121-124 and the actions of the objects being manipulated by referring to the action images and the reconstructed images.

[0213] In the above embodiment, the state log and operation log consist of current values ​​acquired at predetermined time intervals, but are not limited thereto. For example, the control unit 131 or control unit 251 may acquire the value or signal as a current value each time the encoder or sensor outputs a value or signal, and at least one of the state log and operation log may consist of those current values.

[0214] In the above embodiment, the information display area 430 of the surgical image 400 displayed an operation information tab and a settings information tab, but an error information tab may also be displayed.

[0215] Figure 33 shows surgical image 400 when the error information tab is selected.

[0216] As shown in Figure 33, when the Error Information tab is selected, error information 600 is displayed according to the error occurrence history. When any of the error information 600 is selected (in the example in Figure 33, the fourth (bottom row) error information 600 is selected), detailed information 600a is displayed, including the error message and the date and time the error occurred. In addition, a warning mark 424 corresponding to the selected error information 600 is highlighted in the video control area 420. Furthermore, by moving the playback position mark 422 to the position of the warning mark 424, a still image 403a of the device that was affected by the error (in this example, arm device 1) at the time the error occurred is displayed. The still image 403a is an image of the schematic image 403 (video) at the time the error occurred. This allows the observer to easily understand when and what kind of error occurred, and what the operating state of the device that was affected by the error was at the time the error occurred.

[0217] Furthermore, in Figure 33, the observer can move the playback position mark 422 to the position of the warning mark 424, and then operate the operation information tab to display the same information as in Figure 21 in the information display area 430. This allows the observer to view the still image 403a at the time the error occurred while referring to a reconstructed image that visualizes the operator's actions at that time, thus enabling them to understand the operator's actions in detail at the time the error occurred.

[0218] Embodiments of the present invention can be modified in various ways as appropriate within the scope of the technical idea set forth in the claims. [Explanation of symbols]

[0219] 1. Arm device 2 Control device 4. Surgical robots 5. Information Processing Systems 121 Surgical instruments (forceps) 122 Surgical instruments 123 Surgical instruments (forceps, electrical instruments) 124 Surgical instruments (forceps) 140 Operating part 151-154 Sensors 211 Viewer (operation target, display unit) 212 Head Sensor (Sensor) 220 Hand Controller 221 Operating section (object of operation) 221b Movable plate (object to be operated) 222-226 Movable parts (second target of operation) 231-233 Foot pedals (operated by) 234-237 Foot pedal (operated object, control element) 261 Encoder (State Detector) 301 Operating Room Camera (Camera) 310 Storage device 320 Information Processing Devices 400 surgical images 401 Endoscopic image (motion image) 402 Operating room images (motion images, still images) 403, 404 Schematic Images (Operational Images) Images 405-408 432a Image (reconstructed image, schematic image, first image) 432b Image (reconstructed image, first image) Images 433a and 433b (third reconstructed image) 434 images (reconstructed images, second image) 435 images (reconstructed image, second reconstructed image, second image) 440 images (reconstructed images) 450 images (information) 460 images (information) 502 Screenshots (Surgical Images)

Claims

1. A surgical robot system comprising a surgical robot configured to perform surgical procedures on a patient using surgical instruments, and a computer, The aforementioned surgical robot is An arm device comprising multiple arms, wherein multiple types of surgical instruments are detachably attached to the arms, An endoscope configured to capture images showing the surgical site and the tip of the surgical instrument, An operating device including a hand controller configured to remotely control the arm, wherein the hand controller has a pair of movable members used to open and close the tip, The system includes a sensor configured to generate a signal in response to the operation of the tip, The aforementioned computer, The following are obtained: a plurality of images captured by the endoscope during the surgical procedure, and a plurality of signals generated by the sensor during the surgical procedure. A graphical user interface related to the aforementioned surgical procedure is displayed on the display unit. The aforementioned graphical user interface is A video generated from the aforementioned multiple images, showing the manipulation of the tip during the surgical procedure, Multiple timeline regions corresponding to each of the aforementioned multiple arms, A numerical value, generated based on at least one of the plurality of signals, and corresponding to the force applied to the patient's tissue by the tip, is displayed. Each of the aforementioned multiple timeline regions extends linearly in accordance with the timeline of the surgical procedure and visually distinguishes between a first period in which the first type of surgical instrument is attached to the arm and a second period in which the second type of surgical instrument is attached to the arm. The computer displays the numerical values ​​in the graphical user interface in association with the operation of the tip shown in the video. Surgical robotic system.

2. The computer is configured to display the graphical user interface such that the numerical value changes in response to the operation of the tip shown in the video. The surgical robot system according to claim 1.

3. The computer is configured to display the graphical user interface such that the operation of the tip shown in the video and the numerical values ​​change in response to user operations on the video. The surgical robot system according to claim 1.

4. The computer is configured to display a plurality of numerical values ​​corresponding to each of the plurality of surgical instruments on its graphical user interface. The surgical robot system according to claim 1.

5. The aforementioned video shows the operation of the multiple surgical instruments during the surgical procedure. The computer is configured to display the graphical user interface such that the multiple numerical values ​​change in response to the operation of the multiple surgical instruments shown in the video. The surgical robot system according to claim 4.

6. The computer is configured to display the numerical values ​​in the graphical user interface in association with the type of surgical instrument. The surgical robot system according to claim 1.

7. The computer is configured to display the graphical user interface so that the operations of the tip during the first period and the operations of the tip during the second period can be distinguished. The surgical robot system according to claim 1.

8. The computer is configured to display the graphical user interface such that at least one of the plurality of timeline regions corresponds to the endoscope. The surgical robot system according to claim 1.

9. A computer for a surgical robot comprising a plurality of arms and a hand controller having a pair of movable members for opening and closing the tips of surgical instruments attached to the arms, Memory to store the program, A processor that executes the aforementioned program, The aforementioned surgical robot is An endoscope configured to capture images showing the surgical site and the tip, The system includes a sensor configured to generate a signal in response to the operation of the tip, By executing the program, the processor, Means for acquiring a plurality of images captured by the endoscope during a surgical procedure performed on a patient using the surgical robot, means for acquiring a plurality of signals generated by the sensor during the surgical procedure, and It functions as a means for displaying a graphical user interface related to the surgical procedure on the display unit, The aforementioned graphical user interface is A video generated from the aforementioned multiple images, showing the manipulation of the tip during the surgical procedure, Multiple timeline regions corresponding to each of the aforementioned multiple arms, A numerical value, generated based on at least one of the plurality of signals, and corresponding to the force applied to the patient's tissue by the tip, is displayed. Each of the aforementioned multiple timeline regions extends linearly in accordance with the timeline of the surgical procedure and visually distinguishes between a first period in which the first type of surgical instrument is attached to the arm and a second period in which the second type of surgical instrument is attached to the arm. The processor displays the numerical values ​​in the graphical user interface in association with the operation of the tip shown in the video. computer.

10. The processor displays the graphical user interface such that the numerical value changes in response to the operation of the tip shown in the video. The computer according to claim 9.

11. The processor displays the graphical user interface such that the operation of the tip shown in the video and the numerical values ​​change in response to user operations on the video. The computer according to claim 9.

12. The processor displays a plurality of numerical values ​​corresponding to each of the plurality of surgical instruments on the graphical user interface. The computer according to claim 9.

13. A method for providing surgical information for a surgical robot comprising a plurality of arms and a hand controller having a pair of movable members for opening and closing the tips of surgical instruments attached to the arms, The aforementioned surgical robot is An endoscope configured to capture images showing the surgical site and the tip, The system includes a sensor configured to generate a signal in response to the operation of the tip, A step of acquiring a plurality of images captured by the endoscope during a surgical procedure performed on a patient using the surgical robot, The process of acquiring a plurality of signals generated by the sensor during the surgical procedure, The process includes the step of displaying a graphical user interface related to the surgical procedure on a display unit, The aforementioned graphical user interface is A video generated from the aforementioned multiple images, showing the manipulation of the tip during the surgical procedure, Multiple timeline regions corresponding to each of the aforementioned multiple arms, A numerical value, generated based on at least one of the plurality of signals, and corresponding to the force applied to the patient's tissue by the tip, is displayed. Each of the aforementioned multiple timeline regions extends linearly in accordance with the timeline of the surgical procedure and visually distinguishes between a first period in which the first type of surgical instrument is attached to the arm and a second period in which the second type of surgical instrument is attached to the arm. The graphical user interface displays the numerical values ​​in association with the operation of the tip shown in the video. method.

14. The graphical user interface is displayed such that the numerical value changes in response to the operation of the tip shown in the video. The method according to claim 13.

15. The graphical user interface is displayed such that the operation of the tip shown in the video and the numerical values ​​change in response to user operations on the video. The method according to claim 13.

Images