Method for providing surgical images using a surgical robot and information processing system

Abstract

To provide a method of providing an operation image using a surgical robot which can easily and correctly confirm a movement of the surgical robot or a surgical instrument and a manipulation of a manipulation target therefor, and provide an information processing system.SOLUTION: A method of providing an operation image using a surgical robot acquires a state log indicating the state of a manipulation target that changes by manipulating the manipulation target of a manipulation device 2 by an operator, generates a reconstruction image that visualizes the manipulation of the operator to the manipulation target on the basis of the state log, and provides an operation image indicating the movement of the surgical robot or surgical instrument is associated with the reconstruction image.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present invention relates to a method for providing surgical images using a surgical robot and an information processing system. [Background technology]

[0002] Conventionally, surgical operations have been performed using surgical robots. Surgical robots are equipped with a console (operation device) and a surgical robot arm (arm device). Surgical instruments are attached to the surgical robot arm, and the surgical robot arm and surgical instruments are operated by an operator operating the console. It is desirable for any operator to be able to improve their proficiency in operating the console so that patients can undergo surgery with peace of mind.

[0003] The following Patent Document 1 describes a system that uses a depth camera to capture and store images of a surgical operation performed in an operating room using a surgical robot, and then enables playback of a simplified image of the captured image after the operation. The system described in Patent Document 1 also stores history data, such as the state of the surgical robot arm and the state of the console's handheld input device (handheld UID). The stored system data is used to reconstruct the position, orientation, and movement of the robot arm and surgical instruments, and to generate a simplified image of the captured image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2021 / 247050 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the system in Patent Document 1 only allows the operator to view a simplified image of the surgical operation performed by the surgical robot, making it difficult to see how the operator operated the console during the operation.

[0006] In view of these problems, the present invention aims to provide a method for providing surgical images using a surgical robot and an information processing system that enable easy and accurate confirmation of the operation of the surgical robot or surgical instrument and the operation of the target to be operated. [Means for solving the problem]

[0007] The present invention provides a method for generating an image (400) of a surgical operation performed by a surgical robot (4) that includes an arm device (1) to which surgical instruments (121 to 124) are attached and an operating device (2) that an operator operates to drive the arm device (1). The information processing device (320) generates A method for providing a surgical image (400) using a surgical robot (4), comprising: acquiring (S1) a status log indicating the status of an operation object (221, 231 to 237, 211) that changes as an operator operates the operation object (221, 231 to 237, 211) of an operation device (2); and, based on the status log, Changes over time Generate reconstructed images (432a, 432b, 434, 435, 440) visualizing the above (S2), Based on the operator's operation on the operation object (221, 231 to 237, 211) A surgical image (400) is provided (S5) in which motion images (401 to 404) showing the motion of the surgical robot (4) or surgical instruments (121 to 124) are associated with reconstructed images (432a, 432b, 434, 435, 440).

[0008] According to the method of providing surgical images using a surgical robot of the present invention, an observer can easily and accurately confirm the operation of the surgical robot or surgical instrument and the operation of the target to be operated thereby by referring to the operation image and the reconstructed image.

[0009] The information processing system (5) of the present invention includes a storage device (310) that stores a status log indicating the status of the operation targets (221, 231 to 237, 211) that change as the operator operates the operation targets (221, 231 to 237, 211) of an operation device (2) that the operator operates to drive an arm device (1) of a surgical robot (4) to which surgical instruments (121 to 124) are attached, and a storage device (310) that stores a status log indicating the status of the operation targets (221, 231 to 237, 211) that changes as the operator operates the operation targets (221, 231 to 237, 211) based on the status log stored in the storage device (310). Changes over time Generate reconstructed images (432a, 432b, 434, 435, 440) visualizing the Based on the operator's operation on the operation target The surgical system includes an information processing device (320) that provides a surgical image (400) that associates motion images (401-404) showing the motion of a surgical robot (4) or a surgical instrument (121-124) with reconstructed images (432a, 432b, 434, 435, 440).

[0010] According to the information processing system of the present invention, an observer can easily and accurately confirm the operation of a surgical robot or surgical instrument and the operation of an object to be operated thereby by referring to the operation image and the reconstructed image. [Effects of the Invention]

[0011] According to the present invention, the operation of a surgical robot or surgical instrument and the operation of an object to be operated can be easily and accurately confirmed. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram showing the configuration of a surgical robot according to an embodiment. [Figure 2] FIG. 2 is a perspective view showing the appearance of the arm device according to the embodiment. [Figure 3] FIG. 3 is a perspective view showing the appearance of the operating device according to the embodiment. [Figure 4] FIG. 4 is a perspective view showing the appearance of a hand controller according to the embodiment. [Figure 5]FIG. 5 is a perspective view showing the appearance of the foot unit according to the embodiment. [Figure 6] FIG. 6 is a perspective view showing a state in which an operator is using the operating device according to the embodiment. [Figure 7] FIG. 7 is a block diagram showing the configuration of the control device according to the embodiment. [Figure 8] FIG. 8 is a block diagram showing the configuration of an information processing system according to the embodiment. [Figure 9] FIG. 9 is a diagram showing an outline of connections between a surgical robot, an information processing system, etc., and information transmitted and received between the respective devices according to the embodiment. [Figure 10] FIG. 10 is a block diagram showing a usage pattern of an information processing device according to an embodiment. [Figure 11] FIG. 11 is a block diagram showing the configuration of an arm device according to an embodiment. [Figure 12] FIG. 12 is a block diagram showing the configuration of the operating device according to the embodiment. [Figure 13] FIG. 13 is a diagram illustrating an example of current values ​​that constitute a status log transmitted from the operation device to the control device according to the embodiment. [Figure 14] FIG. 14 is a diagram illustrating an example of a status log according to the embodiment. [Figure 15] FIG. 15 is a diagram illustrating an example of current values ​​constituting the operation log transmitted from the arm device to the control device according to the embodiment. [Figure 16] FIG. 16 is a diagram illustrating an example of an operation log according to the embodiment. [Figure 17] FIG. 17 is a diagram schematically illustrating the configuration of a surgical image according to an embodiment. [Figure 18] FIG. 18 is a diagram schematically illustrating the configuration of a surgical image according to an embodiment. [Figure 19] FIG. 19 is an enlarged view schematically showing the configuration of an endoscopic image according to an embodiment. [Figure 20] FIG. 20 is a diagram showing a schematic configuration of an image or the like showing a usage record of a surgical instrument according to an embodiment. [Figure 21] FIG. 21 is an enlarged view schematically showing the configuration of the information display area according to the embodiment. [Figure 22] FIG. 22 is a diagram illustrating the configuration and transition of an image relating to the state of a foot pedal according to an embodiment. [Figure 23] FIG. 23 is a diagram illustrating the configuration and transition of an image relating to the state of a foot pedal according to an embodiment. [Figure 24] FIG. 24 is an enlarged view schematically showing the configuration of the information display area according to the embodiment. [Figure 25] FIG. 25 is a diagram illustrating a configuration of an edited image that is displayed when a bookmark insertion button is operated on a surgical image according to an embodiment. [Figure 26] FIG. 26 is a diagram illustrating a configuration of a surgical image in the case where a hand-drawn image is inserted first according to an embodiment. [Figure 27] FIG. 27 is a flowchart illustrating a process for receiving information necessary for generating a surgical image according to an embodiment. [Figure 28] FIG. 28 is a flowchart illustrating a process for generating a surgical image according to an embodiment. [Figure 29] FIG. 29 is a flowchart showing details of the additional information adding process according to the embodiment. [Figure 30] FIG. 30 is a flowchart illustrating a process for storing a screenshot by adding a bookmark, according to an embodiment. [Figure 31] FIG. 31 is a flowchart showing a process related to displaying a screenshot and playing back content before and after a bookmarked position according to an embodiment. [Figure 32] FIG. 32 is a diagram showing a schematic diagram of an image of an operating device, an image of a patient's abdomen, an image of an equipment storage room, and an image of a corridor leading to an operating room, in accordance with a modified example of an embodiment. [Figure 33] FIG. 33 is a diagram schematically showing the configuration of a surgical image according to a modified example of the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] FIG. 1 is a block diagram showing the configuration of a surgical robot 4.

[0014] The surgical robot 4 is a device used in endoscopic surgery. The surgical robot 4 includes an arm device 1, an operating device 2, and a control device 3 that controls the arm device 1 and the operating device 2. The control device 3 is built into the arm device 1. The arm device 1 and the operating device 2 that incorporate the control device 3 may be installed in the same facility, or may be installed in different facilities via a network.

[0015] FIG. 2 is a perspective view showing the appearance of the arm device 1. As shown in FIG.

[0016] The arm device 1 is a patient-side device equipped with multiple arms to which an endoscope and forceps can be attached. The arm device 1 operates in accordance with drive instructions transmitted from the control device 3 when the operating device 2 is operated by a doctor or other operator.

[0017] The arm device 1 includes a base 101, an operating unit 102, a base arm 103, a support unit 104, and arms 111 to 114. Surgical instruments 121 to 124 are detachably attached to the arms 111 to 114, respectively.

[0018] The operation unit 102 is installed on the base 101 and includes a display unit and an input unit. An operator operates the input unit of the operation unit 102 while looking at the display unit of the operation unit 102 to set the movement speed and the like of the arms 111 to 114 of the arm device 1. The base arm 103 is an arm with a plurality of joints and has its rear end installed on the base 101. The upper surface of the support unit 104 is rotatably connected to the tip of the base arm 103, and moves in conjunction with the movement of the tip of the base arm 103, and rotates around the axis of the tip of the base arm 103.

[0019] The upper ends of the arms 111 to 114 are attached to the underside of the support part 104. Each of the arms 111 to 114 is equipped with 12 axes and joints between the axes. The lower ends of the arms 111 to 114 are provided with supports configured to allow surgical instruments to be attached and detached, and the surgical instruments 121 to 124 are attached to the supports.

[0020] The surgical instruments 121-124 have elongated shafts 121a-124a, and during endoscopic surgery, the shafts 121a-124a of the surgical instruments 121-124 are inserted into the patient's body via a guide tube (trocar) inserted into the patient's abdomen. In this embodiment, the surgical instruments 121, 123, and 124 are forceps, and the surgical instruments (forceps) 121 and 124 are so-called grasping forceps. The surgical instrument (forceps) 123 is a so-called electric scalpel, and is equipped with a hand unit at its tip that is capable of incising and coagulating. The surgical instrument 122 is an endoscope. The surgical instrument (endoscope) 122 is, for example, a 3D videoscope. Hereinafter, the surgical instrument 122 will also be referred to as an "endoscope 122."

[0021] The forceps are not limited to grasping forceps, but may be other types of forceps such as hemostatic forceps or dissecting and ligating forceps, or may be an electric instrument such as an electric scalpel. The surgical instruments 121 to 124 may be replaced with different types of surgical instruments as needed during the endoscopic surgery.

[0022] FIG. 3 is a perspective view showing the appearance of the operating device 2. As shown in FIG.

[0023] The operating device 2 is an operator-side device that is operated by an operator who is a doctor during endoscopic surgery to drive the arm device 1.

[0024] The operation device 2 includes a base 201 , a support portion 202 , a frame member 203 , an operation panel 204 , a viewer unit 210 , two hand controllers 220 , and a foot unit 230 .

[0025] The support part 202 is installed on the top of the base 201 so as to be movable up and down. The frame member 203 is installed on the support part 202. The operation panel 204 is installed on the upper surface of the front center of the frame member 203. On the left and right sides of the operation panel 204, armrests 203a are formed for the operator to rest their elbows on when operating the hand controller 220. The height of the armrests 203a can be changed by moving the support part 202 up and down relative to the base 201.

[0026] The viewer unit 210 is supported at the tip of an arm 213 installed on the upper surface of the support part 202, and the height can be changed by rotating the joint of the arm 213. The viewer unit 210 includes a display part (hereinafter referred to as a viewer) 211 and a head sensor 212. The viewer 211 displays an image (endoscopic image) captured by the endoscope 122. The head sensor 212 is a transmission-type photoelectric sensor that is provided to sandwich the area in front of the viewer 211. When the operator looks into the viewer 211, the operator's head approaches the viewer 211, light is blocked from the head sensor 212, and a state in which the operator is looking at the viewer 211 is detected.

[0027] The two hand controllers 220 each have seven axes and joints between the axes, and are installed on the support part 202. The left and right hand controllers 220 are input devices that allow an operator to operate the arms 111 to 114 of the arm device 1 using their left and right hands, respectively. The configuration of the hand controllers 220 will be described later with reference to FIG. 4.

[0028] The foot unit 230 is installed on the lower front side of the base 201 so as to be movable back and forth. The foot unit 230 is an input device that allows the operator to operate the arm device 1 using his left and right feet. The configuration of the foot unit 230 will be described later with reference to FIG. 5. By operating the operation panel 204, the operator can change settings of the operation device 2, such as the height of the armrest 203a, the height of the viewer unit 210, and the position of the foot unit 230 in the front-rear direction.

[0029] FIG. 4 is a perspective view showing the appearance of the hand controller 220.

[0030] The two hand controllers 220 are configured symmetrically in the left-right direction. The hand controller 220 includes an operation unit 221 and six movable units 222-226.

[0031] The operation unit 221 includes a support shaft 221a, a pair of movable plates 221b, and a pair of hook-and-loop fasteners 221c. The operation unit 221 is a part that is directly operated by an operator using his or her fingers, and is an operation object that is operated by the operator by touching it.

[0032] The support shaft 221a is a cylindrical member installed on the movable part 222. The pair of movable plates 221b are plate-shaped members arranged to sandwich the support shaft 221a. The movable plates 221b are installed on the support shaft 221a so that the end closer to the movable part 222 moves towards and away from the support shaft 221a. The hook-and-loop fastener 221c is installed on the movable plate 221b. The operation part 221 includes an encoder 261 (see FIG. 12) for detecting the opening / closing angle of the pair of movable plates 221b, and an encoder 271 (see FIG. 12) for detecting the amount of rotation of the operation part 221 relative to the movable part 222.

[0033] The movable parts 222 to 226 form an arm that connects the operation part 221 and the support part 202. The movable parts 222 to 226 are parts that the operator does not touch and operate, but rather operate indirectly via the operation part 221. Each joint of the arm formed by the movable parts 222 to 226 is formed by the rotation of two adjacent movable parts about an axis. The movable parts 222 to 226 are each equipped with an encoder 272 to 276 (see FIG. 12) for detecting the amount of rotation of the movable part relative to the adjacent movable part on the opposite side from the operation part 221. That is, the encoders 271 to 276 are provided one at each end of the seven shafts of the hand controller 220.

[0034] It should be noted that an encoder does not necessarily have to be used to detect the amount of rotation of the movable part; various state detectors such as encoders or sensors that detect the position, amount of rotation, angle, presence or absence, etc. of the object to be detected can be used.

[0035] Before starting surgery, the operator presses his / her thumb and index finger against the pair of movable plates 221b so as to sandwich the pair of movable plates 221b. In this state, the hook-and-loop fastener 221c is fastened to the thumb and index finger sandwiching the pair of movable plates 221b. As a result, the thumb and index finger of the left hand are fixed to the operation unit 221 of the left hand controller 220, and the thumb and index finger of the right hand are fixed to the operation unit 221 of the right hand controller 220. Note that the fingers sandwiching the pair of movable plates 221b are not limited to the thumb and index finger, and may be selected so that the operator can easily operate them. For example, the fingers sandwiching the pair of movable plates 221b may be the thumb and middle finger.

[0036] When the operator operates to move the operation unit 221 during surgery, each joint of the hand controller 220 rotates about its axis in accordance with the movement of the operation unit 221. Then, based on the output values ​​of the encoders 271 to 276, the target arm of the arm device 1 is driven, and the surgical instrument attached to the arm moves. Furthermore, if forceps are attached to the target arm, when the operator operates to open and close the pair of movable plates 221b of the operation unit 221 during surgery, the tip of the forceps attached to the arm is driven to open and close based on the output value of the encoder 261 of the operation unit 221.

[0037] FIG. 5 is a perspective view showing the appearance of the foot unit 230. As shown in FIG.

[0038] The foot unit 230 includes foot pedals 231-237 and foot sensors 241-248.

[0039] The operator steps on the foot pedals 231 to 237 to input for operating the arm device 1. That is, the foot pedals 231 to 237 are controls for performing predetermined inputs. When an electric scalpel is attached to the target arm, the foot pedals 234 to 237 are controls for switching the electric scalpel between energized and de-energized.

[0040] By stepping on the foot pedal 231, the surgical instrument operated by the right hand controller 220 is switched to either the surgical instrument 123 or 124. The foot pedal 232 is a clutch pedal. While the foot pedal 232 is being stepped on, the operation of the hand controller 220 is not transmitted to the arm device 1. The foot pedal 233 is a camera pedal. While the foot pedal 233 is being stepped on, both hand controllers 220 can operate the arm to which the endoscope 122 is attached.

[0041] Foot pedals 234 and 236 are incision pedals. While foot pedals 234 and 236 are depressed, high frequency is applied to the tip of the forceps (electric scalpel) so that incision can be performed by the forceps (electric scalpel). Foot pedals 235 and 237 are coagulation pedals. While foot pedals 235 and 237 are depressed, high frequency is applied to the tip of the forceps (electric scalpel) so that coagulation can be performed by the forceps (electric scalpel).

[0042] The foot pedals 231 to 237 are each provided with limit sensors 281 to 287 (see FIG. 12) for detecting whether or not the foot pedal is being stepped on. This allows detection of whether or not the foot pedals 231 to 237 are being stepped on.

[0043] Foot sensors 241-248 are reflective photoelectric sensors for detecting the positions of the operator's feet. Foot sensors 241-244 detect whether the left foot is positioned on foot pedals 231-233, and foot sensors 245-248 detect whether the right foot is positioned on foot pedals 234-237. In detail, foot sensors 241-248 detect hover states, which include states in which the feet are in front of the foot pedals, above the foot pedals, and while the foot pedals are being stepped on, as well as non-hover states in which the feet are not within foot unit 230.

[0044] FIG. 6 is a perspective view showing a state in which an operator is using the operation device 2. As shown in FIG.

[0045] Before the start of surgery, a patient is positioned lying on the operating table below the surgical instruments 121 to 124, a guide tube (trocar) is inserted into the patient's abdomen, and the surgical instruments 121 to 124 are inserted into the patient's body through the guide tube (trocar). The operator sits in a chair, places both arms on the armrests 203a, and places both feet inside or in front of the foot unit 230. The operator fixes their thumb and index finger to the movable plate 221b using hook-and-loop fasteners 221c (see FIG. 4). The operator then places their head inside the viewer unit 210 and looks into the viewer 211. When the head sensor 212 detects that the operator's head is positioned inside the viewer unit 210, the arm device 1 can be operated by the operating device 2.

[0046] When surgery begins, the operator operates the left and right operation units 221 (see FIG. 4) to input a gripping operation to cause the forceps to perform a gripping operation, or a movement operation to move the tip of the forceps. In response to the operation input by the operator to the operation device 2, the arms 111-114 of the arm device 1 and the surgical instruments 121-124 are driven. In this manner, various surgeries are performed.

[0047] The operator removes his head from the viewer unit 210 when giving instructions to assistants in the operating room, when taking a break, when finishing surgery, etc. When the head sensor 212 detects that the operator's head is not positioned within the viewer unit 210, the arm device 1 cannot be operated by the operating device 2. This prevents the arm device 1 from malfunctioning.

[0048] FIG. 7 is a block diagram showing the configuration of the control device 3.

[0049] The control device 3 includes a control unit 31, a storage unit 32, and a communication unit 33. The control unit 31 is configured with, for example, a CPU. The control unit 31 controls each hardware unit of the control device 3 and executes various processes by executing a computer program stored in the storage unit 32. The storage unit 32 is configured with, for example, an SSD or HDD. The communication unit 33 includes a communication interface that can communicate with the arm device 1, the operation device 2, and a storage device 310 (see FIG. 8) based on a predetermined communication standard.

[0050] FIG. 8 is a block diagram showing the configuration of the information processing system 5. As shown in FIG.

[0051] The information processing system 5 includes a storage device 310 and an information processing device 320. The network 330 is, for example, the Internet.

[0052] The storage device 310 includes a control unit 311, a storage unit 312, and a communication unit 313. The control unit 311 is configured with, for example, a CPU. The control unit 311 executes computer programs stored in the storage unit 312 to control each hardware unit of the storage device 310 and perform various processes. The storage unit 312 is configured with, for example, an SSD or HDD. The storage device 310 is communicably connected to the information processing device 320 via a network 330. The communication unit 313 includes a communication interface that is capable of communicating with the information processing device 320, the control device 3, the video processing device 302 (described later), and the operating room camera 301 (see FIG. 9 ) based on a predetermined communication standard, for example, Ethernet or Wi-Fi.

[0053] The information processing device 320 includes a control unit 321, a storage unit 322, and a communication unit 323. The control unit 321 is configured with, for example, a CPU. The control unit 321 controls each hardware unit of the information processing device 320 and executes various processes by executing a computer program stored in the storage unit 322. The storage unit 322 is configured with, for example, an SSD or HDD. The communication unit 323 includes a communication interface that can communicate with the storage device 310 and the observation terminal 340 (see FIG. 10 ) based on a predetermined communication standard such as Ethernet or Wi-Fi.

[0054] FIG. 9 is a diagram showing an outline of the connections between the surgical robot 4, the information processing system 5, etc., and the information transmitted and received between the respective devices.

[0055] Arranged in the operating room are an arm device 1, an operating device 2, a control device 3, an endoscope 122 attached to the arm device 1, an image processing device 302 that processes images obtained by the endoscope 122, an operating room camera 301 that captures images of the entire operating room, and a storage device 310. The control device 3, the image processing device 302, and the operating room camera 301 are communicatively connected to the storage device 310. An information processing device 320 is installed outside the operating room, for example, in a cloud environment, and is communicatively connected to the storage device 310.

[0056] The operation device 2 and the storage device 310 do not necessarily have to be installed in the operating room, but may be installed in another room in the facility.

[0057] The operating room camera 301 transmits captured images (operating room images) to the storage device 310 based on a request from the storage device 310. The endoscope 122 is connected to the image processing device 302. The image processing device 302 transmits images (endoscopic images) captured by the endoscope 122 to the operation device 2. Based on a request from the storage device 310, the image processing device 302 transmits the endoscopic images captured by the endoscope 122 to the storage device 310. The operation device 2 displays the endoscopic images received from the image processing device 302 on the viewer 211 (see FIG. 3 ).

[0058] As described above, the operator, who is a doctor, operates the operation device 2 to drive the arm device 1 and perform surgery.

[0059] At this time, the operation device 2 transmits a drive instruction based on the generated output to the control device 3 in real time in response to the operation of the operator, that is, every time the encoders 261, 271 to 276 and the limit sensors 281 to 287 (see FIG. 12) generate an output. The operation device 2 also transmits a current value indicating the state of the operation target of the operation device 2, which changes in response to the operation of the operator, to the control device 3 at a predetermined time interval (for example, every one second). The control device 3 transmits a state log generated from a plurality of current values ​​received from the operation device 2 to the storage device 310 based on a request from the storage device 310. The control device 3 converts the drive instruction received from the operation device 2 to one for the arm device 1 and transmits the converted drive instruction to the arm device 1 in real time, that is, every time a drive instruction is received from the operation device 2. As a result, the arm device 1 operates in response to the operation of the operation device 2 by the operator. The arm device 1 transmits current values ​​indicating the operation of the arms 111 to 114, etc. to the control device 3 at a predetermined time interval (for example, every one second). The control device 3 transmits an operation log generated from the current value received from the arm device 1 to the storage device 310 based on a request from the storage device 310.

[0060] The storage device 310 stores the status log and operation log received from the control device 3 in the storage unit 312. The storage device 310 also stores the video (operating room images) received from the operating room camera 301 and the video (endoscopic images) received from the video processing device 302 in the storage unit 312. The reception of the operating room images, endoscopic images, status log, and operation log continues for a predetermined period, for example, from the start to the end of use of the operating room where the surgery is performed. The storage device 310 transmits the operating room images, endoscopic images, status log, and operation log stored in the storage unit 312 to the information processing device 320 in real time.

[0061] If necessary, the control unit 311 of the storage device 310 may process the operating room images stored in the storage unit 312. For example, the control unit 311 may blur information included in the operating room images that may lead to the estimation of people and personal information of patients, so that the people and personal information cannot be identified.

[0062] The information processing device 320 stores the operating room images, endoscopic images, status logs, and operation logs received from the storage device 310 in the storage unit 322. In response to a request from the observation terminal 340 (see FIG. 10 ), the information processing device 320 generates a surgical image including the operating room images, endoscopic images, a reconstructed image based on the status log, and a reconstructed image based on the operation log, and transmits the generated surgical image to the observation terminal 340.

[0063] It is difficult for an observer who is trying to learn how to operate the operation device 2 to fully learn the operation by simply watching a skilled operator (expert) operate the operation device 2 from the side. For example, if the view of the operation objects (operation unit 221 or foot pedals 231 to 237) is blocked by part of the operation device 2, the observer cannot grasp what operation is actually being performed. Furthermore, when multiple operation objects are being operated simultaneously, the observer must keep all of the operation objects in his or her field of vision, making it difficult for him or her to grasp the multiple operations.

[0064] Furthermore, when the pair of movable plates 221b of the operation unit 221 are slightly opened or closed, it is difficult to grasp the extent of the operation by looking at the operation unit 221. Also, when the feet are positioned in front of or above the foot pedals 231 to 237, it is not possible to grasp the state of preparation for such an operation by simply looking at the operating room image or the endoscopic image.

[0065] In contrast, in this embodiment, surgical images including operating room images, endoscopic images, reconstructed images based on the status log, and reconstructed images based on the operation log are transmitted to an observation terminal 340 (see FIG. 10) used by an observer. The reconstructed images based on the status log include images that visualize the operator's operation of the operation object of the operation device 2. The display unit 343 of the observation terminal 340 displays an image that visualizes the operator's operation of the operation object of the operation device 2. By referring to these images together with images showing the operation of the surgical robot 4 (arm device 1 and / or operation device 2) and / or the operation of the surgical instruments 121-124, the observer can learn the operation of the surgical robot 4 or the surgical instruments 121-124 and the operation of the operation object for that purpose.

[0066] FIG. 10 is a block diagram showing a usage pattern of the information processing device 320. As shown in FIG.

[0067] The observation terminal 340 is a computer operated by an observer who wishes to learn how an expert operates the operation device 2. The observation terminal 340 includes a control unit 341, a storage unit 342, a display unit 343, an input unit 344, and a communication unit 345.

[0068] The control unit 341 is configured by, for example, a CPU. The control unit 341 controls each hardware unit of the observation terminal 340 and executes various processes by executing a computer program stored in the storage unit 342. The storage unit 342 is configured by, for example, an SSD or HDD. The display unit 343 is configured by, for example, a liquid crystal display. The input unit 344 is configured by, for example, a keyboard and a mouse. The display unit 343 and the input unit 344 may be configured by a touch panel display. The communication unit 345 includes a communication interface capable of communicating with the information processing device 320 based on a predetermined communication standard such as Ethernet or Wi-Fi.

[0069] The control unit 341 of the observation terminal 340 executes a web browser or a predetermined application to acquire surgical images for learning how to operate the operation device 2 from the information processing device 320. Then, the control unit 341 displays the acquired surgical images on the display unit 343. The observer refers to the surgical images displayed on the display unit 343 and learns how the doctor operator operates the operation device 2. The surgical images displayed on the display unit 343 will be described later with reference to FIG. 17 and subsequent figures.

[0070] FIG. 11 is a block diagram showing the configuration of the arm device 1.

[0071] The arm device 1 includes a control unit 131, a storage unit 132, a communication unit 133, a plurality of operation units 140, and sensors 151-154.

[0072] The control unit 131 is configured with, for example, an FPGA or a CPU. The storage unit 132 is configured with, for example, a ROM and a RAM. The communication unit 133 includes a communication interface that can communicate with the control device 3 based on a predetermined communication standard.

[0073] The operating unit 140 corresponds to each operating portion of the base arm 103 and the arms 111 to 114 of the arm device 1. The operating unit 140 corresponds to, for example, one joint portion of the arm. Each operating unit 140 includes a motor 141 and an encoder 142. The motor 141 is a stepping motor. The encoder 142 outputs the driving amount of the motor 141. One encoder 142 is provided at one end of each of the 12 shafts included in each of the arms 111 to 114.

[0074] Sensors 151 to 154 are installed on arms 111 to 114, respectively, and detect attachment and detachment of surgical instruments to and from arms 111 to 114. Sensors 151 to 154 may be, for example, sensors that detect when a surgical instrument is electrically connected to an arm, or photoelectric sensors that detect when a surgical instrument is physically attached or detached.

[0075] The name and serial number of the surgical instrument are assigned to the surgical instrument attached to the arm device 1. When the surgical instrument is attached to the arm 111-114, the control unit 131 stores the name and serial number assigned to the surgical instrument in the memory unit 132.

[0076] The control unit 131 drives the motor 141 of the corresponding operating unit 140 based on a drive instruction (see FIG. 9 ) received from the control device 3 via the communication unit 133. The control unit 131 also transmits the current value constituting the operation log based on the output value of the encoder 142, the detection signals of the sensors 151-154, and the names and serial numbers of the surgical instruments 121-124 to the control device 3 via the communication unit 133. The operation log, the detection signals of the sensors 151-154, and the names and serial numbers of the surgical instruments 121-124 are transmitted from the control device 3 to the information processing device 320 via the storage device 310 and stored in the storage unit 322 of the information processing device 320.

[0077] FIG. 12 is a block diagram showing the configuration of the operation device 2. As shown in FIG.

[0078] The operation device 2 includes a control unit 251, a storage unit 252, a communication unit 253, a viewer 211, a head sensor 212, an operation unit 221, movable units 222 to 226, foot pedals 231 to 237, and foot sensors 241 to 248. For convenience, Fig. 12 shows the configuration of one of the left and right hand controllers 220.

[0079] The control unit 251 is configured with, for example, an FPGA or a CPU. The storage unit 252 is configured with, for example, a ROM and a RAM. The communication unit 253 includes a communication interface that can communicate with the endoscope 122 and the control device 3 based on a predetermined communication standard.

[0080] The operation unit 221 includes encoders 261 and 271. The movable units 222 to 226 include encoders 272 to 276, respectively. The foot pedals 231 to 237 include limit sensors 281 to 287, respectively. The encoders 261 and 271 to 276 output the amount of movement or rotation of the corresponding parts.

[0081] Specifically, encoder 261 of operation unit 221 outputs the amount by which a pair of movable plates 221b (see FIG. 4) of operation unit 221 are pressed. Encoders 271 to 276 output the amount of rotation between operation unit 221 and movable unit 222, the amount of rotation between movable units 222 and 223, the amount of rotation between movable units 223 and 224, the amount of rotation between movable units 224 and 225, the amount of rotation between movable units 225 and 226, and the amount of rotation between movable unit 226 and support unit 202 (see FIG. 3), respectively. Limit sensors 281 to 287 detect whether foot pedals 231 to 237 are stepped on. Detection signals from limit sensors 284 to 287 are information indicating whether foot pedals 234 to 237 are switched between energized and de-energized.

[0082] The control unit 251 displays the endoscopic image received from the video processing device 302 on the viewer 211. The control unit 251 transmits current values ​​constituting a status log based on the output values ​​of the encoders 261, 271 to 276 and the detection signals of the limit sensors 281 to 287 to the control device 3 via the communication unit 253. The status log is transmitted from the control device 3 to the information processing device 320 via the storage device 310 and stored in the storage unit 322 of the information processing device 320.

[0083] Fig. 13 is a diagram illustrating an example of current values ​​constituting a status log transmitted from the operation device 2 to the control device 3. Information in which current values ​​such as those shown in Fig. 13 are stored together with the current time at predetermined intervals for a predetermined period becomes the status log.

[0084] 13, the current values ​​indicating the current state of each part include the output values ​​of the encoders 261, 271 to 276 corresponding to the left and right hand controllers 220, the angle between the pair of movable plates 221b calculated from the output value of the encoder 261 corresponding to the left operation part 221, and the angle between the pair of movable plates 221b calculated from the output value of the encoder 261 corresponding to the right operation part 221. The angle between the pair of movable plates 221b is calculated by the control part 251 of the operation device 2.

[0085] 13, the current values ​​of the 1_1 axis to 1_7 axis of the hand controller (right) indicate the output values ​​of encoders 271 to 276 provided at one end of the seven axes of the right hand controller 220. The current value of the 1_grip angle of the right hand controller indicates the angle between the pair of movable plates 221b calculated from the output value of encoder 261 corresponding to the right operation unit 221. Similarly, the current values ​​of the 1_1 axis to 1_7 axis of the left hand controller indicate the output values ​​of encoders 271 to 276 provided at one end of the seven axes of the left hand controller 220. The current value of the 1_grip angle of the left hand controller indicates the angle between the pair of movable plates 221b calculated from the output value of encoder 261 corresponding to the left operation unit 221. The current values ​​constituting the status log also include detection signals of limit sensors 281 to 287 corresponding to foot pedals 231 to 237, detection signals of foot sensors 241 to 248, and a detection signal of the head sensor 212.

[0086] The control unit 251 of the operation device 2 transmits such current values ​​to the control device 3 at intervals of, for example, one second. The control device 3 stores a set of the received multiple current values ​​and multiple current times corresponding to each current value as a status log in the storage unit 32.

[0087] Fig. 14 is a diagram showing an example of a status log. Fig. 14 shows a status log of the angle between the pair of movable plates 221b calculated from the output value of the encoder 261 corresponding to the left operation unit 221, and includes the current time every second within a predetermined period and the current value (angle) at that current time.

[0088] Fig. 15 is a diagram showing an example of current values ​​constituting the operation log transmitted from the arm device 1 to the control device 3. Information in which current values ​​such as those shown in Fig. 15 are stored together with the current time for a predetermined period at predetermined intervals constitutes the operation log. For convenience, Fig. 15 shows current values ​​corresponding to each part of the arm 111 and a part of the arm 112.

[0089] 15, the current values ​​of axes 1_1 to 1_12 of operation arm 1 indicate the output values ​​of encoder 142 provided at one end of each of the 12 axes of arm 111. Similarly, the current values ​​of axes 2_1 to 2_12 of operation arm 2 indicate the output values ​​of encoder 142 provided at one end of each of the 12 axes of arm 112 (current values ​​of axes 2_5 to 2_12 are omitted in FIG. 15).

[0090] When the control device 3 receives a drive instruction from the operating device 2, it converts the received drive instruction into a format suitable for the arm device 1 and transmits it to the arm device 1. The arm device 1 drives each motor 141 based on the received drive instruction.

[0091] The current values ​​constituting the operation log include, as the current values ​​of each part, the output values ​​of each encoder 142 that indicate the amount of operation of each motor 141 that has operated based on a drive instruction.

[0092] The control unit 131 of the arm device 1 transmits such current values ​​at intervals of, for example, one second to the control device 3. The control device 3 stores a set of the received multiple current values ​​and multiple current times corresponding to each current value as an operation log in the storage unit 132.

[0093] Fig. 16 is a diagram showing an example of an operation log. Fig. 16 shows an operation log of the motor operation amount of a predetermined movable part of the arm 111, and includes the current time every second within a predetermined period and the current value (operation amount) at that current time.

[0094] Next, an image displayed on the display unit 343 of the observation terminal 340 (see FIG. 10) will be described.

[0095] When the control unit 321 of the information processing device 320 receives instruction information from the observation terminal 340, it transmits the endoscopic images and operating room images stored in the memory unit 322 to the observation terminal 340, and also generates various images and information based on the status log and operation log stored in the memory unit 322 and transmits them to the observation terminal 340. The observation terminal 340 displays surgical images 400, including the endoscopic images and operating room images and various information based on the status log and operation log, on the display unit 343. The display of the surgical images 400 on the display unit 343 of the observation terminal 340 may be so-called download playback, in which display begins after the entire period of images and information has been received from the information processing device 320, or so-called streaming playback, in which display is started while images and information is being received from the information processing device 320.

[0096] The images displayed on the display unit 343 are not limited to images transmitted from the information processing device 320 to the observing terminal 340. The images displayed on the display unit 343 may be images generated by the control unit 341 of the observing terminal 340 based on the status log or operation log received from the information processing device 320.

[0097] FIG. 17 is a diagram showing a schematic configuration of a surgical image 400. As shown in FIG.

[0098] The surgical image 400 includes an endoscopic image 401, an operating room image 402, schematic images 403 and 404, display switch buttons 411 and 412, a video control area 420, and an information display area 430. The surgical image 400 also includes an image 440 showing information from the head sensor 212, an image 450 showing a surgical instrument usage record, and an image 460 showing the number of views. Images 440, 450, and 460 will be described later with reference to FIG. 20.

[0099] 17, endoscopic image 401, operating room image 402, and schematic images 403 and 404 are all moving images. Schematic image 403 is a moving image in which the operating states of each part of arm device 1 are constructed in three dimensions based on the operation log of arm device 1. Schematic image 404 is a moving image in which the operating state of hand controller 220 of operation device 2 is constructed in three dimensions based on the state log of operation device 2. These four moving images are synchronized with each other and show the states at the same time.

[0100] Display switching buttons 411 and 412 are buttons for switching between a mode in which four videos (endoscopic image 401, operating room image 402, and schematic images 403 and 404) are displayed side by side, and a mode in which one of the four videos is enlarged and displayed. When display switching button 411 is operated, the four videos are displayed side by side as shown in Fig. 17, and when display switching button 412 is operated, one video is enlarged and displayed as shown in Fig. 18.

[0101] When display switch button 411 is operated, a screen for setting the positions at which the four videos are to be arranged is displayed, and by setting this screen, the four videos can be displayed in desired positions, as shown in Fig. 17. When display switch button 412 is operated, a screen for setting which of the four videos is to be arranged is displayed, and by setting this screen, a desired one of the four videos can be displayed, as shown in Fig. 18.

[0102] FIG. 19 is an enlarged view that schematically shows the configuration of the endoscopic image 401. As shown in FIG.

[0103] In the endoscopic image 401, display areas 401a to 401e showing information about the surgical instruments 121 to 124 attached to the arms 111 to 114 of the arm device 1 are displayed together with the image captured by the endoscope 122. The endoscopic image 401 is the same image as the image displayed on the viewer 211 of the operation device 2.

[0104] The display area 401a shows the operational state of the foot pedal 232, which is a clutch pedal. When a foot is positioned on the foot pedal 232, the outer periphery of the display area 401a is colored. When the foot pedal 232 is stepped on, the interior of the display area 401a is colored. The display area 401b shows the operational state of the foot pedal 233, which is used to operate the arm to which the endoscope 122 is attached. When a foot is positioned on the foot pedal 233, the outer periphery of the display area 401b is colored, and when the foot pedal 233 is stepped on, the interior of the display area 401b is colored.

[0105] Display area 401c shows the operating state of foot pedals 234, 235 for operating forceps (electric scalpels). When the foot is positioned on foot pedal 234 or 235, the periphery of display area 401c is colored, when foot pedal 234 is pressed, a first color is applied to the interior of display area 401c, and when foot pedal 235 is pressed, a second color is applied to the interior of display area 401c. The first color is, for example, light blue, and the second color is, for example, yellow.

[0106] Display area 401d shows the operating state of foot pedals 236, 237 for operating forceps (electric scalpels), and display area 401e shows the operating state of foot pedals 236, 237 for operating other forceps (electric scalpels). As described above, each time foot pedal 231 is pressed, the target of foot pedal 236, 237 is switched between forceps 123, 124. When the foot is positioned on foot pedal 236 or 237, the outer periphery of the target display area of ​​display areas 401d, 401e is colored; when foot pedal 236 is pressed, the interior of the target display area of ​​display areas 401d, 401e is colored a first color; and when foot pedal 237 is pressed, the interior of the target display area of ​​display areas 401d, 401e is colored a second color.

[0107] Furthermore, display areas 401c to 401e display, for example, the names of surgical instruments 121, 123, and 124 attached to arms 111, 113, and 114. Whether or not a surgical instrument is attached and the names of the surgical instruments are generated based on the detection signals of sensors 151 to 154 and the names and serial numbers of surgical instruments 121 to 124 acquired when attached to arms 111 to 114.

[0108] By referring to the display areas 401a to 401e in the endoscopic image 401, the observer can grasp the operating states of the foot pedals 232 to 237 and can grasp what surgical instruments are attached to the arms 111 to 114.

[0109] 19, a dividing line 401f that divides the area of ​​the endoscopic image 401 into four parts can also be displayed within the endoscopic image 401. This allows the observer to refer to the four areas divided by the dividing line 401f as a guide to see in which area each surgical instrument is positioned, and to learn how to position and move each surgical instrument depending on the surgical situation.

[0110] Returning to FIG. 17, the video control area 420 includes a play button 421 , a play position mark 422 , a play speed setting button 423 , a warning mark 424 , a bookmark insertion button 425 , a bookmark mark 426 , and an edit start button 427 .

[0111] When the playback button 421 is operated, the endoscopic image 401, the operating room image 402, and the schematic images 403 and 404 are switched between playback and pause.

[0112] The playback position mark 422 is a mark that indicates the playback position of the endoscopic image 401, the operating room image 402, and the schematic images 403 and 404 on the timeline 420a. The playback position can be changed by operating the playback position mark 422. When the playback speed setting button 423 is operated, a submenu is opened, and the playback speed can be set in the submenu. The attention mark 424 indicates the position on the timeline 420a where an event requiring the observer's attention has occurred, such as the position on the timeline 420a where an error has occurred in the surgical robot 4.

[0113] When the bookmark insertion button 425 is operated, a bookmark mark 426 is added to the position of the playback position mark 422. In addition, a screenshot of the surgical image 400 at the time the bookmark insertion button 425 is operated is stored in the storage unit 322 of the information processing device 320. The storage of the screenshot of the surgical image 400 will be described later with reference to FIG. 25 .

[0114] When the start editing button 427 is operated, the observer can write text and figures by clicking and dragging on the surgical image 400. When the insert bookmark button 425 is operated in this state, a screenshot of the surgical image 400 is stored in the storage unit 322 of the information processing device 320 together with the text and figures written on the surgical image 400. The fact that the start editing button 427 enables writing text and figures on the surgical image 400 will be described later with reference to FIG. 26 .

[0115] FIG. 20 is a diagram showing a schematic configuration of images 440, 450, and 460 displayed within a surgical image 400. As shown in FIG.

[0116] The band areas in the images 440, 450, and 460 correspond to the timeline 420a in the video control area 420 shown in FIG.

[0117] The band region of image 440 showing information about head sensor 212 includes portions displayed in black and portions displayed in white. The portions displayed in black indicate a state in which the operator is looking into viewer 211 by head sensor 212, i.e., a state in which the operator's approach to viewer 211 has been detected by head sensor 212. The portions displayed in white indicate a state in which the operator is not looking into viewer 211 by head sensor 212, i.e., a state in which the operator's approach to viewer 211 has not been detected by head sensor 212.

[0118] Image 450 showing the surgical instrument usage history includes band areas numbered 1 to 4. Band areas 1 to 4 correspond to arms 111 to 114, respectively. Hatched areas within the band areas indicate the type of forceps or endoscope attached to arms 111 to 114 and the duration of attachment. In the example shown in FIG. 20 , band areas 1 and 3 indicate that the forceps attached to arms 111 and 113 were replaced during the surgery. Band area 2 indicates that the endoscope 122 attached to arm 112 was removed several times near the latter half of the surgery. This is because the endoscope 122 was removed from arm 112 to wipe off fogging on the head of the endoscope 122. Band area 4 indicates that the forceps 124 attached to arm 114 was never removed.

[0119] Furthermore, when a cursor such as a mouse is placed over a band area in image 450, surgical instrument information 451 is displayed, including the name of the forceps or endoscope that was attached at the time position where the cursor was placed, and the time period during which the forceps or endoscope was used, as shown in band area number 4.

[0120] The band region of image 460 showing the number of views includes areas of different shades. This shade indicates the number of times the viewer has viewed each position. For example, a portion displayed in white indicates that the viewer has viewed this portion zero times, and each step of the shade increases the number of views by a predetermined number.

[0121] Furthermore, when a cursor such as a mouse is placed over the band area in the image 460, viewing count information 461 indicating the number of viewings at the time position where the cursor is placed is displayed.

[0122] FIG. 21 is an enlarged view that schematically shows the configuration of information display area 430 when the operation information tab is selected.

[0123] The display contents of the information display area 430 are displayed in synchronization with the playback positions of the endoscopic image 401, the operating room image 402, and the schematic images 403 and 404.

[0124] When the operation information tab is selected, information display area 430 includes display selection area 431, two images 432a, two images 432b, two images 433a, two images 433b, two images 434, and images 435 to 437. Images 432a, 432b, 433a, 433b, and 434 are arranged on the left and right sides within information display area 430.

[0125] The left image 432a schematically shows the pressed state of the pair of movable plates 221b of the left hand controller 220, and the right image 432a schematically shows the pressed state of the pair of movable plates 221b of the right hand controller 220. The left image 432b shows the angle corresponding to the pressed amount of the pair of movable plates 221b of the left hand controller 220, and the right image 432b shows the angle corresponding to the pressed amount of the pair of movable plates 221b of the right hand controller 220. The images 432a and 432b are generated based on the status log. By referring to the images 432a and 432b, the observer can understand to what extent the pair of movable plates 221b of the operation unit 221 are open or closed.

[0126] Left and right images 433a respectively show the clamping state of the tips of the forceps operated by the left and right hand controllers 220. Left and right images 433b respectively show the angle corresponding to the clamping amount of the forceps operated by the left and right hand controllers 220. Images 433a and 433b are generated based on the operation log. By referring to images 433a and 433b, the observer can understand to what extent the tips of the corresponding forceps are opening and closing.

[0127] The left image 434 shows a graph corresponding to the depression state of the foot pedals 234 and 235, and the right image 434 shows a graph corresponding to the depression state of the foot pedals 236 and 237. In the image 434, ON indicates a state in which the foot pedal is depressed, and OFF indicates a state in which the foot pedal is not depressed. The image 434 shows the depression state going back 30 seconds. The image 434 is generated based on the state log. By referring to the left image 434, an observer can understand the timing and duration when the foot pedals 234 and 235 were depressed, and by referring to the right image 434, an observer can understand the timing and duration when the foot pedals 236 and 237 were depressed.

[0128] The image 435 schematically shows the position of the operator's feet on the foot unit 230 and the depression states of the foot pedals 231 to 237. The image 435 is generated based on the status log.

[0129] 22 and 23 are diagrams showing the structure and transition of the image 435.

[0130] As shown in FIG. 22, the image 435 includes pedal images 471 to 477 corresponding to the foot pedals 231 to 237, and a foot image 481 that is displayed depending on the standby state of the foot.

[0131] The control unit 321 of the information processing device 320 determines whether each of the foot pedals 231-237 is in a hover state or a non-hover state based on the detection signals of the foot sensors 241-248. As described above, the hover state includes a state in which the foot is in front of the foot pedal, a state in which the foot is above the foot pedal, and a state in which the foot pedal is being stepped on, and the non-hover state includes a state in which the foot is not inside the foot unit 230. The control unit 321 determines whether each of the foot pedals 231-237 is being stepped on based on the detection signals of the limit sensors 281-287. Then, the control unit 321 generates an image 435 based on the determination result.

[0132] 22, when neither foot nor pedal image 471 is positioned within foot unit 230, pedal images 471 to 477 are displayed with normal frame thickness, and foot image 481 is not displayed. This indicates that neither foot is operating or ready to operate foot pedals 231 to 237.

[0133] When the feet are positioned in front of or above the foot pedals from the state shown in the upper part of Fig. 22, foot image 481 is displayed, as shown in the lower part of Fig. 22. In the example shown in the lower part of Fig. 22, two foot images 481 are positioned in front of pedal images 473 and 475, respectively. This indicates that the left foot is positioned in front of or above foot pedal 233, and the right foot is positioned in front of or above foot pedal 235. It also indicates that both feet are ready to operate foot pedals 233 and 235.

[0134] When the foot pedal is depressed from the state shown in the lower part of Fig. 22, the outer periphery of the corresponding pedal image is colored, as shown in the upper part of Fig. 23. In the example shown in the upper part of Fig. 23, the outer periphery of pedal image 477 is colored. This indicates that foot pedal 237 is being depressed.

[0135] When the foot pedal 233 is depressed from the state shown in the upper part of Fig. 23, the outer periphery of the pedal image 473 is colored, as shown in the lower part of Fig. 23. Also, in the state shown in the lower part of Fig. 23, the right foot has moved outside the foot unit 230, so there is no foot image 481 corresponding to the right foot. This indicates that the right foot is not ready for operation.

[0136] By referring to the pedal images 471 to 477 and the foot image 481 in the image 435, the observer can understand the operation and preparation state of the foot pedals 231 to 237.

[0137] 21, image 436 shows the detection result based on the head sensor 212 and the cumulative time that the operator has looked into the viewer 211, as determined by the head sensor 212. Image 436 is generated based on the status log. By referring to image 436, the observer can determine whether or not the operator is looking into the viewer 211, and how long the operator has been looking into the viewer 211 cumulatively during surgery.

[0138] Image 437 shows the names and serial numbers of the surgical instruments 121-124 attached to the arms 111-114 of the arm device 1. Image 437 is generated based on the detection signals of sensors 151-154 and the names and serial numbers of the surgical instruments 121-124 acquired when they were attached to the arms 111-114. By referring to image 437, the observer can ascertain the current names and serial numbers of the surgical instruments 121-124.

[0139] FIG. 24 is an enlarged view that schematically shows the configuration of the information display area 430 when the setting information tab is selected.

[0140] When the setting information tab is selected, the information display area 430 includes a display selection area 431 and images 438 and 439 .

[0141] Image 438 is an image that schematically shows the pivot positions of the surgical instruments 121-124 attached to the arms 111-114. A pivot position is a position on a surgical instrument where the relative position between the surgical instrument and the arm to which the surgical instrument is attached does not change when the surgical instrument moves in response to movement of the arms 111-114. The surgical instruments 121-124 are moved by the arms 111-114 with the pivot positions as fulcrums. Image 438 shows the distance between each pivot position.

[0142] Image 439 is an image showing the settings of the operation device 2. Image 439 shows the load (weight of the hand controller 220) when the operator operates the operation unit 221, the ratio (scaling) between the movement amount of the hand controller 220 and the actual movement amount of the surgical instrument, the height of the armrest 203a, and the position (depth) of the foot unit 230 in the front-to-rear direction.

[0143] By referring to the images 438 and 439, the observer can grasp the pivot positions of the surgical instruments 121 to 124 and the settings of the operating device 2.

[0144] The display selection area 431 may include tabs other than the operation information and setting information, and by selecting the tabs, various information may be displayed in the information display area 430. For example, information about other devices related to the surgery may be displayed.

[0145] FIG. 25 is a diagram showing a schematic configuration of an edited image 500 that is displayed when the bookmark insertion button 425 is operated on the surgical image 400. As shown in FIG.

[0146] When the bookmark insertion button 425 is operated on the surgical image 400, the surgical image 400 at that time is automatically saved as a screenshot 502 in the storage unit 322 of the information processing device 320.

[0147] The edited image 500 includes a bookmark display area 501 , a screenshot 502 , an edit tool area 503 , a text input area 504 , a close button 511 , and a loop playback button 512 .

[0148] The bookmark display area 501 displays a bookmark item 501a corresponding to the set bookmark. When a bookmark item 501a is operated, the bookmark item 501a is selected, and the screenshot 502, image 502a, and text 504a corresponding to the selected bookmark item 501a are displayed. When the bookmark insertion button 425 is operated on the surgical image 400 and the edited image 500 is displayed, the bookmark item 501a is selected. When the trash can icon in the bookmark item 501a is operated, the corresponding bookmark is deleted.

[0149] The screenshot 502 is a screenshot of the surgical image 400 corresponding to the selected bookmark item 501a. The observer can insert a hand-drawn image 502a into the screenshot 502 by dragging the mouse over the screenshot 502. The observer can also change the color and line thickness of the shape written in the screenshot 502 by operating various buttons and sliders in the editing tool area 503. The observer can add text 504a to the bookmark item 501a by selecting the text input area 504 and inputting characters via the keyboard.

[0150] When the close button 511 is operated, the image 502a and text 504a set in the edited image 500 are stored in the storage unit 322 of the information processing device 320 in association with the screenshot 502. The edited image 500 is then closed, and the surgical image 400 is displayed again. By saving the screenshot 502, the image 502a, and the text 504a, the observer can subsequently refer to the saved content to smoothly learn the operation performed by the operator.

[0151] When the loop playback button 512 is operated, similarly to the close button 511, the image 502a and text 504a set in the edited image 500 are associated with the screenshot 502 and stored in the storage unit 322 of the information processing device 320, the edited image 500 is closed, and the surgical image 400 is displayed again. In this case, furthermore, in the surgical image 400, the video before and after (for example, ±5 seconds) the bookmark corresponding to the bookmark item 501a that was selected when the edited image 500 was closed is repeatedly played. This allows the observer to smoothly progress with their learning by repeatedly referring to each image for scenes that they want to focus on.

[0152] Note that the number of times the video is repeatedly played when the loop playback button 512 is operated is not particularly limited, and playback may be repeated until the viewer gives an instruction to end playback, or playback may end after a preset number of repetitions. Also, the video does not necessarily have to be played repeatedly, and may end after being played once.

[0153] When a lecture is given using observation terminal 340, the lecturer can insert hand-drawn image 502a to draw the students' attention to the part indicated by image 502a. This allows the students to smoothly learn the operations performed by the operator.

[0154] In Figure 25, we have explained the case where a hand-drawn image 502a is inserted into the screenshot 502 after the bookmark insertion button 425 is operated in the surgical image 400, but it is also possible to insert a hand-drawn image into the surgical image 400 first.

[0155] FIG. 26 is a diagram showing a schematic configuration of a surgical image 400 when a hand-drawn image 491 is inserted first.

[0156] 17 and 18, when the edit start button 427 is operated, an edit tool area 428 is displayed in place of the edit start button 427 in the video control area 420 of the surgical image 400, as shown in Fig. 26. The edit tool area 428 includes an icon for changing the color of the hand-drawn lines and an end icon for ending writing.

[0157] 26, the observer operates the mouse to insert the hand-drawn image 491 and presses the bookmark insertion button 425. As a result, a screenshot 502 of the surgical image 400 and the hand-drawn image 491 are automatically saved in the storage unit 322 of the information processing device 320. Then, an edited image 500 including the screenshot 502 with the hand-drawn image 491 inserted is displayed. Note that the hand-drawn image 491 may be inserted while the surgical image 400 is being played back, or may be inserted after the playback of the surgical image 400 is temporarily stopped.

[0158] Next, the processing performed by the control unit 321 of the information processing device 320 will be described with reference to FIGS.

[0159] FIG. 27 is a flowchart showing the process of receiving the information necessary to generate the surgical image 400.

[0160] In step S1, the control unit 321 of the information processing device 320 receives the endoscopic image, the operating room image, the status log, and the operation log transmitted in real time from the storage device 310 and stores them in the storage unit 322. The status log includes a log indicating the status (grip angle) of the operation unit 221 based on the output of the encoder 261, a log indicating the status (amount of movement) of the movable units 222 to 226 based on the output of each of the encoders 272 to 276, a log indicating the status (e.g., energized or non-energized) of the foot pedals 231 to 237 based on the output of each of the limit sensors 281 to 287, and a log indicating the status of the viewer 211 (whether the operator is approaching the viewer) based on the output of the head sensor 212. The operation log includes a log indicating the operation (amount of operation) of the operating unit 140 of the arm device 1 based on the output of the encoder 142.

[0161] 28 is a flowchart showing the process of generating the surgical image 400. Each process in FIG.

[0162] In step S2, the control unit 321 generates first to third reconstructed images based on the status log and the operation log. The first reconstructed image includes images 432a, 432b, 434, 435, and 440 that visualize operations on operation objects (operation unit 221, foot pedals 231 to 237, and viewer 211) that the operator touches and operates. The second reconstructed image includes image 435 that schematically shows a state of preparation for operation on the operation objects (foot pedals 231 to 237) (a state in which the feet are positioned in front of or above the foot pedals). The third reconstructed image includes images 433a and 433b that visualize the operation of the forceps attached to the arm of the arm device 1.

[0163] Also, in step S2, the control unit 321 generates a schematic image 404 based on the log indicating the state of the operating unit 221 (grip angle; in Figure 13, the 1_grip angle of the hand controller (right) and the 1_grip angle of the hand controller (left)) and the log indicating the state of the movable parts 222 to 226 (movement amount; in Figure 13, the 1_1 axis to 1_7 axis of the hand controller (right) and the 1_1 axis to 1_7 axis of the hand controller (left)), and generates a schematic image 403 based on the operation log.

[0164] In step S3, the control unit 321 generates a surgical image 400 based on the endoscopic image 401, the operating room image 402, the schematic images 403 and 404 generated in step S2, and the first to third reconstructed images generated in step S2. In step S4, the control unit 321 adds additional information to the surgical image 400 generated in step S3.

[0165] FIG. 29 is a flowchart showing the details of the additional information addition process in step S4 of FIG.

[0166] In step S11, the control unit 321 of the information processing device 320 adds information indicating the portions of the endoscopic images 401 that have already been viewed among the endoscopic images 401 for the entire period and the number of times they have been viewed (image 460 indicating the number of times viewed in FIG. 20) to the surgical image 400. In step S12, the control unit 321 adds information indicating whether or not the surgical instruments 121 to 124 are attached in chronological order (image 450 indicating the use record of the surgical instruments in FIG. 20) and information indicating the type of surgical instrument (surgical instrument information 451 in FIG. 20 and image 437 in FIG. 21) to the surgical image 400.

[0167] 28, in step S5, the control unit 321 provides the surgical image 400 to which the additional information was added in step S4 to the observation terminal 340. After executing the processes of steps S1 to S4 for the endoscopic images, operating room images, status logs, and operation logs for a predetermined period, the control unit 321 executes the process of step S5. The control unit 321 repeatedly executes the processes of steps S1 to S5 until it receives an instruction from the observation terminal 340 to end the display of the surgical image 400, and ends the process of FIG. 28 when it receives the end instruction.

[0168] The control unit 321 may execute the process of step S5 after executing the processes of steps S1 to S4 for the endoscopic images, operating room images, status logs, and operation logs for the entire period. In this case, the repeated processes of steps S1 to S5 are not necessary.

[0169] In the processing of Figure 28, in the process of providing surgical image 400 in step S5, control unit 321 provides, on display unit 343 of observation terminal 340, an image (video) in which endoscopic image 401, operating room image 402, schematic images 403 and 404, video control area 420, each image in information display area 430, and images 440, 450, and 460 are displayed in synchronization with each other while changing over time.

[0170] FIG. 30 is a flowchart showing the process of storing a screenshot 502 by adding a bookmark.

[0171] In step S21, the control unit 321 of the information processing device 320 determines whether the bookmark insertion button 425 has been operated. If the bookmark insertion button 425 has been operated, in step S22, the control unit 321 stores the playback position of the endoscopic image 401 and a screenshot of the surgical image 400 at the time the bookmark insertion button 425 was operated. In step S23, the control unit 321 determines whether an instruction to end playback of the surgical image 400 has been received from the observation terminal 340. If the end instruction has not been received, the control unit 321 returns the process to step S21; if the end instruction has been received, the control unit 321 ends the process of FIG. 30.

[0172] FIG. 31 is a flowchart showing the process of displaying the screenshot 502 and playing back the content before and after the bookmarked position.

[0173] In step S31, the control unit 321 of the information processing device 320 determines whether a bookmark has been selected. The selection of a bookmark is performed by operating the bookmark item 501a in the edited image 500, operating the bookmark mark 426 in the surgical image 400, and operating the bookmark insertion button 425 in the surgical image 400.

[0174] When a bookmark is selected, in step S32, the control unit 321 provides the target screenshot 502 and the hand-drawn images 502a and 491 and text 504a associated with the screenshot 502 to the observing terminal 340. As a result, as shown in FIG. 25 , an edited image 500 including the screenshot 502, the images 502a and 491, and the text 504a is displayed on the display unit 343 of the observing terminal 340.

[0175] In step S33, the control unit 321 determines whether the loop playback button 512 of the edited image 500 has been operated. If the loop playback button 512 has been operated, the edited image 500 is closed, and the surgical image 400 is displayed. Then, in step S34, the control unit 321 transmits information to the observing terminal 340 to repeatedly play back a predetermined period (for example, from -5 seconds to +5 seconds) before and after the selected bookmark position. That is, for example, ±5 seconds from the selected bookmark position is loop-played on the observing terminal 340. If the loop playback button 512 has not been operated, the process proceeds to step S35.

[0176] In step S35, the control unit 321 determines whether the close button 511 of the edited image 500 has been operated. If the close button 511 has not been operated, the control unit 321 returns the process to step S33. If the close button 511 has been operated, the edited image 500 is closed and the surgical image 400 is displayed. In this way, the process of FIG. 31 ends.

[0177] <Effects of the embodiment> The control unit 321 of the information processing device 320 acquires a status log (see FIG. 13) showing the state of the operation object (operation object) that changes when the operator operates the operation unit 221 (see FIG. 4), foot pedals 231 to 237 (see FIG. 5), and viewer 211 (see FIG. 3) of the operation device 2 (step S1 in FIG. 27), and generates images 432a, 432b, 434, 435 (see FIG. 21) and image 440 (see FIG. 20) (reconstructed images) that visualize the operator's operations on the operation object based on the status log (step S2), and provides a surgical image 400 (see FIGS. 17 and 18) that associates an endoscopic image 401, an operating room image 402, and schematic images 403 and 404 (see FIG. 17) (operation images) that show the operation of the surgical robot 4 or surgical instruments 121 to 124 (see FIG. 2) with the reconstructed image (step S5 in FIG. 28).

[0178] By referring to the operation image and the reconstructed image, the observer can easily and accurately confirm the operation of the surgical robot 4 or the surgical instruments 121 to 124 and the operation of the operation target for that purpose.

[0179] The operation unit 221 (operation target) is moved in response to an operation by the operator, and the status log includes information (position, amount of rotation, angle, presence or absence, etc.) based on the output of the encoder 261 and sensors (status detectors) that detect the status of the operation unit 221 (operation target). According to this configuration, by using the output of the encoder 261 and the like that detects the status of the operation target of the operation device 2, it is possible to generate images 432a, 432b (reconstructed images) that visualize how the operator actually moved the operation target.

[0180] The operation unit 221 (operation target) is moved in response to an operation by the operator, and the reconstructed image includes an image 432a (see FIG. 21) (schematic image) that schematically shows the degree of movement of the operation unit 221 (operation target) due to the operation by the operator. With this configuration, the subtle adjustments made by the operator can be visualized in a schematic manner, and the observer can intuitively grasp how far the operator has moved the operation unit 221. As a result, even an inexperienced person can smoothly acquire the skill of operating an operation target by an expert by checking the subtle operation made by the expert using the image 432a.

[0181] The image 432a (schematic image) is a schematic image of the operation target on the plane along which the pair of movable plates 221b (operation target) of the operation unit 221 moves. With this configuration, the observer can accurately confirm the amount of movement of the pair of movable plates 221b on the movement plane by referring to the image 432a, and therefore can more accurately grasp the movement of the pair of movable plates 221b.

[0182] The reconstructed image includes an image 432b (see FIG. 21) that numerically indicates the degree of movement of the operation unit 221 (operation object) due to the operator's operation. With this configuration, the extent to which the operator has moved the operation object can be quantitatively indicated, for example, by the distance or angle moved by the operation object, and the observer can quantitatively grasp the degree of movement of the operation object. This allows the observer to efficiently and accurately learn how to operate the operation object, making it easier to reproduce the techniques of an expert.

[0183] The operation targets include foot pedals 234-237 (operators) that switch between energizing and de-energizing the forceps (electrical instruments) attached to the arm device 1, the status log includes information indicating the switching between energizing and de-energizing by the foot pedals 234-237 (operators), and the reconstructed image includes an image 434 (see FIG. 21 ) that schematically shows the period during which the foot pedals 234-237 (operators) were operated. With this configuration, an observer using the observation terminal 340 can understand whether the operator energized the electric instrument sporadically or continuously over a certain period of time when treating an affected area with the forceps (electrical instrument) shown in the endoscopic image 401. This allows the observer to learn what kind of energizing operation to perform on the electric instrument when using an electric instrument such as an electric scalpel to incise or coagulate an affected area.

[0184] The operation object includes a viewer 211 (display unit) through which the operator observes the endoscopic image 401, the status log includes information based on the output of a head sensor 212 (sensor) that detects the operator's approach to the viewer 211, and the reconstructed image includes an image 440 (see FIG. 20 ) that schematically shows the duration and interruption periods of the operator's approach to the viewer 211. The duration of the operator's approach to the viewer 211 suggests that the operator is performing the treatment while viewing the image on the viewer 211, and the interruption periods suggest that the operator is not viewing the image on the viewer 211 and has interrupted the treatment. With this configuration, the observer can understand the intervals between treatments by the operator by referring to the image 440. This allows the observer to learn what state the treatment shown in the endoscopic image 401 is in when it is appropriate to interrupt the treatment and give instructions to an assistant, change forceps, take a rest, etc.

[0185] The reconstructed image includes images 434 and 435 that schematically show the operation timings of the foot pedals 231 to 237 (operation targets). The operation timings are shown by the timings when the signal value turns ON in image 434 and the timings when the outer peripheries of pedal images 471 to 477 (see FIG. 23) in image 435 are colored. With this configuration, the observer can easily understand at what timing the operator operated the operation targets during the treatment shown in endoscopic image 401.

[0186] Step S2 in FIG. 28 includes generating an image 435 (second reconstructed image) that schematically shows the state of preparation for operation based on outputs from foot sensors 241-248 (sensors) that detect the state of preparation for operation of the foot pedals 231-237 (operation targets). In the surgical image 400, the action image and the second reconstructed image are associated. As shown in the lower part of FIG. 22, when a foot image 481 is displayed and the periphery of the pedal image corresponding to the foot image 481 is not colored, it is clear that the foot is in a state of preparation for operation of the corresponding foot pedal. This process allows the observer to easily understand the timing at which the operator prepared to operate the foot pedals 231-237 during the procedure shown in the endoscopic image 401.

[0187] Step S1 in FIG. 27 includes a process of acquiring an operation log of the operating unit 140 of the arm device 1, which is driven by an operation on the operation unit 221 (operation target), and step S2 in FIG. 28 includes a process of generating images 433a and 433b (third reconstructed images) that visualize the drive state of the operating unit 140 based on the operation log. In the surgery image 400, the endoscopic image 401, the operating room image 402, and the schematic images 403 and 404 (operation images) are associated with the third reconstructed image. This process allows the observer to compare the operation of the operation unit 221 by the operator with how the operating unit 140 of the arm device 1 is actually driven in response to this operation.

[0188] The operation device 2 includes an operation unit 221 (operation object) and movable units 222-226 (second operation objects) that move in conjunction with the operator's contact with and operation of the operation unit 221 (operation object). The operation image includes a schematic image 404 that visualizes the overall movement of the operation unit 221 and movable units 222-226 based on the status log. This processing allows the observer to smoothly confirm how the hand controller 220 actually moved in response to the operator's operation, for example, by referring to the schematic image 404 together with images 432a and 432b (reconstructed images).

[0189] The control unit 321 of the information processing device 320 adds an image 460 relating to the number of viewings in FIG. 20 to the surgical image 400 as information indicating the portion of the surgical images that have already been viewed and the number of viewings (step S11 in FIG. 29), and provides the surgical image 400 with the image 460 added (step S5 in FIG. 28). This process allows the observer to understand the surgical steps that have been frequently referred to, such as the steps in which complex procedures and operations were performed. This allows the observer to efficiently learn the procedures and operations for the operation targets.

[0190] The operation object that the operator touches and operates is the operation unit 221 for opening and closing the tip of the forceps attached to the arm device 1. With this configuration, the observer can understand the operation of the operation object by the skilled person's hands, and can easily reproduce the skilled person's operation for the procedure.

[0191] Based on the outputs of sensors 151-154 that detect the attachment and detachment of surgical instruments 121-124 to arm apparatus 1, control unit 321 of information processing device 320 adds image 450 relating to the surgical instrument usage record of FIG. 20 to surgery image 400 as information indicating the chronological presence or absence of attachment of surgical instruments 121-124 (step S12 of FIG. 29), and provides surgery image 400 with image 450 added (step S5 of FIG. 28). This process allows the observer to grasp the timing of attachment, detachment, and replacement of surgical instruments 121-124 to arm apparatus 1 during the procedure shown in endoscopic image 401. This makes it easier for the observer to reproduce the procedures of an expert using surgical instruments 121-124.

[0192] The control unit 321 of the information processing device 320 adds the surgical instrument information 451 of FIG. 20 and the image 437 of FIG. 21 to the surgical image 400 as information indicating the types of surgical instruments 121-124 attached to the arm device 1 (step S12 of FIG. 29), and provides the surgical image 400 to which the surgical instrument information 451 and the image 437 have been added (step S5 of FIG. 28). This process allows the observer to understand what surgical instruments were used depending on the progress of the surgery. This makes it easier for the observer to reproduce the techniques of an expert using the surgical instruments 121-124.

[0193] The operation objects that the operator touches and operates are the foot pedals 231 to 237 of the operation device 2. With this configuration, the observer can understand how an expert operates the operation objects with his or her feet, and can easily reproduce the operation of the expert with his or her feet. For example, when using the foot pedal 233 to switch the operation of the hand controller 220 to adjusting the angle of the endoscope 122, the observer can refer to the image 435 (reconstructed image) to understand how frequently the angle of the endoscope 122 is being adjusted.

[0194] The foot pedals include right foot pedals 234 to 237 and left foot pedals 231 to 233. The observer can understand the operation of the foot pedals 231 to 237 with the left and right feet.

[0195] The operation objects that the operator touches and operates include the operation unit 221 and foot pedals 231-237 of the hand controller 220 for operating the surgical instruments 121-124 attached to the arm device 1, and the reconstructed image includes images 432a and 432b (first images) that visualize the operation of the operation unit 221, and images 434 and 435 (second images) that visualize the operation of the foot pedals 231-237. In the surgery image 400, as shown in FIG. 21 , the images 432a and 432b (first images) are positioned above the image 435 (second image). With this configuration, the vertical positions of the operator's hands and feet match the vertical positions of the operation objects in the surgery image 400, allowing the observer to more smoothly learn the operation by the operator.

[0196] In the step of providing surgical images (step S5 in FIG. 28), the control unit 321 of the information processing device 320 changes the endoscopic image 401, the operating room image 402, and the schematic images 403 and 404 (operation images) and the images 432a, 432b, 434, and 435 (reconstructed images) over time. This processing allows the observer to smoothly grasp the operation of the operation target as the surgery progresses through the operation images and reconstructed images that change over time, i.e., moving images.

[0197] In the step of providing the surgical image 400 (step S5 in FIG. 28), the control unit 321 of the information processing device 320 synchronizes the endoscopic image 401, the operating room image 402, and the schematic images 403 and 404 (operation images) with the images 432a, 432b, 434, and 435 (reconstructed images). This process allows the observer to smoothly confirm the operation of the operation target corresponding to the treatment shown in the operation image.

[0198] The control unit 321 of the information processing device 320 accepts the designation of any time point via the bookmark insertion button 425 while the surgical image 400 that changes over time is being displayed (step S21 in FIG. 30), and automatically saves a screenshot 502 (surgical image 400) at the designated time point (step S22). Because the observer can save the surgical image 400 at any time while the surgical image 400 that changes over time is being displayed, the observer can smoothly learn the operations during the surgical procedure.

[0199] For example, if the operator operates quickly and is unable to smoothly compare endoscopic image 401, operating room image 402, and schematic images 403 and 404 (operation images) with images 432a, 432b, 434, and 435 (reconstructed images), the observer can save screenshot 502 corresponding to surgical image 400 at that time, as shown in FIG. 25. Furthermore, after the surgery, if the observer receives a lecture from an expert while referring to surgical image 400 and the expert makes annotations or the like on surgical image 400 as shown in FIGS. 25 and 26, the observer can save images 502a and 491 corresponding to the annotations along with screenshot 502 corresponding to surgical image 400. This allows the observer to smoothly learn the operations for the procedure by subsequently referring to saved screenshot 502.

[0200] The control unit 321 of the information processing device 320 accepts the designation of an arbitrary time point via the bookmark insertion button 425 while the surgical image 400, which changes over time, is displayed (step S21 in FIG. 30 ), accepts a playback instruction for the designated time point via the loop playback button 512 (step S33 in FIG. 31 ), and plays back the surgical image 400 for a predetermined period before and after the designated time point based on the playback instruction (step S34). This process allows the observer to designate an arbitrary time point during the display of the surgical image 400, which changes over time, and check the action image and reconstructed image around that time point by playing it back. For example, if the operator's operations are too fast to smoothly compare the action image and the reconstructed image, the observer can designate that time point and check the action image and reconstructed image around that time by playing it back. This allows the observer to smoothly learn the operations for the surgical procedure.

[0201] 17, in the surgery image 400, an endoscopic image 401 and schematic images 403 and 404 (operation images) are associated with an operating room image 402 obtained from an operating room camera 301 that captures an image of the operating room in which the arm device 1 is installed. As shown in FIG. 32, an image 405 of the operating device 2, an image 406 of the patient's abdomen, an image 407 of the equipment storeroom, and an image 408 of the corridor leading to the operating room may be captured by the camera, and the images 405 to 408 may be displayed together with the operation image in the surgery image 400. In this case, the images 405 to 408 are moving images captured simultaneously with the operation images.

[0202] As described above, in the surgical image 400, the endoscopic image 401 and the schematic images 403 and 404 (operation images) are associated with images obtained from a camera capturing at least one of the operating room where the arm device 1 is installed, the control device 2, the patient's abdomen, the instrument storeroom, and the corridor leading to the operating room. This allows the observer to confirm the status of the operating room, the control device 2, the patient's abdomen, the instrument storeroom, and the corridor according to the progress of the procedure shown in the operation images. For example, the observer can confirm the assistance and actions of the assistant in the operating room, the angles of the joints of the hand controller 220 of the control device 2, the angle of the guide tube (trocar) in the patient's abdomen, and the entry and exit and movement of the assistant in the instrument storeroom or the corridor according to the progress of the procedure. This allows the observer to further learn about their own and their team's actions during the procedure.

[0203] The information processing system 5 shown in FIG. 8 includes a storage device 310 and an information processing device 320. The storage device 310 stores a status log indicating the status of an operation target that changes when the operator operates the operation target of an operation device 2 for controlling the operation of an arm device 1 of a surgical robot 4 to which surgical instruments 121 to 124 are attached. The operation targets are, for example, an operation unit 221 (see FIG. 4), foot pedals 231 to 237 (see FIG. 5), and a viewer 211 (see FIG. 3). The information processing device 320 generates images 432a, 432b, 434, 435, and 440 (reconstructed images) that visualize the operator's operations on the operation targets based on the status log stored in the storage device 310, and provides a surgical image 400 in which an endoscopic image 401, an operating room image 402, and schematic images 403 and 404 (operation images) that indicate the operation of the surgical robot 4 or the surgical instruments 121 to 124 are associated with the reconstructed images.

[0204] According to this configuration, the observer can easily and accurately confirm the operation of the surgical robot 4 or the surgical instruments 121 to 124 and the operation of the operation target by referring to the operation image and the reconstructed image.

[0205] <Example of change> In the above embodiment, a schematic diagram of the pair of movable plates 221b is shown in image 432a (see FIG. 21) indicating the degree of movement of operation unit 221 by the operator's operation, and the degree of opening of the pair of movable plates 221b in this schematic diagram is drawn in accordance with the operation of operation unit 221. However, this is not limiting, and instead of the schematic diagram of the pair of movable plates 221b, a meter that schematically indicates the degree of opening of the pair of movable plates 221b may be displayed in image 432a.

[0206] Image 432a was an image that two-dimensionally and schematically showed a pair of movable plates 221b (operation objects) of operation unit 221, but instead may be an image that three-dimensionally and schematically shows movable plates 221b (operation objects).

[0207] In the above embodiment, the image 434 shown in FIG. 21 indicates the depression state of the foot pedals corresponding to the left and right hand controllers 220 as ON or OFF. However, this is not limited to this. An output value indicating the strength of energy applied to the corresponding electric scalpel when the foot pedal is ON may be displayed. In this case, the strength of energy applied to the electric scalpel when the foot pedal is ON may be changeable during surgery by changing the settings of the energy generator. For example, if the strength of energy applied to the electric scalpel when the foot pedal is ON is set to V1, the height of the graph in image 434 is V1 while the foot pedal is depressed. If the strength of energy applied to the electric scalpel when the foot pedal is ON is set to V2 (>V1), the height of the graph in image 434 is V2 while the foot pedal is depressed. When the foot pedal is not depressed, the height of the graph in image 434 is 0.

[0208] In the above embodiment, image 434 shown in FIG. 21 shows the depression state of foot pedals 234 to 237 based on the status log, but in addition, the ON or OFF state of the voltage actually applied to the forceps may also be shown based on the operation log.

[0209] In the above embodiment, the endoscopic image 401, the operating room image 402, and the schematic images 403 and 404 are images that change continuously over time, i.e., are moving images, and the display of other parts of the surgical image 400 also changes continuously over time. However, this is not limiting, and each part of the surgical image 400 may be a plurality of still images (frame-by-frame moving images) that change at predetermined time intervals (e.g., every few seconds). Furthermore, even when repeated playback is performed using the loop playback button 512 in FIG. 25, each part of the surgical image 400 may be a frame-by-frame moving image.

[0210] In the above embodiment, the endoscopic image 401, the operating room image 402, and the schematic images 403 and 404 (operation images) are synchronized with the images 432a, 432b, 434, and 435 (reconstructed images). However, as long as the operation images and the reconstructed images are associated with each other, they need not be synchronized with each other. For example, the reconstructed images may be displayed with a slight delay compared to the operation images. In other words, the time at which the reconstructed images are to be displayed may be delayed from the time at which the operation images are to be displayed. The delay time of the reconstructed images may be adjustable to make it easier for the observer to learn. This allows the observer to move their line of sight to the reconstructed image after viewing the operation image and confirm in the reconstructed image the operation they previously viewed in the operation image.

[0211] Similarly, in the modification of the above embodiment, images 405-408 (see FIG. 32) are synchronized with the motion images and reconstructed images, but they may not be synchronized with the motion images and / or reconstructed images. Also, images 405-408 do not need to be moving images for the entire period of the motion images and / or reconstructed images, and may be displayed only for a predetermined period.

[0212] In the above embodiment, the screenshot 502 corresponding to the surgical image 400 is saved in the memory unit 322 of the information processing device 320 by using the bookmark insertion button 425, but this is not limiting and the screenshot 502 may be saved in the memory unit 342 of the observation terminal 340.

[0213] In the above embodiment, the operating room images, endoscopic images, operation logs, and status logs stored in the storage device 310 are transmitted to the information processing device 320 in real time during surgery, but they may also be transmitted after surgery.

[0214] In the above embodiment, the surgical image 400 includes images 401-404, images 432a-435, etc. However, the surgical image 400 may include at least one of the images 401-404 (operation images) and at least one of the images 432a, 432b, 434, 435, and 440 (reconstructed images that visualize the operator's operations). In this case, too, the observer can easily and accurately confirm the operation of the surgical robot 4 or the surgical instruments 121-124 and the operation of the target to be operated by referring to the operation image and the reconstructed image.

[0215] In the above embodiment, the status log and the operation log are configured with current values ​​acquired at predetermined time intervals, but this is not limited to this. For example, each time an encoder or a sensor outputs a value or signal, the control unit 131 or the control unit 251 may acquire the value or signal as a current value, and at least one of the status log and the operation log may be configured with these current values.

[0216] In the above embodiment, the information display area 430 of the surgical image 400 displays an operation information tab and a setting information tab, but may also display an error information tab.

[0217] FIG. 33 is a diagram showing the surgical image 400 when the error information tab is selected.

[0218] As shown in FIG. 33, when the error information tab is selected, error information 600 is displayed according to the error occurrence history. Furthermore, when one of the error information 600 is selected (in the example of FIG. 33, the fourth (bottom row) error information 600 is selected), detailed information 600a including the error message for the selected error and the date and time of the error occurrence is displayed. Furthermore, an attention mark 424 corresponding to the selected error information 600 is highlighted in the video control area 420. Furthermore, by moving the playback position mark 422 to the position of the attention mark 424, a still image 403a of the device that is the target of the error (in this example, the arm device 1) at the time the error occurred is displayed. The still image 403a is an image of the schematic image 403 (video) at the time the error occurred. This allows the observer to easily understand when and what kind of error occurred, and what operating state the device that is the target of the error was in at the time the error occurred.

[0219] 33, the observer can move the playback position mark 422 to the position of the attention mark 424, and then operate the operation information tab to display the same information as in FIG. 21 in the information display area 430. This allows the observer to view the still image 403a at the time the error occurred while also referring to a reconstructed image that visualizes the operator's operation at that time, thereby enabling the observer to understand in detail the operator's operation at the time the error occurred.

[0220] The embodiments of the present invention can be modified in various ways as appropriate within the scope of the technical idea defined in the claims. [Explanation of symbols]

[0221] 1 Arm device 2 Control device 4. Surgical robots 5. Information Processing Systems 121 Surgical instruments (forceps) 122 Surgical instruments 123 Surgical instruments (forceps, electrical instruments) 124 Surgical instruments (forceps) 140 Operating unit 151~154 Sensors 211 Viewer (operation target, display unit) 212 Head Sensor (Sensor) 220 Hand Controller 221 Operation unit (operation target) 221b Movable plate (operation target) 222~226 Movable part (second operation object) 231~233 Foot pedal (operation target) 234-237 Foot pedals (operation objects, operators) 261 Encoder (Status Detector) 301 Operating Room Camera (Camera) 310 Storage device 320 Information Processing Devices 400 surgical images 401 Endoscopic images (motion images) 402 Operating Room Images (Motion Images, Images) 403, 404 Schematic image (operation image) 405~408 images 432a Image (reconstructed image, schematic image, first image) 432b Image (reconstructed image, first image) 433a, 433b images (third reconstructed images) 434 Image (Reconstructed image, 2nd image) 435 images (reconstructed image, second reconstructed image, second image) 440 images (reconstructed images) 450 images (information) 460 Images (information) 502 Screenshots (Surgery Images)

Claims

1. A method for providing surgical images using a surgical robot, in which an information processing device generates and provides images of a surgery performed by a surgical robot having an arm device to which a surgical instrument is attached and an operation device operated by an operator to drive the arm device, comprising: acquiring a status log indicating a status of an operation object that changes as the operator operates the operation object of the operation device; generating a reconstructed image that visualizes a change over time in the operation of the operator with respect to the operation object based on the status log; A method for providing surgical images using a surgical robot, which provides a surgical image that associates the reconstructed image with an operational image showing the operation of the surgical robot or the surgical instrument based on the operator's operation on the operation object.

2. the operation target is moved in response to an operation by the operator, the status log includes information based on an output of a status detector that detects a status of the operation target; A method for providing surgical images using the surgical robot according to claim 1.

3. the operation target is moved in response to an operation by the operator, the reconstructed image includes a schematic image that schematically shows a degree of movement of the operation target due to the operation of the operator, A method for providing surgical images using the surgical robot according to claim 1.

4. the schematic image is an image that schematically depicts the operation target on a plane on which the operation target moves; A method for providing surgical images using the surgical robot according to claim 3.

5. the operation target is moved in response to an operation by the operator, the reconstructed image includes an image that numerically indicates a degree of movement of the operation target due to the operation of the operator. A method for providing surgical images using the surgical robot according to claim 1.

6. the surgical instrument comprises an electrical instrument; the operation target includes an operator that switches between energizing and deenergizing the electrical appliance attached to the arm device, the status log includes information indicating switching between energization and de-energization by the operator; the reconstructed image includes an image that schematically shows a period during which the operator was operated; A method for providing surgical images using the surgical robot according to claim 1.

7. the operation object includes a display unit on which the operator observes an endoscopic image, the status log includes information based on an output of a sensor that detects the operator's approach to the display unit; the reconstructed image includes an image that schematically shows a duration and an interruption period of the operator's approach to the display unit; A method for providing surgical images using the surgical robot according to claim 1.

8. the reconstructed image includes an image that schematically shows an operation timing for the operation target, A method for providing surgical images using the surgical robot according to claim 1.

9. The method further includes a step of generating a second reconstructed image that schematically shows a state of preparation for operation of the operation object based on an output from a sensor that detects that a part of the operator is present within a predetermined range for operating the operation object; In the surgical image, the operational image and the second reconstructed image are associated with each other. A method for providing surgical images using the surgical robot according to claim 1.

10. acquiring an operation log of a motion unit of the arm device driven by an operation on the operation object; generating a third reconstructed image that visualizes the driving state of the operation unit based on the operation log, In the surgical image, the operational image and the third reconstructed image are associated with each other. A method for providing surgical images using the surgical robot according to claim 1.

11. the operation device includes the operation object and a second operation object that moves in conjunction with the operator touching and operating the operation object, the operation image includes an image visualizing overall movements of the operation object and the second operation object based on the status log; A method for providing surgical images using the surgical robot according to claim 1.

12. The method of claim 11, further comprising providing information indicating which portions of the motion image have already been viewed during the entire duration from the start to the end of the motion image and the number of times they have been viewed. A method for providing surgical images using the surgical robot according to claim 1.

13. the surgical instrument includes a forceps; the operation object is an operation unit for opening and closing the tip of the forceps attached to the arm device, A method for providing surgical images using the surgical robot according to claim 1.

14. and providing information indicating whether the forceps are attached or detached in time series based on an output of a sensor that detects attachment or detachment of the forceps to the arm device. A method for providing surgical images using the surgical robot according to claim 13.

15. further comprising the step of providing information indicating the type of the surgical instrument attached to the arm device. A method for providing surgical images using the surgical robot according to claim 1.

16. The operation object is a foot pedal of the operation device. A method for providing surgical images using the surgical robot according to claim 1.

17. The foot pedals include a foot pedal for the right foot and a foot pedal for the left foot. A method for providing surgical images using the surgical robot according to claim 16.

18. the surgical instrument includes a forceps; the operation object includes an operation unit of a hand controller and a foot pedal for driving the arm device to operate the forceps, the reconstructed image includes a first image that visualizes a change in an operation on the operation unit over time, and a second image that visualizes a change in an operation on the foot pedal over time, In the surgical images, the first image is disposed above the second image. A method for providing surgical images using the surgical robot according to claim 1.

19. In the step of providing the surgical images, the motion images and the reconstructed images are varied over time. A method for providing surgical images using the surgical robot according to claim 1.

20. and synchronizing the operational image and the reconstructed image with each other in the step of providing the surgical image. A method for providing surgical images using the surgical robot according to claim 19.

21. receiving a designation of an arbitrary time point during display of the surgical image that changes over time; and automatically saving the surgical image at a specified time point. A method for providing surgical images using the surgical robot according to claim 19.

22. receiving a designation of an arbitrary time point during display of the surgical image that changes over time; receiving a playback instruction at a specified point in time; and reproducing the surgical images for a predetermined period before and after the specified time point based on the reproduction instruction. A method for providing surgical images using the surgical robot according to claim 19.

23. In the surgical images, the operation image is associated with an image obtained from a camera photographing at least one of an operating room in which the arm device is installed, the operating device, the patient's abdomen, an instrument storeroom, and a corridor leading to the operating room. A method for providing surgical images using the surgical robot according to claim 1.

24. a storage device that stores a status log indicating a status of an operation target of an operation device that an operator operates to drive an arm device of a surgical robot to which a surgical instrument is attached, the status log indicating a status of the operation target that changes as the operator operates the operation target of the operation device; an information processing device that generates a reconstructed image that visualizes changes over time in the operator's operation of the operation object based on the status log stored in the storage device, and provides a surgical image that associates the reconstructed image with an operational image that shows the operation of the surgical robot or the surgical instrument based on the operator's operation of the operation object.

Images