Information processing device, information processing system, and information processing program

Abstract

[Problem] To provide an information processing device which can switch opposite parties among an operation room terminal, an instruction room terminal, and a participation room terminal that are capable of performing mutual communication of information including a video, speech, and an annotation. [Solution] A server 11 comprises a control circuit 17 that is connected to an operation room terminal 13, an instruction room terminal 14, and a participation room terminal 43 through a network 12, and that executes processes. The control circuit 17 executes a process on an acquired video, a process on acquired speech, a process on an acquired annotation, and a process for switching information transmission states. The control circuit 17 is configured to be activated by switching modes among a first mode, a second mode, and a third mode different in information transmission state.

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing system, and an information processing program connected to an operating room terminal, an instruction room terminal, and a participant room terminal via a network. [Background technology]

[0002] Medical professionals certified by the medical association videotape surgical procedures and record the footage on a recording device, such as a hard disk drive (HDD) or a digital versatile disc (DVD). Medical professionals certified by the medical association are called expert doctors. Young doctors can view the surgical footage recorded on these recording devices to hone their surgical skills. An example of such technology is described in Patent Document 1.

[0003] Patent Document 1 states: "An imaging means configured to be able to control imaging conditions and to image the affected area; an acquisition means for acquiring information about the imaging conditions set in the imaging means; a control means for controlling communication conditions for transmitting image data via a network according to the imaging results of the affected area by the imaging means, based on the imaging conditions; A medical imaging system comprising: is stated. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-40874 Summary of the Invention [Problem to be solved by the invention]

[0005] The inventors of the present application recognized the problem that the technology described in Patent Document 1 does not allow switching between the operating room terminal, the instruction room terminal, and the participant room terminal with which information including video, audio, and annotations can be mutually communicated.

[0006] The object of the present disclosure is to provide an information processing device, an information processing system, and an information processing program that can switch between the operating room terminal, the instruction room terminal, and the participating room terminal with which information including video, audio, and annotations can be mutually communicated. [Means for solving the problem]

[0007] The information processing device of the present disclosure is an information processing device having a communication unit connected to each of an operating room terminal provided in an operating room and having a display unit for displaying video and annotations, a microphone for acquiring audio from within the operating room, and a speaker for outputting audio acquired from outside the operating room, an instruction room terminal provided in an instruction room separated from the operating room and having a display unit for displaying video and annotations, a microphone for acquiring audio from within the instruction room, and a speaker for outputting audio acquired from outside the instruction room, and a participation room terminal provided in a participation room separated from the operating room and the instruction room and having a display unit for displaying video and annotations, a microphone for acquiring audio from within the participation room, and a speaker for outputting audio acquired from outside the participation room, a storage unit in which a non-transitory program is stored, and a control circuit connected to the communication unit and the storage unit and which executes processing when the non-transitory program connected to the storage unit is started, and the control circuit processes video acquired from the operating room terminal, the instruction room terminal, and the participating room terminal, and and a process for switching a transmission state of information including video, audio, and annotations between the operating room terminal, the operating room terminal, the participating room terminal, and the communication unit, and the control circuit switches between a first mode, a second mode, and a third mode in which the transmission state of information differs, and when the first mode is activated, audio can be communicated bidirectionally between the operating room terminal and the operating room terminal, and audio can be transmitted unidirectionally from the operating room terminal to the participating room terminal ... the control circuit switches between a first mode, a second mode, and a third mode in which the transmission state of information differs, and the control circuit switches between a first mode, a second mode, and a third mode in which the transmission state of information differs, and when the control circuit switches between a first mode, a second mode, and a third mode in which the transmission state of information differs, and the control circuit switches between a first mode, a second mode, and a third mode in which the transmission state of information differs, and the control circuit switches between a first mode, a second mode, and a third mode in which the transmission state of information differs, and the control circuit switches between a first mode, a second mode, and a third mode in which the transmission When the second mode is activated, audio can be transmitted in two directions between the participating room terminal and the instruction room terminal, audio can be transmitted in one direction from the instruction room terminal to the instruction room terminal, audio cannot be transmitted between the participating room terminal and the operation room terminal, and annotations can be transmitted in one direction between the instruction room terminal and the instruction room terminal, and audio can be transmitted in one direction from the operation room terminal to the instruction room terminal, and audio cannot be communicated between the participating room terminal and the operation room terminal, and annotations can be communicated in two directions between the instruction room terminal and the participating room terminal, andAn information processing device in which annotations cannot be communicated between the participating room terminal and the operating room terminal, and annotations cannot be communicated between the instructor room terminal and the operating room terminal, and when the third mode is activated, audio can be communicated bidirectionally between the participating room terminal and the instructor room terminal, audio can be communicated bidirectionally between the participating room terminal and the operating room terminal, audio can be communicated bidirectionally between the operating room terminal and the instructor room terminal, audio can be communicated bidirectionally between the operating room terminal and the instructor room terminal, annotations can be communicated bidirectionally between the participating room terminal and the operating room terminal, and annotations can be communicated bidirectionally between the operating room terminal and the instructor room terminal, and when any of the first mode, the second mode, or the third mode is activated, video can be communicated bidirectionally between the operating room terminal and the instructor room terminal, video can be communicated bidirectionally between the operating room terminal and the participating room terminal, and video can be communicated bidirectionally between the instructor room terminal and the participating room terminal. [Effects of the Invention]

[0008] According to the present disclosure, during surgery, it is possible to switch between the operating room terminal, the instructor room terminal, and the participant room terminal, with which information including video, audio, and annotations can be mutually communicated. For example, a surgeon using an operating room terminal can reduce stress caused by the situation in the operating room being viewed from the outside, i.e., by an instructor using an instructor room terminal and participants using participant room terminals. In addition, the instructor room terminal can provide appropriate guidance and annotations to the operating room terminal depending on the surgery being performed by the surgeon. [Brief explanation of the drawings]

[0009] [Figure 1A] 1 is a block diagram showing a first specific example of a remote instruction system, which is an example of an information processing system of the present disclosure. [Figure 1B] FIG. 2 is a block diagram showing an example of a server included in the remote instruction system. [Figure 1C] FIG. 1 is a block diagram showing an example of an operating room included in a remote teaching system. [Figure 1D]FIG. 1 is a block diagram showing an example of a teaching room included in a remote teaching system. [Figure 1E] FIG. 1 is a block diagram showing an example of a participation room included in a remote instruction system. [Figure 2A] FIG. 2A is a schematic diagram showing an example of a display on a monitor installed in an operating room. [Figure 2B] FIG. 2B is a schematic diagram showing an example of a display on a monitor installed in a teaching room. [Figure 3] FIG. 10 is a block diagram showing a second specific example of the remote instruction system of the present disclosure. [Figure 4] 1 is a flowchart illustrating an example of a remote instruction method performed by the remote instruction system of the present disclosure. [Figure 5] 10 is an example of a screen displayed on the monitor of the teaching room terminal. [Figure 6] FIG. 2 is a schematic diagram illustrating a mode switching function according to the present disclosure. [Figure 7] FIG. 10 is a conceptual diagram showing an API when a proctor mode is selected. [Figure 8] FIG. 10 is a conceptual diagram showing an API when a demo mode is selected. [Figure 9] FIG. 10 is a conceptual diagram showing an API when full mode is selected. [Figure 10] 10 is a conceptual diagram showing a sequence process performed when an icon on a teaching room terminal is operated in the present disclosure. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0010] (Summary of the Disclosure) In the information processing system disclosed herein, an operating room terminal used by a subject receiving medical, specifically surgical, instruction and an instruction room terminal used by an instructor instructing the subject are connected via a network. A remote instruction system is disclosed as an example of an information processing system. A remote instruction program is disclosed as an example of an information processing program. The remote instruction system disclosed herein is primarily intended for use in so-called remote proctoring, in which a surgeon performs surgery on a patient in an operating room and provides real-time surgical instruction from a location remote from the operating room. Images used in the remote instruction system include both 2D and 3D images. The images include annotations and pointers. The 3D images are, for example, stereotypes. The remote instruction system disclosed herein includes the following use examples.

[0011] The remote instruction system disclosed herein can be used in a first scenario, for example, real-time instruction in a "one-on-one" situation. Here, the first "one" is a young surgeon, and the second "one" is a supervising surgeon. In order to obtain certification from various academic societies, the young surgeon submits video footage taken during surgery as proof of skill to the various academic societies. When creating the video footage to submit to the various academic societies, the young surgeon can use the remote instruction system during actual surgery and receive remote instruction from a supervising surgeon. The young surgeon can use the video footage actually taken during surgery as proof of skill.

[0012] The remote instruction system of the present disclosure can be used in a second scenario, for example, a video clinic in a "1 vs. n" situation. Here, "1" is a junior surgeon. "n" are instructors other than the junior surgeon, and "n" is an integer equal to or greater than 1. For example, a junior surgeon submits video footage as proof of skill to become a medical specialist certified by an academic association. The remote instruction system records a video of the junior surgeon performing surgery without guidance from an instructor, and shares the video with instructors, colleagues, surgeons affiliated with other medical facilities, etc., allowing the junior surgeon to receive guidance and hold discussions.

[0013] The remote guidance system disclosed herein can be used in a third scenario, for example, a live open conference in a "1 vs. 1 vs. n" situation. Here, the first "1" is the surgeon, the second "1" is the supervising surgeon, and the "n" are the participants. The remote guidance system provides information, including video of the surgery, to a conference, public lecture, or other venue while the surgeon is performing the surgery. The remote guidance system has a function for the instructor to provide guidance to the surgeon and display annotations and pointers during the surgeon's surgery. The remote guidance system disclosed herein has a function for providing guidance, annotations, and pointers to the surgeon and participants. The remote guidance system disclosed herein has a function for not transmitting video and audio, including discussions between the supervising surgeon and participants, to the surgeon.

[0014] The remote instruction system of the present disclosure has a function that can be used in video-on-demand situations. The remote instruction system of the present disclosure has a function to automatically record and upload information including, for example, video obtained in a first scene, a second scene, and a third scene, instruction, annotations, and pointers. The remote instruction system of the present disclosure has a function to upload files including various information. The remote instruction system of the present disclosure has a function to manage uploaded videos. The remote instruction system of the present disclosure has a function to manage viewing of uploaded videos. The remote instruction system of the present disclosure has a function to manage sharing of uploaded videos.

[0015] The remote instruction system of the present disclosure has the function of live-streaming information including the video, instruction, annotations, and pointers obtained in the third scene in one direction to multiple users simultaneously.

[0016] This disclosure can be used for training specialists, resolving surgeon shortages, researching and learning advanced surgical cases, medical training in rural areas, and remote instruction by instructors. It can also be used when actually performing surgery at medical institutions. It is also effective for career paths for female surgeons and for providing medical services with minimal regional disparities.

[0017] Next, at least one specific example of a remote instruction system including the remote instruction device, the remote instruction method, and the remote instruction program of the present disclosure will be described with reference to the drawings.

[0018] (First specific example) In the first example, a patient undergoes surgery in an operating room, and surgical guidance is provided in real time from a location remote from the operating room. More specifically, the patient lies on an operating table, and surgical personnel manually perform the surgery. The first example is shown in Figures 1A, 1B, 1C, 1D, 1E, 2A, and 2B.

[0019] (Overall configuration) 1A includes a server 11, an operating room terminal 13, an instructor room terminal 14, and a participant room terminal 43. The server 11, the operating room terminal 13, the instructor room terminal 14, and the participant room terminal 43 are connected to each other via a network 12. The server 11 is also connected to a processing terminal 81 via the network 12. The network 12 includes the Internet, an intranet, a wireless LAN (Local Area Network), a mobile phone system network such as 4G or 5G, and the like.

[0020] The operating room terminal 13 and the instructor room terminal 14 can be indirectly connected via the server 11. Furthermore, once the operating room terminal 13 and the instructor room terminal 14 are authenticated by the server 11, the operating room terminal 13 and the instructor room terminal 14 can also perform direct bidirectional communication, bypassing the server 11. Note that FIG. 1A shows an example in which a single operating room terminal 13 is connected to the server 11, a single instructor room terminal 14 is connected to the server 11, and a single participating room terminal 43 is connected to the server 11; however, multiple operating room terminals 13 may be connected to the server 11. Furthermore, multiple instructor room terminals 14 may be connected to the server 11. Furthermore, multiple participating room terminals 43 may be connected to the server 11. The network 12 is established by at least one of wireless communication and wired communication. Note that wireless communication includes Wi-Fi (registered trademark).

[0021] In the network 12 and server 11, the communication circuits for signals including information such as video, images, audio, and data are constructed using a unique interface and compression transmission technology. The unique interface includes a signal processing encoder, a browser, and an application for connecting to the server 11 using an internet line. By compressing high-quality video, real-time two-way communication is possible between the instructor room terminal 14, operating room terminal 13, and participant room terminal 43.

[0022] The operating room terminal 13 is used by a subject receiving surgical instruction. The subjects include the surgeon, nurses, assistants, etc. The operating room terminal 13 is installed, for example, in an operating room 15. The instructor room terminal 14 is used by an instructor who provides surgical instruction to the subject. The instructor room terminal 14 is installed in an instructor room 16. The operating room 15 and the instructor room 16 are located, for example, in different locations. The different locations may be different countries, different prefectures, different cities, towns, or villages, different buildings, different rooms in the same building, etc. The participant room terminal 43 is used by a trainee to view the situation in the operating room 15 and to study the instructor's instruction. The participant room terminal 43 is used in a location other than the operating room 15 and the instructor room 16. An example of the configuration of the server 11, the operating room 15, the instructor room 16, and the participant room terminal 43 is as follows:

[0023] (Server description) As shown in FIG. 1B , the server 11 includes a control circuit 17, an operation unit 18, a monitor 19, a communication unit 20, and a memory unit 22. The server 11 is installed in a location separate from the operating room 15 and the instructor room 16. The control circuit 17 includes a video processing unit 82, an audio processing unit 83, an annotation processing unit 84, a robot information processing unit 85, an auxiliary information processing unit 86, a switching unit 105, and a mode selection support unit 92. The control circuit 17 performs various processes and judgments based on various information acquired from the operating room terminal 13, the instructor room terminal 14, and the participant room terminal 43, and stores the processing results and judgment results in a database in the memory unit 22. The various information includes, for example, video, audio, annotations, pointers, folders, files, data, etc. The control circuit 17 may convert audio information into text in real time and interpret it into a foreign language such as English. The interpretation may be performed automatically using artificial intelligence technology such as deep learning.

[0024] The video processing unit 82 has a function to process video acquired from the operating room terminal 13, the instructor room terminal 14, and the participant room terminal 43, and video read from the memory unit 22. The audio processing unit 83 has a function to process audio acquired from the operating room terminal 13, the instructor room terminal 14, and the participant room terminal 43, and audio read from the memory unit 22. The annotation processing unit 84 has a function to process annotations acquired from the operating room terminal 13, the instructor room terminal 14, and the participant room terminal 43, and annotations read from the memory unit 22. The annotation processing unit 84 also has a function to associate information including the operation of the robot main body 48 acquired by the robot information processing unit 85 with annotation information generated in the instructor room terminal 14 and store the information in the memory unit 22.

[0025] The robot information processing unit 85 has the function of acquiring information, including the operation of the master control 49 and the operation of the robot main body 48, from the operating room terminal 13, and processing the acquired information. The auxiliary information processing unit 86 can generate auxiliary information with characteristics that change the medical fee depending on the type of mode activated by the switching unit 105. The medical fee will be described later. The communication unit 20 is connected to the network 12. Signals received by the communication unit 20 via the network 12 are input to the control circuit 17. Furthermore, signals output from the communication unit 20 are transmitted via the network 12 to the operating room terminal 13, the instruction room terminal 14, and the participating room terminal 43.

[0026] The operation unit 18 is constructed with at least one element of a touch switch, a display screen such as a liquid crystal panel, a keyboard, a mouse, etc. When the administrator of the server 11 operates the operation unit 18, a signal corresponding to the operation content is input to the control circuit 17. The monitor 19 has a display screen. The display screen of the monitor 19 is constructed with a touch panel, a liquid crystal panel, etc. The content displayed on the display screen of the monitor 19 is controlled by the control circuit 17. The display screen of the operation unit 18 and the display screen of the monitor 19 may be provided separately or may be integrated.

[0027] The memory unit 22 may be either an internal type provided inside the main body constituting the server 11 or an external type provided outside the main body. The external memory unit 22 may also be a hard disk, DVD, stick memory, etc. The memory unit 22 stores non-transitory programs, data, information, etc. used for processing and judgment, etc., performed by the control circuit 17. The learning data for machine learning stored in the memory unit 22 may use data from previous natural language processing. This allows foreign surgeons to participate and is expected to improve their surgical skills. The memory unit 22 stores applications launched by the processing, judgment, control, etc., performed by the control circuit 17, and various information, data, etc. used for processing, judgment, control, etc., performed by the control circuit 17. The memory unit 22 also stores various information, data, etc., to be sent to the operating room terminal 13, the training room terminal 14, and the participation room terminal 43. The memory unit 22 also stores evaluation criteria to be sent to the participation room terminal 43.

[0028] The mode selection support unit 92 has a function of supporting the mode selection operation performed on the teaching room terminal 14. The mode selection support unit 92 can perform the following support processing, for example.

[0029] (First support process) The mode selection support unit 92 stores the mode selected at the instruction room terminal 14 as data for each medical condition, each surgical content, each surgeon, and each participant in the memory unit 22. The mode selection support unit 92 also stores the number of instructions and the duration of the instructions provided from the instruction room terminal 14 to the operating room terminal 13 and the participant room terminal 43, as well as the number of annotations and the duration of the annotations, in the memory unit 22. Thereafter, when surgery is performed in the operating room 15, the mode selection support unit 92 recommends the mode in which the surgery proceeded most smoothly from the data stored in the memory unit 22 as an appropriate mode from the server 11 to the instruction room terminal 14, depending on the surgical content, case, surgeon, etc. The mode selection support unit 92 can determine that the mode selected when, for example, the time required from the start of surgery to the end of surgery was the shortest, the number of instructions given from the instruction room terminal 14 to the operating room terminal 13 during surgery and the time required were the shortest, and the number of annotations given and the time required were the shortest, is the ``mode in which the surgery proceeded most smoothly.''

[0030] (Second support process) The mode selection support unit 92 stores, as data in the memory unit 22, the relationship between the mode selected at the instruction room terminal 14 for each surgical content, medical condition, surgeon, participant, etc. Furthermore, the mode selection support unit 92 acquires, from the participation room terminal 43, an evaluation of the instruction and annotation sent from the instruction room terminal 14 to the participation room terminal 43, and stores the acquired evaluation in the memory unit 22. The evaluation acquired from the participation room terminal 43 will be described later. Then, at the time of surgery performed after the evaluation has been stored in the memory unit 22, the mode selection support unit 92 performs processing to recommend, from the server 11 to the instruction room terminal 14, the mode that was selected at the time the surgery that received the highest evaluation was performed, depending on the surgical content, case, surgeon, participant, etc.

[0031] (Third support process) Before the instructor operates the training room terminal 14 to select a mode, the mode selection support unit 92 performs a process of acquiring a desired mode from at least one of the operating room terminal 13 or the participant room terminal 43. Then, the mode selection support unit 92 performs a process of recommending the acquired desired mode to the training room terminal 14.

[0032] The instructor can select one of a plurality of modes, for example, the first to third modes, by operating the operation unit 18 of the training room terminal 14 in accordance with the first to third scenes described above. The instructor can also select one of the first to third modes based on information recommended by the mode selection support unit 92.

[0033] The switching unit 105 has a function of activating the selected mode. The first to third modes differ in the state of transmission of various information between the operating room terminal 13 and the instructor room terminal 14, the state of transmission of various information between the instructor room terminal 14 and the participant room terminal 43, and the state of transmission of various information between the operating room terminal 13 and the participant room terminal 43. The switching unit 105 includes a proctor mode circuit 120, a demo mode circuit 121, and a full mode circuit 122. Specific functions of the proctor mode circuit 120, the demo mode circuit 121, and the full mode circuit 122, as well as specific examples of the first to third modes, will be described later.

[0034] (Operating room description) As shown in FIG. 1C, an operating room 15 is equipped with an operating room terminal 13, an operating field camera 23, a first camera 24, a second camera 25, a monitor 27, a speaker 28, a microphone 29, and an operation unit 30. The operating field camera 23 includes an endoscope, a laparoscopic camera, and a robotic camera. The operating room terminal 13 is a computer used by a subject receiving instruction from an instructor. The subjects include the surgeon, trainee, nurse, surgical assistant, etc. The operating room terminal 13 controls and manages the monitor 27, the operating field camera 23, the first camera 24, the second camera 25, the microphone 29, the speaker 28, etc.

[0035] The operating room terminal 13 includes a control circuit 31, a memory unit 32, and a communication unit 33. The control circuit 31 is mainly composed of, for example, a video card (also called a graphics board) and includes an encoder and decoder for signal data conversion, an arithmetic circuit for various processes, comparisons, and calculations, and a memory for temporarily storing information. The operating room terminal 13 is connected to an operation unit 30, a monitor 27, an operating field camera 23, a first camera 24, a second camera 25, a microphone 29, and a speaker 28 so as to be able to transmit and receive signals. The communication unit 33 receives signals transmitted from the server 11 via the network 12. The signals received by the communication unit 33 are processed by the control circuit 31. Furthermore, signals output from the communication unit 33 are transmitted to the server 11 via the network 12. The signals input to and output from the communication unit 33 include information such as video, images, audio, data, annotations, and pointers.

[0036] The monitor 27 has a display screen constructed using a touch panel. The touch panel may be capacitive, resistive, or infrared. The monitor 27 and the operating room terminal 13 are connected via a communication line, such as HDMI (High-Definition Multimedia Interface, registered trademark) or USB (Universal Serial Bus). Information signals, including video, images, and audio, are transmitted bidirectionally between the monitor 27 and the operating room terminal 13. FIG. 2A shows an example of a display screen 34 of the monitor 27. The display screen 34 can display a display section 34A that displays video captured by the surgical field camera 23, a display section 34B that displays video captured by the first camera 24, a display section 34C that displays video captured by the second camera 25, and a display section 34D that displays video transmitted from the instruction room terminal 14. The display screen 34 can also display an operation menu. The display screen 34 is compatible with high-resolution video.

[0037] The operation unit 30 is configured with at least one element of a keyboard, a mouse, and a touch panel display screen. The operation unit 30 is connected to a control circuit 31 of the operating room terminal 13. When the subject operates the operation unit 30, a signal corresponding to the operation is input to the control circuit 31. The display screen of the operation unit 30 and the display screen 34 of the monitor 27 may be provided separately or may be integrated. The display screen 34 of the monitor 27 shown in FIG. 2A is an example in which the display screen is integrated with the display screen of the operation unit 30. By operating the operation unit 30, the surgeon, nurse, etc. can select a desired mode for each surgery, condition, surgical content, surgeon, or participant, and transmit the selected mode from the operating room terminal 13 to the server 11.

[0038] The storage unit 32 may be either an internal type provided inside the main body constituting the operating room terminal 13, or an external type provided outside the main body. The external storage unit 32 may be a hard disk, a DVD (Digital Versatile Disc), a stick memory, etc. The storage unit 32 stores non-transitory programs, data, information, etc. used for processing, judgment, control, etc. performed by the control circuit 31.

[0039] The endoscope (lapaoscope) 23 and the robot body 48 are medical devices intended to observe the inside of a patient, i.e., the human body, who is the surgical target. The surgical field camera 23 may have a structure and function capable of performing predetermined surgery and specimen collection in addition to the function of observing the inside of the human body. The surgical field camera 23 converts and outputs a signal from an image captured of the inside of the human body, specifically the surgical field. The surgical field camera 23 may also have a microphone. The surgical field camera 23 and the operating room terminal 13 are connected via a communication line, for example, an SDI (Serial Digital Interface), and digital video signals and digital audio signals are transmitted from the surgical field camera 23 to the operating room terminal 13.

[0040] The first camera 24 and the second camera 25 convert the video captured inside the operating room 15 into a signal and output it. The first camera 24 and the second camera 25 are cameras that overlook the inside of the operating room 15. The first camera 24, for example, captures video of the entire inside of the operating room 15. The capturing range (space) of the second camera 25 is narrower than the capturing range of the first camera 24. The first camera 24 and the second camera 25 differ in their installation locations, capturing locations, and capturing directions. The first camera 24 and the second camera 25 are connected in parallel to the operating room terminal 13, for example, via HDMI. The first camera 24 and the second camera 25 capture video of the entire team, including surgical assistants and nurses, i.e., the position of the surgeon, the position of the assistant, and the movement of the assistant's forceps.

[0041] The first camera 24 and the second camera 25 may be installed in any manner in the operating room 15, such as fixed, movable, or held by the hand of the subject. Although not shown, signals including electronic medical record information, signals including CT (Computed Tomography) images, signals including MRI (Magnetic Resonance Imaging) images, signals including echography (ultrasound examination) images, etc. may be input to the operating room terminal 13, and these signals may be replaced with the signals from the first camera 24 or the second camera 25.

[0042] The microphone 29 is provided in the operating room 15 and converts the sound in the operating room 15 into a signal and outputs it. The signal output from the microphone 29 is transmitted to the operating room terminal 13. The microphone 29 may be provided in either the first camera 24 or the second camera 25, or may be provided independently of the first camera 24 and the second camera 25.

[0043] The speaker 28 is connected to the operating room terminal 13. When a signal including audio is input from the instruction room terminal 14 to the operating room terminal 13 via the network 12, the speaker 28 can output the audio to the operating room 15. The speaker 28 may be provided in either the monitor 27 or the main body of the operating room terminal 13, or may be provided independently from the monitor 27 and the main body of the operating room terminal 13.

[0044] (Explanation of the Guidance Room) As shown in FIG. 1D , the teaching room 16 is equipped with a teaching room terminal 14, an operation unit 35, a monitor 36, a camera 37, a speaker 38, and a microphone 44. The teaching room terminal 14 is a computer used by a trainer who trains the trainee. Trainers include surgeons, university professors, authorities in the medical field, scholars, etc. The teaching room terminal 14 is connected to a server 11 via a network 12 and also to an operating room terminal 13 via the network 12. Bidirectional communication is possible between the teaching room terminal 14 and the server 11, between the teaching room terminal 14 and the operating room terminal 13, and between the server 11 and the teaching room terminal 14. The teaching room terminal 14 is connected to the operation unit 35, the monitor 36, the camera 37, the speaker 38, and the microphone 44.

[0045] The teaching room terminal 14 may be a fixed terminal, a mobile terminal, or a portable terminal, or may be a unit that combines multiple of these terminals. The teaching room terminal 14 has a control circuit 39, a memory unit 40, and a communication unit 41. The control circuit 39 is mainly composed of a video card (also called a graphics board), and has an encoder and decoder that convert signals into data, an arithmetic circuit that performs various processing, comparisons, and calculations, a memory that temporarily stores information, etc.

[0046] The storage unit 40 may be either an internal type provided inside the main body constituting the teaching room terminal 14, or an external type provided outside the main body. The external storage unit 40 may be a hard disk, a DVD (Digital Versatile Disc), a stick memory, etc. The storage unit 40 stores non-transient programs, data, information, etc. used for processing and judgment performed by the control circuit 39.

[0047] The instruction room terminal 14 is connected to the operation unit 35, monitor 36, camera 37, microphone 44, and speaker 38 so as to be able to send and receive signals. The communication unit 41 receives signals transmitted from the server 11 via the network 12, and receives signals transmitted from the operating room terminal 13 via the network 12. The signals received by the communication unit 41 are input to the control circuit 39. Furthermore, the signals output from the communication unit 41 are transmitted to the server 11 via the network 12. The signals input to and output from the communication unit 41 include information such as video, images, audio, and data.

[0048] As shown in FIG. 2B, the monitor 36 has a display screen 42 constructed using a touch panel. The touch panel may be capacitive, resistive, or infrared. The monitor 36 and the instruction room terminal 14 are connected via a communication line, e.g., HDMI. Information signals, including video, images, and audio, are exchanged between the monitor 36 and the instruction room terminal 14. The display screen 42 includes a display section 42A for displaying the video captured by the surgical field camera 23 transmitted via the network 12, a display section 42B for displaying the video captured by the first camera 24 transmitted via the network 12, a display section 42C for displaying the video captured by the second camera 25 transmitted via the network 12, a display section 42D for displaying the video captured by the camera 37, a dedicated menu 42E for adding annotations and pointers, and other operation menus. The display screen 42 supports high-resolution video. The monitor 36 can also display evaluations of instruction and annotations, desired modes, and other information transmitted from the server 11.

[0049] The operation unit 35 includes a keyboard, a mouse, a touch panel display screen, a touch pen, etc. The operation unit 35 is connected to the training room terminal 14. When the instructor operates the operation unit 35, a signal corresponding to the operation is input to the training room terminal 14. The display screen of the operation unit 35 and the display screen of the monitor 36 may be provided separately or may be integrated. FIG. 2B shows an example in which the display screen of the operation unit 35 and the display screen 42 of the monitor 36 are integrated. The instructor can use a touch pen or a mouse, etc. to operate the operation menu and dedicated menu 42E of the display screen 42. The instructor can also add annotations by drawing on the display unit 42A. The technical meaning of annotation will be described later.

[0050] The camera 37 converts the video captured inside the teaching room 16 into a signal and outputs it. The camera 37 is connected to the teaching room terminal 14, for example, via HDMI. The camera 37 may be installed in the teaching room in any manner, such as a fixed type, a movable type, or a type held by a staff member.

[0051] The microphone 44 is provided in the teaching room 16 and converts the sound from the teaching room 16 into a signal and outputs it. The signal output from the microphone 44 is transmitted to the teaching room terminal 14. The microphone 44 may be provided in the camera 37 or may be provided independently of the camera 37.

[0052] The speaker 38 is connected to the teaching room terminal 14. When a signal including audio is output from the operating room terminal 13 and input to the teaching room terminal 14 via the server 11, the speaker 38 can output the audio from the operating room 15 in the teaching room 16. The speaker 38 may be provided on either the monitor 36 or the main body of the teaching room terminal 14, or may be provided independently from the monitor 36 and the main body of the teaching room terminal 14. The audio from the speaker 38 can be switched by operating the operation unit 35.

[0053] (Explanation of the participant room terminal) As shown in FIG. 1E, the participant room terminal 43 is used in the participant room 116. The participant room terminal 43 is a computer used by participants, such as trainees or young surgeons studying general medical procedures. The medical procedures may occur during or after surgery. As shown in FIG. 1A, the participant room terminal 43 is connected to the operating room terminal 13 and the instructor room terminal 14 via the server 11. The participant room terminal 43 can transmit signals containing information to the operating room terminal 13 and can receive signals containing information from the operating room terminal 13. The participant room terminal 43 can transmit signals containing information to the instructor room terminal 14 and can receive signals containing information from the instructor room terminal 14. The participant room terminal 43 may be a fixed terminal, a mobile terminal, or a portable terminal, or may be a unit combining multiple terminals.

[0054] The participation room terminal 43 has a control circuit 139, a memory unit 140, and a communication unit 141. The control circuit 139 has an arithmetic circuit that performs various processes, comparisons, and calculations, a memory that temporarily stores information, etc. The memory unit 140 stores non-temporary programs, data, information, etc. used for the processes and judgments performed by the control circuit 139.

[0055] The participation room terminal 43 is connected to the operation unit 135, monitor 136, camera 137, microphone 144, and speaker 138 so as to be able to send and receive signals. The communication unit 141 receives signals transmitted from the server 11 via the network 12. The signals received by the communication unit 141 are input to the control circuit 139. Furthermore, the signals output from the communication unit 141 are transmitted to the server 11 via the network 12. The signals input to and output from the communication unit 141 include various types of information, such as video, images, audio, data, folders, files, etc.

[0056] The monitor 136 creates a display screen using a touch panel. The touch panel may be of any of the following types: capacitive, resistive, or infrared. The monitor 136 and the participation room terminal 43 are connected via a communication line, for example, HDMI. Signals containing various information are exchanged between the monitor 136 and the participation room terminal 43.

[0057] The operation unit 135 includes a keyboard, a mouse, a touch panel display screen, a touch pen, etc. The operation unit 135 is connected to the participation room terminal 43. When a participant operates the operation unit 135, a signal corresponding to the operation is input to the participation room terminal 43. The display screen of the operation unit 135 and the display screen of the monitor 136 may be provided separately or may be integrated. The participation room terminal 43 acquires information including evaluation criteria necessary for evaluation from the server 11. By operating the operation unit 135, participants can send evaluations of the instruction and annotations acquired from the instruction room terminal 14 to the server 11. This evaluation can be sent for each surgery, each surgical content, each medical condition, each surgeon, or each participant. The evaluation criteria may be any of a predetermined graded evaluation, an evaluation using predetermined fixed phrases, an evaluation independently set by the participant, etc. Furthermore, participants can operate the operation unit 135 to transmit to the server 11 their desired mode for each surgery, each medical condition, each surgical content, each surgeon, and each participant.

[0058] The camera 137 converts video captured inside the participation room 116 into a signal and outputs it. The camera 137 is connected to the participation room terminal 43, for example, via HDMI. The camera 137 may be installed in the instruction room in any manner, such as a fixed type, a movable type, or a type held by a staff member. The microphone 144 is installed in the participation room 116 and captures audio from the participation room 116. The audio captured by the microphone 144 is processed by the participation room terminal 43. The speaker 138 is connected to the participation room terminal 43. A signal including audio is output from the participation room terminal 43, and the speaker 138 outputs the audio in the participation room 116. The speaker 138 may be installed in either the monitor 136 or the main body of the participation room terminal 43, or may be installed independently of the monitor 136 and the main body of the participation room terminal 43.

[0059] Furthermore, the participation room terminal 43 transmits a signal including information about the participation room terminal 43 to the server 11. The information about the participation room terminal 43 includes an identification number unique to the participation room terminal 43, the participant's password, the participant's authentication ID, etc. The server 11 processes the identification number unique to the participation room terminal 43, the participant's password, the participant's authentication ID, etc. Once the server 11 processes the identification number unique to the participation room terminal 43, the participant's password, the participant's authentication ID, etc., mutual communication of information can be carried out between the server 11 and the participation room terminal 43.

[0060] (Description of processing terminal) The processing terminal 81 is an example of a computer that processes the instructor's medical fees when the instructor's actions are recognized as medical procedures. The processing terminal 81 has a control circuit 87, an operation unit 88, a display unit 89, a memory unit 90, and a communication unit 91. The control circuit 87 has a function of processing the operations of the operation unit 88 and a function of controlling the information displayed on the display unit 89. The operation unit 88, the display unit 89, the memory unit 90, and the communication unit 91 are connected to the control circuit 87. The display unit 89 is, for example, a monitor. The operation unit 88 includes a keyboard, a mouse, a touch panel, etc. The memory unit 90 stores software, programs, data, etc. used for the processing and judgments performed by the control circuit 87. The communication unit 91 is connected to the server 11 via the network 12.

[0061] In the present disclosure, the processing terminal 81 can process medical fees for the instructor based on the instructor's act of providing remote instruction to the operating room terminal 13 using the instruction room terminal 14. The processing terminal 81 can acquire auxiliary information generated by the auxiliary information processing unit 86 of the server 11 from the server 11. The processing terminal 81 may also have a function of creating a database of the attendance history of seminars attended by young surgeons in order to obtain certification.

[0062] (Functions and usage examples of the remote instruction system) With the patient lying on an operating table in operating room 15, surgery is performed on the patient, essentially manually by surgical personnel. Video and audio from operating room 15 are processed by operating room terminal 13, and as shown in Fig. 2A, the display screen 34 of monitor 27 can be divided into multiple display sections 34A, 34B, 34C, and 34D for display. The video from first camera 24 displayed on display section 34A is a 2D video and a 3D video.

[0063] The images displayed on the display units 34A, 34B, 34C, and 34D include the positions of the surgical personnel, their postures, hand movements, the placement positions of the instruments, the orientation of the instruments, the operating orientation of the instruments relative to the affected area of ​​the patient, and the operating positions of the instruments relative to the affected area of ​​the patient. The surgical personnel include the surgeon, nurses, assistants, etc. The instruments include an operating table, imaging devices, diagnostic imaging devices, measuring devices, endoscopes, forceps, tweezers, scissors, scalpels, needle holders, etc.

[0064] The size and display position of the display units 34A, 34B, 34C, and 34D can be set as desired by operating the operation menu. In addition, by operating the operation unit 30, images captured by the operative field camera 23, the first camera 24, and the second camera 25, and audio from the microphone 29 can be transmitted to the instruction room terminal 14 via the network 12.

[0065] The teaching room terminal 14 receives information including video, images, audio, and data. The display screen 42 of the monitor 36 can display display sections 42A, 42B, 42C, and 42D. The image on display section 42A is the same as the image on display section 34A, the image on display section 42B is the same as the image on display section 34B, the image on display section 42C is the same as the image on display section 34C, and the image on display section 42D is the same as the image on display section 34D. The images displayed on the display screen 42 are 2D and 3D images.

[0066] The presence, size, and display position of the display units 42A, 42B, 42C, and 42D can be set by operating the operation menu of the display screen 42 or by operating the operation unit 35. The speaker 38 in the instructor room 16 can output audio transmitted from the operating room 15. The instructor can then add and display annotations and pointers superimposed on the image displayed on the display unit 42A by touching the tip of a stylus to the display screen 42. Annotations include explanations, methods, procedures, positions, etc. related to the surgery, and can be represented primarily by a line drawing B1. A pointer without a drawing B1 may also be displayed.

[0067] The dedicated menu 42E is operated when displaying a drawing B1 annotation or pointer on the image displayed on the display screen. The dedicated menu 42E includes options such as a timer (delay display), line thickness, rectangle, circle, snapshot, recording, chat, whiteboard, hand raising, line priority control, line color, line thickness, type of standard pattern represented by line, partial erase, and full erase. The instructor can add drawing B1 to the image on the display unit 42A by adding a curved or straight drawing B1, adding an arrow drawing B1, or enclosing a specific location with a circular drawing B1 made of lines. In addition to displaying drawing B1 annotations and pointers on the image on the display unit 42A, the instructor can also provide audio instruction. Audio instruction is collected by a microphone 44.

[0068] The drawing B1 on the display unit 42A includes the incision location in the patient's affected area, the incision method, the incision direction, the position where the tissue is pulled with forceps, the pulling direction, how to create a peeled surface by pulling, various treatments, etc. The audio instruction can provide in real time an oral explanation corresponding to the drawing B1, an explanation of the technical meaning and reason, comprehensive knowledge and ideas including the patient's QOL (Quality of life), etc., and an explanation of important instruction content for the surgeon when performing surgery.

[0069] The instructor room terminal 14 transmits the drawing B1 provided on the display screen 42, the audio picked up by the microphone 44, and the video captured by the camera 37 to the operating room terminal 13 via the network 12. The operating room terminal 13 displays the annotations and pointers of the drawing B1 transmitted from the instructor room terminal 14 as drawing B2, superimposed on the video on the display unit 34A, as shown in FIG. 2A, for example. The operating room terminal 13 also outputs the audio instruction transmitted from the instructor room terminal 14 from the speaker 28. Therefore, while the surgery is being performed, the subject in the operating room 15 can receive instruction on the surgery from the instructor in real time.

[0070] The interface and compressed transmission technology in the remote instruction system 10 of the present disclosure enables two-way communication. Therefore, appropriate instruction and explanations regarding the procedure to be performed in the operating room 15 can be transmitted in real time from the instruction room 16. This allows the procedure in the operating room 15 to proceed smoothly. Specifically, it makes it possible to apply the knowledge and experience of an experienced surgeon to the instruction of a junior surgeon by applying it to actual cases.

[0071] Furthermore, when the operating room 15 and the instruction room 16 are located in different locations, an instructor living in a rural area can instruct the subject without visiting the location (e.g., an urban area) where the operating room 15 is located. Conversely, if the person receiving instruction in the operating room 15 lives in a medically underserved area such as a remote area or an island, and is a young surgeon who is responsible for medical care in that area, he or she can also receive instruction from the instructor in the instruction room 16.

[0072] Furthermore, the surgeon can add annotations by drawing on the monitor 27. The operation of the surgeon adding annotations on the monitor 27 is the same as the operation of the instructor adding annotations on the monitor 36. A signal including the annotations added on the monitor 27 is transmitted from the operating room terminal 13 to the server 11. The server 11 can transmit the annotations added on the monitor 27 to the instructor room terminal 14 and the participant room terminal 43.

[0073] Furthermore, participants can add annotations by drawing on the monitor 136. The operation by which participants add annotations on the monitor 136 is the same as the operation by which an instructor adds annotations on the monitor 136. A signal including the annotations added on the monitor 136 is transmitted from the participant room terminal 43 to the server 11. The server 11 can transmit the annotations added on the monitor 136 to the instructor room terminal 14 and the operating room terminal 13.

[0074] (Second specific example) The second example is an example in which surgery is performed on a patient in an operating room, and surgical guidance is given in real time from a location away from the operating room. More specifically, this is an example in which a surgeon uses a medical robot system to perform surgery on a patient while the patient is lying on an operating table.

[0075] (composition) As shown in FIG. 3 , the operating room 15 is equipped with a medical robot system 45, a first camera 24, a second camera 25, a microphone 29, an operating room terminal 13, a monitor 27, a speaker 28, and an operation unit 30. The medical robot system 45 includes a control unit (surgeon console) 46, a vision cart 47, a robot body (patient cart) 48, and the like. The control unit 46 includes a master control 49, a monitor 50, a microphone 51, a speaker 52, a chair, operation buttons, and the like. When the surgeon operates the master control 49, the display screens of the monitors 50 and 55 can be switched on and off, the display content of the display screens can be controlled, and the operation and functions of the robot body 48 can be controlled. The operating arm of the robot body 48 operates in accordance with the hand movements of the surgeon operating the master control 49. The medical robot system 45 is connected to the operating room terminal 13 for bidirectional communication.

[0076] The control unit 46 is operated by the surgeon sitting in a chair. The monitor 50 is capable of displaying 3D images, 2D images, cross-sectional images, etc. The cross-sectional images include CT (Computed Tomography) images and MRI (Magnetic Resonance Imaging) images acquired before surgery. In addition, audio data from the microphone 51 is transmitted to the operating room terminal 13. Furthermore, the speaker 52 outputs the audio transmitted from the operating room terminal 13. The robot main body 48 has multiple operating arms and an operating field camera 53. Surgical instruments are attached to each of the multiple operating arms.

[0077] When the surgeon sits in the chair and operates the master control 49 and various operation buttons, an operation signal is output from the operating unit 46 and the robot main body 48 functions. An image of the patient's affected area captured by the surgical field camera 53 is transmitted to the operating room terminal 13. In addition, an audio signal collected by the microphone 51 is transmitted to the operating room terminal 13. The audio transmitted from the operating room terminal 13 is output from the speaker 38 via the network 12.

[0078] The vision cart 47 has a monitor 55 and a computer 56. The computer 56 is connected to the operating room terminal 13, the control unit 46, and the robot body 48 so as to enable two-way communication, and is also connected to the monitor 55. The computer 56 processes signals transmitted from the operating room terminal 13, processes signals transmitted from the control unit 46, and processes signals transmitted from the robot body 48. The signals transmitted from the operating room terminal 13 to the computer 56 include information such as video and audio instructions, annotations of the drawing B1, etc.

[0079] The computer 56 also outputs a signal for the image to be displayed on the monitor 50 of the control unit 46, a signal for the audio to be output from the speaker 52 of the control unit 46, and a signal for the image to be displayed on the monitor 55 of the vision cart 47. The monitor 50 can display the image from the surgical field camera 53, the image from the first camera 24, and the image from the second camera 25. The monitor 50 can also display the image from the camera 37 transmitted from the server 11. The monitor 50 can switch between displaying 3D images and 2D images. Therefore, when the surgeon wants to view the 3D image, he or she does not need to move his or her line of sight.

[0080] It should be noted that the monitor 27 does not necessarily have to be provided in Fig. 3. Furthermore, the other configurations shown in Fig. 3 are the same as the other configurations shown in Fig. 1. Furthermore, the computer 56 processes signals including the operation details and operation details of the control unit 46, and sends control signals to the robot main body 48. Furthermore, the computer 56 processes signals including the operation details of the robot main body 48 and signals including the operation details and operation details of the control unit 46, and transmits the processing results to the server 11 via the operating room terminal 13.

[0081] (Example of operation of the second specific example) When surgery is performed using the medical robot system 45, the video and audio from the operating room 15 are transmitted to the instruction room terminal 14 via the operating room terminal 13 and the network 12. The video from the surgical field camera 53 is then displayed on the display unit 42A in FIG. 2B, the video from the first camera 24 is displayed on the display unit 42B, and the video from the second camera 25 is displayed on the display unit 42C. In the instruction room 16, the video on the display unit 42A is annotated with the drawing B1, and audio instruction is also provided.

[0082] A signal including the annotation of drawing B1 and a signal including audio instruction are output from the instruction room terminal 14, and the signal is transmitted to the medical robot system 45 via the network 12 and the operating room terminal 13. Therefore, when surgery is performed using the medical robot system 45, the drawing B1 transmitted from the instruction room terminal 14 can be displayed as drawing B2 on the display unit 34A of the monitor 50, as shown in FIG. 2A. Note that FIG. 2A shows an example in which the display units 34A, 34B, 34C, and 34D are displayed on the display screen 34 of the monitor 50. Furthermore, the audio transmitted from the instruction room terminal 14 is output from the speaker 52 of the control unit 46. Therefore, the second specific example can achieve the same effect as the first specific example.

[0083] Furthermore, the surgeon can add annotations by drawing on the monitors 50 and 55. The operation of the surgeon adding annotations on the monitors 50 and 55 is the same as the operation of the instructor adding annotations on the monitor 36. A signal including the annotations added on the monitors 50 and 55 is transmitted to the server 11 via the vision cart 47 and the operating room terminal 13. The server 11 can transmit the annotations added on the monitors 50 and 55 to the instructor room terminal 14 and the participant room terminal 43. As shown in FIG. 2A , the monitors 50 and 55 can display the same display screen 34 as the display screen 34 of the monitor 27. In other words, the display screen 34 of the monitors 50 and 55 can switch between displaying 2D images and 3D images. It is also possible to display only annotations on the display screen 34 of the monitors 50 and 55.

[0084] (Application example) The process of displaying the drawing B1 and pointer added to the 2D or 3D image on the display unit 42A of the monitor 36 in the instruction room 16 as the drawing B2 and pointer on the 3D image displayed on the monitors 27 and 55 in the operating room 15, e.g., coordinate conversion, may be performed by either the control circuit 39 of the instruction room terminal 14 or the control circuit 31 of the operating room terminal 13. The size and position of each of the display units 42A, 42B, 42C, and 42D displayed on the display screen 42 of the monitor 36 can be changed as desired by operating the operation unit 35 or the operation menu. For example, if the image of the operating room 15 is divided into multiple sections in chronological order at predetermined time intervals, the sizes of the display units 42A and 42B can be made different for each section.

[0085] Furthermore, an environment in which a single teaching room 16 is provided with a plurality of teaching room terminals 14, etc., or an environment in which a plurality of teaching rooms 16 are each provided with a teaching room terminal 14, etc., may be possible. In this case, the plurality of teaching rooms 16 may be located in different locations. Furthermore, among the plurality of teaching room terminals 14, the processing that can be performed on a specific teaching room terminal 14 may be different from the processing that can be performed on the other teaching room terminals 14. For example, processing such as playing, stopping, pausing, fast-forwarding, and fast-rewinding recorded video that is performed in the operating room 15 may be performed only on a specific teaching room terminal 14.

[0086] Furthermore, when images from the monitors 36 connected to the multiple teaching room terminals 14 are displayed on the monitors 27, 50, and 55 installed in the operating room 15, the multiple monitors 36 may be distinguished from one another by using an identification symbol or color coding for each different monitor 36. Furthermore, when audio is input to the microphone 44 connected to the teaching room terminal 14 to which the monitor 36 is connected, a display process may be performed such as surrounding the displayed image of the corresponding monitor 36 with a colored frame, thereby making it possible to identify any one of the multiple teaching room terminals 14. Furthermore, among the multiple teaching room terminals 14, it is sufficient to annotate the drawing B1 on the monitor 36 of any one of the teaching room terminals 14, and the monitors 36 of the other teaching room terminals 14 may only display the image.

[0087] The process of changing the color of drawings displayed on the monitors 27, 50, and 55 of the operating room 15 for each of the multiple monitors 36 can be performed by either the control circuit 17 of the server 11 or the control circuit 31 of the operating room terminal 13. Furthermore, when multiple monitors 36 are provided, the control circuit 17 of the server 11 can also perform a process of restricting the selection of the same drawing color on each of the monitors 36. Furthermore, the control circuit 31 of the operating room terminal 13 can also perform a process of displaying on the monitors 27, 50, and 55 of the operating room terminal 13 which of the multiple monitors 36 the drawing displayed on each of the monitors 27, 50, and 55 was created on.

[0088] Furthermore, the control circuit 17 of the server 11 can also perform processing to distribute in real time or in recorded form via the network 12 the images displayed on the monitors 27, 50, 55 in the operating room 15 and the audio output from the speakers 28, 52 in the operating room 15 to a participating room terminal 43 different from the operating room terminal 13 and the instructor room terminal 14. The participating room terminal 43 can receive 2D images, 3D images, and audio from the server and play back the images and audio. For convenience, the participating room terminal 43 does not transmit images or audio to the server 11. The participating room terminal 43 includes fixed terminals, mobile terminals, portable terminals, goggle terminals, projectors, etc.

[0089] 2B, a corner of the display screen 42 of the monitor 36 may display a display section T1 indicating the time when the annotation signal of the drawing B1 was transmitted from the teaching room terminal 14, a display section T2 indicating the time when the signal was transmitted to the operating room terminal 13 via the server 11 and then returned to the teaching room terminal 14 via the server 11, and a display section T3 indicating the time required for one-way signal transmission between the teaching room terminal 14 and the operating room terminal 13. Here, there is a time difference between the time when the annotation signal of the drawing B1 was transmitted from the teaching room terminal 14 and the time when the signal was transmitted to the operating room terminal 13 via the server 11 and then returned to the teaching room terminal 14 via the server 11.

[0090] Furthermore, the subject can send recorded video of surgical procedures from the operating room terminal 13 to the instructor room terminal 14 when no surgical procedures are being performed. In this case, the operating room terminal 13 does not need to be located in the operating room 15. The recorded video is stored in a folder in the memory unit 32 of the operating room terminal 13. The folders for storing the recorded video are stored in the memory unit 32, categorized according to the body part, case, surgeon, surgical procedure, date and time, operating room number (operation order), etc. The recorded video can then be sent from the operating room terminal 13 to the instructor room terminal 14 via the server 11, and the recorded video can be displayed on the display screen 42 of the monitor 36, allowing the subject to receive instruction from the instructor. In other words, the recorded video can be shared between both the operating room terminal 13 and the instructor room terminal 14.

[0091] The instructor annotates the drawing B1 on the display screen 42 of the monitor 36 and provides audio instruction, just as in a real-time surgery. In this case, an operation menu (operation buttons) for playing, stopping, pausing, fast-forwarding, rewinding, etc. the recorded video is displayed on the monitor 27 in the operating room 15. These operations may be performed on at least one of the monitors 27, 50, and 55 and the monitor 36. Furthermore, the operating room terminal 13 may have permission to grant permission to the instructor room terminal 14 to perform the above operations.

[0092] (Overview of control examples) Fig. 4 is a flowchart comprehensively illustrating the control examples described in the first specific example, the second specific example, and the application example. Fig. 4 shows steps including processing and judgment performed by the server 11, the operating room terminal 13, the teaching room terminal 14, and the participating room terminal 43. The server 11, the operating room terminal 13, the teaching room terminal 14, and the participating room terminal 43 can also execute each step in an order different from the order of the steps shown in Fig. 4.

[0093] In step S10, the operating room terminal 13 displays video of the surgery-related actions on at least one of the monitors 27, 50, and 55. In step S10, the operating room terminal 13 also transmits a signal including information about the operating room terminal 13 to the server 11. The information about the operating room terminal 13 includes an identification number unique to the operating room terminal 13, the password of the surgeon, the authentication ID of the surgeon, etc.

[0094] Meanwhile, in step S11, the training room terminal 14 displays the image captured by the camera 37 on the monitor 36. Also, in step S11, the training room terminal 14 transmits a signal including information about the training room terminal 14 to the server 11. The information about the training room terminal 14 includes an identification number unique to the training room terminal 14, the instructor's password, the instructor's authentication ID, etc. The server 11 receives signals transmitted from the operating room terminal 13 and signals transmitted from the training room terminal 14 in step S12, and authenticates the operating room terminal 13 and the training room terminal 14. In step S13, the server 11 transmits the signal received from the operating room terminal 13 to the training room terminal 14, and transmits the signal received from the training room terminal 14 to the operating room terminal 13.

[0095] In step S14, the operating room terminal 13 acquires the information transmitted from the server 11. In step S15, the teaching room terminal 14 acquires the information transmitted from the server 11. When the processing of step S14 is performed in the operating room terminal 13 and the processing of step S15 is performed in the teaching room terminal 14, direct two-way communication can be performed between the operating room terminal 13 and the teaching room terminal 14, which have been authenticated by the server 11, bypassing the server 11.

[0096] In step S18, the operating room terminal 13 displays the video sent from the teaching room terminal 14 on at least one of the monitors 27, 50, and 55, and transmits information including the video of the operating room 15 and information including the audio to the server 11. In step S19, the teaching room terminal 14 displays the video sent from the operating room terminal 13 on the monitor 36, and transmits information including the video of the teaching room 16 and information including the audio to the server 11. In step S20, the server 11 receives a signal sent from the operating room terminal 13 and stores it in the memory unit 22, and receives a signal sent from the teaching room terminal 14 and stores it in the memory unit 22. Furthermore, in step S21, the server 11 transmits a signal including the video and audio to the operating room terminal 13 and transmits a signal including the video and audio to the teaching room terminal 14.

[0097] The operating room terminal 13 receives the video sent from the server 11 and displays it on at least one of the monitors 27, 50, and 55 in step S22. The training room terminal 14 receives the video sent from the server 11 and displays it on the monitor 36 in step S23. The instructor adds an annotation, for example, the drawing B1 in FIG. 2(B), to the video displayed on the monitor 36 in step S24. The instructor also adds audio instruction from the microphone 44 in step S24. The training room terminal 14 then processes the signal of the added annotation of the drawing B1 in step S24 and transmits a signal corresponding to the added annotation of the drawing B1 and a signal corresponding to the audio instruction to the server 11 in step S25.

[0098] In step S26, the server 11 receives a signal from the instruction room terminal 14, and in step S27, transmits the signal to the operating room terminal 13. In step S28, the operating room terminal 13 receives the signal from the server 11. Then, in step S29, the operating room terminal 13 processes the received signal and displays the annotation of the drawing B1 on at least one of the monitors 27, 50, and 55, as shown in FIG. 2A. In addition, audio instruction is output from at least one of the speakers 28 and 52. Note that the operating room terminal 13 can add annotations in step S50 without obtaining information from the instruction room terminal 14.

[0099] Furthermore, in step S20, the server 11 receives a signal containing video and audio from the operating room 15 and a signal containing video and audio from the instructor room 16. Therefore, if the participant room terminal 43 is directly connected to the server 11 via the network 12 at the time of step S20, real-time distribution to the participant room terminal 43 is possible. The server 11 can also store the information received in step S20 in the memory unit 22. Therefore, if the participant room terminal 43 is connected to the server 11 after the server 11 performs the processing of step S20, the user of the participant room terminal 43 can use the information stored in the memory unit 22 of the server 11 on a video-on-demand basis. Note that, when surgery is not being performed in real time, the operating room terminal 13 may be connected to the server 11, and the information stored in the memory unit 22 may be used on a video-on-demand basis.

[0100] Furthermore, the second control can be selected when video from the operating room 15 is recorded in the storage unit 22 of the server 11 or when video from the operating room 15 is simultaneously live-streamed to all participating room terminals 43. Furthermore, although not shown in FIG. 4, it is also possible to select the first control and perform the processing of steps S18 to S28 between the operating room terminal 13 and the training room terminal 14, bypassing the server 11. In this case, steps S20, S21, S26, and S27 in FIG. 4 are skipped. Programs for switching between the first control and the second control are stored in the storage unit 32 of the operating room terminal 13, the storage unit 22 of the control circuit 17 of the server 11, and the storage unit 40 of the training room terminal 14, respectively.

[0101] 4, the server 11 can transmit various types of information to the participation room terminal 43. The various types of information that the server 11 transmits to the participation room terminal 43 include information acquired from the operating room terminal 13 and information acquired from the instruction room terminal 14. The participation room terminal 43 can receive various types of information from the server 11 in step S30. In step S30, the participation room terminal 43 can display the information received from the server 11, such as video and images, on the monitor 136. In step S30, the participation room terminal 43 can also output and display the information received from the server 11, such as audio, on the speaker 138.

[0102] Furthermore, in step S31, the participating room terminal 43 can transmit various types of information to the server 11. In step S26, the server 11 can receive various types of information from the participating room terminal 43 and store the processing results of the various types of information in the memory unit 22. In step S13, the server 11 can transmit various types of information acquired from the participating room terminal 43 to the operating room terminal 13 and the instruction room terminal 14.

[0103] 4 and 10, the server 11 stores various information acquired from the operating room terminal 13, the instructor room terminal 14, and the participant room terminal 43, and information sent to the operating room terminal 13, the instructor room terminal 14, and the participant room terminal 43, in a database in the storage unit 22, and performs a process of creating an updated history of the database. For example, necessary information, including the date and time of the surgery, the name of the person in charge of the surgery, the name of the surgeon, the names of assistants in the operating room 15, the name of the instructor, the names of the participants, the patient's disease name and case, the surgical site on the patient, the surgical procedure, and the equipment used, is compiled into a database to generate a briefing sheet, which is then stored in the storage unit 22. The necessary information stored in the storage unit 22 is distinguished by a distinction symbol assigned to each surgery.

[0104] This information is input into the operating room terminal 13 or the instructor room terminal 14 and transmitted from the operating room terminal 13 or the instructor room terminal 14 to the server 11. Furthermore, various types of information generated during surgery, such as video, images, instructions, annotations, audio, the movement of the robot body 48, and the operation details of the master control 49, are associated with the punctuation marks and stored in the storage unit 22. Therefore, in any of the operating room terminal 13, the instructor room terminal 14, and the participant room terminal 43, desired information can be searched for by entering the punctuation marks or part of the required information. Furthermore, information including the operation details of the master control 49 and the operation of the robot body 48 is associated with a time code that encodes the time information at which the annotation was generated and stored in the storage unit 22.

[0105] The control circuit 17 of the server 11 can also classify various types of information for different instructors, different surgeons, different diseases, different surgical procedures, different cases, and different parts of the patient's body, and store the information in a storage area such as the memory unit 22. The control circuit 17 of the server 11 also has a function of generating learning data for instruction based on the information and data stored in the memory unit 22. The generated learning data is stored in the memory unit 22. The learning data for instruction is used to provide instruction for performing surgery using the medical robot system 45.

[0106] Furthermore, the auxiliary information processing unit 86 of the control circuit 17 has the function of generating auxiliary information related to the instructor's medical fees based on the information and data stored in the memory unit 22. The auxiliary information for medical fees includes the date of the instructor's remote instruction, the elapsed time from the start to the end of the instruction, the content of the instruction, etc. The auxiliary information also includes points and other information calculated from the elapsed time from the start to the end of the instruction and the content of the instruction. The instruction includes the patient's disease name and case, the surgical site on the patient, the surgical procedure, etc., in the actual surgery, as well as lectures for trainees and young surgeons. The auxiliary information generated by the server 11 can be used by third parties. If remote instruction is covered by insurance, the young doctor's certification information generated by the server 11 can be used to calculate the medical fees received by the instructor.

[0107] (supplementary explanation) The remote instruction device, remote instruction method, and remote instruction program described in this embodiment are not limited to the above-mentioned embodiment. For example, each camera may be a 360-degree camera. The monitor may be provided in VR (Virtual Reality) goggles worn on the face or an HMD (Head Mounted Display) worn on the head. The mobile terminal and portable terminal include a smartphone and a tablet terminal.

[0108] The computer in the present disclosure includes a personal computer, a workstation, a host computer, a mainframe, a supercomputer, etc. The network connecting the server, the operating room terminal, and the teaching room terminal so as to enable two-way communication may be the Internet, an intranet, or an extranet.

[0109] The first location is a location where the operating room terminal 13 is installed, and the first location may be, for example, an operating room, lecture room, hall, space, room, etc. The second location is a location where the teaching room terminal 14 is installed, and the second location may be, for example, an operating room, lecture room, hall, waiting room, remote teaching room, space, room, etc. Furthermore, the second location is not limited to any one location, and may be multiple locations. The computer included in the server may be a single electrical or electronic component, or may be a unit including multiple electrical or electronic components. The electrical or electronic component includes a processor, a control circuit, and a module. The server may have either a single computer or multiple computers. Furthermore, the communication unit and the memory unit may be separate components or devices communicatively connected to the computer.

[0110] Furthermore, if the remote instruction system 10 is configured not to utilize both an environment in which the participation room terminal 43 connects to the server 11 to receive real-time distribution and an environment in which the participation room terminal 43 connects to the server 11 to use information stored in the memory unit 22 on a video-on-demand basis, steps S18, S19, S20, S21, S22, and S23 shown in FIG. 4 can be skipped.

[0111] (Third Specific Example) 1B, the switching unit 105 has a proctor mode circuit 120, a demo mode circuit 121, and a full mode circuit 122. The proctor mode circuit 120 activates the proctor mode (proctering mode), the demo mode circuit 121 activates the demo mode (demonstration mode), and the full mode circuit 122 activates the full mode circuit. The activation and deactivation of the proctor mode circuit 120, the demo mode circuit 121, and the full mode circuit 122 can be switched by operating the teaching room terminal 14.

[0112] An example of a screen displayed on the monitor 36 of the teaching room terminal 14 is shown in Figure 5. The monitor 36 displays a proctor mode icon 201, a demo mode icon 202, a full mode icon 203, a participant screen 204, a main screen 205, and a sub-screen 206. The proctor mode icon 201, the demo mode icon 202, and the full mode icon 203 generate a mode selection section. When the proctor mode icon 201, the demo mode icon 202, or the full mode icon 203 is operated, the auxiliary information processing unit 86 generates a mode activation request.

[0113] When the proctor mode icon 201 is activated, the proctor mode circuit 120 is activated. When the demo mode icon 202 is activated, the demo mode circuit 121 is activated. When the full mode icon 203 is activated, the full mode circuit 122 is activated.

[0114] The icon color of the activated mode, i.e., the active mode, is displayed darker than the icon color of the inactive mode. Also, the active mode is displayed with a design that shows that it is selected. This is to make the currently active mode clear.

[0115] The main screen 205 is a screen that displays videos and images acquired from the operating room 15. The main screen 205 mainly displays videos and images captured by the operating field camera 23, or images of the inside of the body after an incision with a scalpel. The main screen 205 may also display academic conference materials or explanatory materials stored in the memory unit 32 of the operating room terminal 13. The main screen 205 may also display materials stored on the cloud. The sub-screen 206 is a screen that displays videos and images captured by the first camera 24 and the second camera 25.

[0116] The proctor mode circuit 120 activates the proctor mode. The proctor mode is a mode selected, for example, when an instructor in the instructor room 16, such as an academically certified doctor (expert physician (surgeon)), provides surgical instruction to the surgeon in the operating room 15 and participants in the participant room 116. The demo mode circuit 121 activates the demo mode. The demo mode is a mode in which, for example, the instructor, an academically certified doctor (expert physician (surgeon)), provides instruction to the surgeon on how to improve his or her skills. The full mode circuit 122 activates the full mode. The full mode is a mode in which, for example, a conversation can be shared by everyone, including the instructor, surgeon, and participants.

[0117] When proctor mode is activated, two-way communication of information including audio is possible between the operating room terminal 13 and the instructor room terminal 14. When proctor mode is activated, one-way communication of information including audio is possible from the operating room terminal 13 to the participant room terminal 43. When proctor mode is activated, one-way communication of information including audio is possible from the instructor room terminal 14 to the participant room terminal 43. When proctor mode is activated, information including audio cannot be sent from the participant room terminal 43 to the instructor room terminal 14, and information including audio cannot be sent from the participant room terminal 43 to the operating room terminal 13.

[0118] Furthermore, when the proctor mode is activated, information including annotations can be sent from the instructor room terminal 14 to the operating room terminal 13, and information including annotations can be sent from the instructor room terminal 14 to the participant room terminal 43. When the proctor mode is activated, information including annotations cannot be sent from the participant room terminal 43 to the operating room terminal 13, and information including annotations cannot be sent from the participant room terminal 43 to the instructor room terminal 14.

[0119] For this reason, when proctor mode is activated, the surgeon cannot see the annotation exchanges taking place between the instructor and the participants. In other words, the surgeon in the operating room 15 cannot hear the audio information being exchanged between the instructor in the instruction room 16 and the participants in the participation room 116. This allows the surgeon to concentrate on the surgery. Meanwhile, the instructor in the instruction room 16 and the participants in the participation room 116 can freely discuss the good and bad points of the current surgery without putting pressure on the surgeon.

[0120] When the demo mode is activated, two-way communication of information including audio is possible between the participant room terminal 43 and the instructor room terminal 14. When the demo mode is activated, one-way communication of information including audio is possible from the operating room terminal 13 to the instructor room terminal 14. When the demo mode is activated, information including audio cannot be transmitted from the instructor room terminal 14 to the operating room terminal 13.

[0121] Furthermore, when the demo mode is activated, information including annotations can be communicated bidirectionally between the instruction room terminal 14 and the participating room terminal 43. When the demo mode is activated, information including annotations cannot be transmitted from the participating room terminal 43 to the operating room terminal 13, and information including annotations cannot be transmitted from the operating room terminal 13 to the participating room terminal 43.

[0122] When full mode is activated, two-way communication of information including audio is possible between the participating room terminal 43 and the instruction room terminal 14, and two-way communication of information including audio is possible between the participating room terminal 43 and the operating room terminal 13. When full mode is activated, two-way communication of information including annotations is possible between the participating room terminal 43 and the instruction room terminal 14, and two-way communication of information including annotations is possible between the participating room terminal 43 and the operating room terminal 13.

[0123] Regardless of which mode is activated, two-way communication of information including video is possible between the operating room terminal 13 and the instruction room terminal 14. Moreover, regardless of which mode is activated, two-way communication of information including video is possible between the operating room terminal 13 and the participant room terminal 43. Moreover, regardless of which mode is activated, two-way communication of information including video is possible between the instruction room terminal 14 and the participant room terminal 43.

[0124] FIG. 6 is an image diagram of mode switching. When proctor mode is activated, participant P1 participates by looking at the monitor 136. When demo mode is selected, participant P1 can receive explanations about the surgery from instructor P2. When full mode is activated, participant P1 can participate in the surgery while watching the interaction between the surgeon and instructor.

[0125] Fig. 7 is a conceptual diagram of an API (Application Programming Interface) when proctor mode is activated, Fig. 8 is a conceptual diagram of an API when demo mode is activated, and Fig. 9 is a conceptual diagram of an API when full mode is activated.

[0126] 10 shows a mode change sequence according to an embodiment of the present disclosure. In step S40, the mode icon on the monitor 36 of the teaching room terminal 14 is operated. The instructor can select a mode by operating the teaching room terminal 14 in step S40. The instructor can select a mode according to the first to third scenes. For example, the instructor can select the demo mode for the first scene. The instructor can select the full mode for the second scene. The instructor can select the proctor mode for the third scene.

[0127] Then, the selected mode is transmitted from the teaching room terminal 14 to the server 11. In step S41, the server 11 receives a signal of the selected mode from the teaching room terminal 14 and stores it in the storage unit 22. Then, in step S42, the switching unit 105 of the server 11 performs a process of activating the selected mode.

[0128] Then, commands to perform control according to the activated mode are sent from the server 11 to each of the operating room terminal 13, the instructor room terminal 14, and the participant room terminal 43. In step S43, the operating room terminal 13 displays the activated mode on the monitor 27. The monitor 55 of the vision cart 47 also displays the activated mode. The monitor 50 of the control unit 46 also displays the activated mode. In step S44, the instructor room terminal 14 displays the activated mode on the monitor 36. In step S45, the participant room terminal 43 displays the activated mode on the monitor 136.

[0129] The instructor may select each mode based on different conditions for the first, second, and third scenes. An example is as follows:

[0130] After performing steps S41 and S42, the server 11 again acquires the desired mode from the operating room terminal 13 and the participating room terminal 43, and acquires an evaluation from the participating room terminal 43, in step S38 before the operation of step S40 is performed. Furthermore, in step S39, which is performed following step S38 and before step S40, the server 11 can recommend to the instruction room terminal 14 as an appropriate mode the mode in which the surgery progressed most smoothly from the data stored in the memory unit 22 in the first support process described above, depending on the surgical content, case, surgeon, etc.

[0131] Furthermore, in step S39, the server 11 can recommend to the instruction room terminal 14 the mode that was selected at the time of the surgery that received the highest evaluation in the second support process described above, depending on the surgery content, case, surgeon, participants, etc. Furthermore, in step S39, the server 11 can recommend to the instruction room terminal 14 the mode preference received from the operating room terminal 13 or the participant room terminal 43. Therefore, in step S40, the instructor can display the evaluation and the mode preference on the monitor 36 of the instruction room terminal 14. In addition, in step S40, the instructor can operate the mode icon and select a mode based on the evaluation and the mode preference.

[0132] (Effects of the embodiment) The remote training system 10 disclosed herein allows young surgeons and other participants to view real-time images captured during surgery while a certified surgeon (expert physician) explains surgical precautions and provides specific advice for passing the certification exam. This allows for the improvement of the surgical skills of young surgeons and other participants who have a limited number of cases. Furthermore, after receiving one-on-one, real-time remote instruction from an instructor, the surgeon can review various information stored in the memory unit 22 of the server 11, such as video, images, and audio, on the operating room terminal 13. Furthermore, any user who logs in with an authorized login ID and password can review the information again on any terminal. Therefore, the surgeon can obtain valuable video that allows them to consider their strengths and weaknesses and take individual measures to pass the certification exam.

[0133] However, watching long videos of a surgery is painful, and it is not easy to grasp the important points of the surgery. This has been a challenge in training young surgeons. In this embodiment, while the participating young surgeons are viewing real-time images taken during the surgery, a certified surgeon explains important surgical points and summarizes the key points. This is therefore beneficial for improving the surgical skills of young surgeons who have only a small number of cases.

[0134] The summary of key points refers to general surgical precautions. This remote instruction system 10 demonstrates its true potential by taking into account the unique characteristics of each patient in each individual case of each surgeon. This includes individual factors that cannot be corrected by passively watching the video, such as the habits and ways of thinking of each surgeon.

[0135] According to the present disclosure, during surgery, it is possible to switch between the operating room terminal 13, the instructor room terminal 14, and the participant room terminal 43, with whom information including video, audio, and annotations can be exchanged. Therefore, it is possible to select a mode that matches the compliance requirements for live surgery established by various academic societies. Furthermore, the surgeon using the operating room terminal 13 can reduce stress caused by the situation in the operating room being viewed from the outside, i.e., by the instructor using the instructor room terminal 14 and the participants using the participant room terminal 43. Furthermore, the instructor room terminal 14 can provide appropriate instruction and annotations to the operating room terminal 13 according to the surgery being performed by the surgeon.

[0136] When the switching unit 105 of the server 11 of the present disclosure receives a mode activation request from the training room terminal 14 to indicate which mode to activate, it activates the requested mode. This allows the surgeon in the operating room to concentrate on the surgery, while the instructor can switch modes depending on the situation of the surgery. The annotation processing unit 84 of the server 11 of the present disclosure associates information including the operation of the robot main body acquired by the robot information processing unit with annotation information generated by the training room terminal 14 and stores the associated information in the memory unit 22. This allows for the generation of learning data to be used when providing instruction to junior surgeons or less experienced surgeons on how to perform surgery using the robot main body 48.

[0137] The auxiliary information processing unit 86 of the present disclosure can generate auxiliary information used to process the medical fees of the instructor on the processing terminal 81. Furthermore, the auxiliary information processing unit 86 can generate auxiliary information with a characteristic that causes the medical fees to differ depending on the type of mode activated by the switching unit 105. Therefore, when the remote instruction by the instructor is covered by insurance, the auxiliary information can be utilized, improving convenience for the user of the processing terminal 81.

[0138] An example of the technical meaning of the matters disclosed in this embodiment is as follows: The remote guidance system 10 is an example of an information processing system. The server 11 is an example of an information processing device. The monitors 27, 50, and 55 are examples of display units installed in an operating room. The microphones 29 and 51 are examples of microphones that capture audio within the operating room. The speakers 28 and 52 are examples of speakers that output audio captured from outside the operating room. The operating room terminal 13, the control unit 46, and the computer 56 are examples of operating room terminals.

[0139] The monitor 36 is an example of a display unit provided in the teaching room. The microphone 44 is an example of a microphone that acquires audio from within the teaching room. The speaker 38 is an example of a speaker that outputs audio acquired from outside the teaching room. The teaching room terminal 14 is an example of a teaching room terminal. The monitor 136 is an example of a display unit provided in the participation room. The microphone 144 is an example of a microphone that acquires audio from within the participation room. The speaker 138 is an example of a speaker that outputs audio acquired from outside the participation room. The communication unit 20 is an example of a communication unit of an information processing device. The memory unit 22 is an example of a memory unit. The robot information processing unit 85 is an example of a robot information processing unit.

[0140] The proctor mode is an example of a first mode. The demo mode is an example of a second mode. The full mode is an example of a third mode. The switching unit 105 can be defined as a switch or a switching circuit. The video processing unit 82 can be defined as a video processor or a video processing circuit. The audio processing unit 83 can be defined as an audio processor or an audio processing circuit. The annotation processing unit 84 can be defined as an annotation processor or an annotation processing circuit. The robot information processing unit 85 can be defined as a robot information processor or a robot information processing circuit. The auxiliary information processing unit 86 can be defined as an auxiliary information processor or an auxiliary information processing circuit.

[0141] Furthermore, this embodiment is not limited to what is disclosed using the drawings, and various modifications are possible without departing from the spirit of the present invention. For example, the control unit 46 shown in FIG. 3 may be provided in a location different from the operating room 15 where the robot body 48 is provided. In this case, the control unit 46 is connected to the computer 56 of the vision cart 47 via a network. The network may be either wireless or wired.

[0142] The control circuit of the information processing device disclosed in this embodiment performs the following processes: activates a mode selected on the instruction room terminal, stores the activated mode in a memory unit, stores the number of annotations and the required time for annotations made from the instruction room terminal to the operating room terminal and the participating room terminal for each type of surgery, each condition, each surgeon, and each participant performed in the operating room, in the memory unit, and determines which of the activated modes allowed the surgery in the operating room to proceed most smoothly based on the number of annotations and the required time, and recommends the determined mode to the instruction room terminal.

[0143] Furthermore, the control circuit performs the following processes: activating the mode selected on the instruction room terminal; acquiring from the participating room terminal an evaluation of the annotation sent from the instruction room terminal to the participating room terminal and storing the acquired evaluation in a memory unit; and recommending to the instruction room terminal the mode that was selected at the time the surgery that received the highest evaluation was performed.

[0144] Furthermore, the control circuit performs the process of activating the mode selected on the instruction room terminal, and the process of acquiring the desired mode from at least one of the operating room terminal or the participating room terminal before the mode is selected on the instruction room terminal, and recommending the desired acquired mode to the instruction room terminal. [Industrial Applicability]

[0145] The present disclosure can be used as an information processing device, an information processing system, and an information processing program connected to an operating room terminal, an instruction room terminal, and a participant room terminal via a network. [Explanation of symbols]

[0146] 10... remote instruction system, 11... server, 27, 36, 50, 51, 136... monitor, 29, 44, 51, 144... microphone, 28, 38, 52, 138... speaker, 13... operating room terminal, 46... control unit, 56... computer, 20... communication unit, 82... video processing unit, 83... audio processing unit, 84... annotation processing unit, 105... switching unit, 22... memory unit, 85... robot information processing unit, 86... auxiliary information processing unit

Claims

1. an operating room terminal provided in an operating room, the operating room terminal having a display unit for displaying images and annotations, a microphone for capturing audio within the operating room, and a speaker for outputting audio captured from outside the operating room; a teaching room terminal provided in a teaching room separated from the operating room, the teaching room terminal having a display unit for displaying images and annotations, a microphone for capturing audio within the teaching room, and a speaker for outputting audio captured from outside the teaching room; a participation room terminal provided in a participation room separated from the operating room and the instruction room, the participation room terminal having a display unit for displaying images and annotations, a microphone for capturing audio within the participation room, and a speaker for outputting audio captured from outside the participation room; a communication unit connected to each of the devices via a network; a storage unit in which a non-transitory program is stored; a control circuit connected to the communication unit and the storage unit, and configured to execute a process when a non-transitory program connected to the storage unit is started; An information processing device having: The control circuit Processing of images acquired from the operating room terminal, the teaching room terminal, and the participant room terminal; Processing of audio acquired from the operating room terminal, the teaching room terminal, and the participant room terminal; Processing of instruction and annotations acquired from the instruction room terminal; A process of switching the transmission state of information including video, audio, and annotations between the operating room terminal, the instruction room terminal, and the participant room terminal and the communication unit; Run the control circuit is activated by switching among a first mode, a second mode, and a third mode, which differ in the state of transmission of the information; When the first mode is activated, audio can be communicated bidirectionally between the operating room terminal and the instructor room terminal, audio can be transmitted unidirectionally from the operating room terminal to the participating room terminal, audio can be transmitted unidirectionally from the instructor room terminal to the participating room terminal, annotations can be transmitted unidirectionally from the instructor room terminal to the operating room terminal, and annotations can be transmitted unidirectionally from the instructor room terminal to the participating room terminal, When the second mode is activated, audio can be communicated bidirectionally between the participating room terminal and the teaching room terminal, audio can be transmitted unidirectionally from the operating room terminal to the teaching room terminal, audio cannot be communicated between the participating room terminal and the operating room terminal, annotations can be communicated bidirectionally between the teaching room terminal and the participating room terminal, annotations cannot be communicated between the participating room terminal and the operating room terminal, and annotations cannot be communicated between the teaching room terminal and the operating room terminal, When the third mode is activated, audio can be communicated bidirectionally between the participating room terminal and the teaching room terminal, audio can be communicated bidirectionally between the participating room terminal and the operating room terminal, audio can be communicated bidirectionally between the operating room terminal and the teaching room terminal, annotations can be communicated bidirectionally between the participating room terminal and the teaching room terminal, annotations can be communicated bidirectionally between the participating room terminal and the operating room terminal, and annotations can be communicated bidirectionally between the operating room terminal and the teaching room terminal, An information processing device that, regardless of whether the first mode, the second mode, or the third mode is activated, can communicate video bidirectionally between the operating room terminal and the instruction room terminal, can communicate video bidirectionally between the operating room terminal and the participating room terminal, and can communicate video bidirectionally between the instruction room terminal and the participating room terminal.

2. 2. The information processing device according to claim 1, The information processing device, wherein the control circuit stores a result of video processing, a result of audio processing, and a result of annotation processing in the storage unit.

3. 3. The information processing device according to claim 1, A robot body is provided in the operating room, The robot body is connected to the operating room terminal and is configured to operate based on the operation of a master control operated by a surgeon, The control circuit is an information processing device having a robot information processing unit that acquires information including the operation of the master control and the operation of the robot main body from the operating room terminal.

4. 4. The information processing device according to claim 3, The control circuit is an information processing device that associates information including the operation of the robot main body acquired by the robot information processing unit with information on annotations generated in the teaching room and stores the information in the memory unit.

5. 2. The information processing device according to claim 1, the communication unit is connected via a network to a processing terminal that processes medical fees for the instructor based on an action of the instructor providing remote instruction to the operating room terminal using the instruction room terminal, the control circuit generates auxiliary information used by the processing terminal to process medical fees; An information processing device, wherein the auxiliary information generated by the control circuit has a characteristic of varying the medical fee depending on whether the first mode, the second mode, or the third mode is activated.

6. an operating room terminal provided in an operating room, the operating room terminal having a display unit for displaying images and annotations, a microphone for capturing audio within the operating room, and a speaker for outputting audio captured from outside the operating room; a teaching room terminal provided in a teaching room separated from the operating room, the teaching room terminal having a display unit for displaying images and annotations, a microphone for capturing audio within the teaching room, and a speaker for outputting audio captured from outside the teaching room; a participation room terminal provided in a participation room separated from the operating room and the instruction room, the participation room terminal having a display unit for displaying images and annotations, a microphone for capturing audio within the participation room, and a speaker for outputting audio captured from outside the participation room; an information processing device connected to the operating room terminal, the instruction room terminal, and the participant room terminal via a network; and The information processing device includes: a storage unit in which a non-transitory program is stored; a control circuit that executes a process when the non-transitory program stored in the storage unit is started; An information processing system having: The control circuit Processing of images acquired from the operating room terminal, the teaching room terminal, and the participant room terminal; a processing unit for processing audio acquired from the operating room terminal, the instruction room terminal, and the participant room terminal; Processing annotations acquired from the teaching room terminal; A process of switching the transmission state of information including video, audio, and annotations between the operating room terminal, the teaching room terminal, and the participating room terminal; Run the control circuit is activated by switching among a first mode, a second mode, and a third mode, which differ in the state of transmission of the information; When the first mode is activated, audio can be communicated bidirectionally between the operating room terminal and the instructor room terminal, audio can be transmitted unidirectionally from the operating room terminal to the participating room terminal, audio can be transmitted unidirectionally from the instructor room terminal to the participating room terminal, annotations can be transmitted unidirectionally from the instructor room terminal to the operating room terminal, and annotations can be transmitted unidirectionally from the instructor room terminal to the participating room terminal, When the second mode is activated, audio can be communicated bidirectionally between the participating room terminal and the teaching room terminal, audio can be transmitted unidirectionally from the operating room terminal to the teaching room terminal, audio cannot be communicated between the participating room terminal and the operating room terminal, annotations can be communicated bidirectionally between the teaching room terminal and the participating room terminal, annotations cannot be communicated between the participating room terminal and the operating room terminal, and annotations cannot be communicated between the teaching room terminal and the operating room terminal, When the third mode is activated, audio can be communicated bidirectionally between the participating room terminal and the teaching room terminal, audio can be communicated bidirectionally between the participating room terminal and the operating room terminal, audio can be communicated bidirectionally between the operating room terminal and the teaching room terminal, annotations can be communicated bidirectionally between the participating room terminal and the teaching room terminal, annotations can be communicated bidirectionally between the participating room terminal and the operating room terminal, and annotations can be communicated bidirectionally between the operating room terminal and the teaching room terminal, An information processing system in which, regardless of whether the first mode, the second mode, or the third mode is activated, video can be communicated bidirectionally between the operating room terminal and the instruction room terminal, video can be communicated bidirectionally between the operating room terminal and the participating room terminal, and video can be communicated bidirectionally between the instruction room terminal and the participating room terminal.

7. an operating room terminal provided in an operating room, the operating room terminal having a display unit for displaying images and annotations, a microphone for capturing audio within the operating room, and a speaker for outputting audio captured from outside the operating room; a teaching room terminal provided in a teaching room separated from the operating room, the teaching room terminal having a display unit for displaying images and annotations, a microphone for capturing audio within the teaching room, and a speaker for outputting audio captured from outside the teaching room; a participation room terminal provided in a participation room separated from the operating room and the instruction room, the participation room terminal having a display unit for displaying images and annotations, a microphone for capturing audio within the participation room, and a speaker for outputting audio captured from outside the participation room; an information processing program that causes a control circuit of an information processing device connected to the information processing device via a network to execute processing, The control circuit, Processing of images acquired from the operating room terminal, the teaching room terminal, and the participant room terminal; Processing of audio acquired from the operating room terminal, the teaching room terminal, and the participant room terminal; Processing annotations acquired from the teaching room terminal; A process of switching a transmission state of information including video, audio, and annotations between the operating room terminal, the instruction room terminal, and the participant room terminal and the information processing device; Execute the control circuit is activated by switching among a first mode, a second mode, and a third mode, which differ in the state of transmission of the information; When the first mode is activated, audio can be communicated bidirectionally between the operating room terminal and the instructor room terminal, audio can be transmitted unidirectionally from the operating room terminal to the participating room terminal, audio can be transmitted unidirectionally from the instructor room terminal to the participating room terminal, annotations can be transmitted unidirectionally from the instructor room terminal to the operating room terminal, and annotations can be transmitted unidirectionally from the instructor room terminal to the participating room terminal, When the second mode is activated, audio can be communicated bidirectionally between the participating room terminal and the teaching room terminal, audio can be transmitted unidirectionally from the operating room terminal to the teaching room terminal, audio cannot be communicated between the participating room terminal and the operating room terminal, annotations can be communicated bidirectionally between the teaching room terminal and the participating room terminal, annotations cannot be communicated between the participating room terminal and the operating room terminal, and annotations cannot be communicated between the teaching room terminal and the operating room terminal, When the third mode is activated, audio can be communicated bidirectionally between the participating room terminal and the teaching room terminal, audio can be communicated bidirectionally between the participating room terminal and the operating room terminal, audio can be communicated bidirectionally between the operating room terminal and the teaching room terminal, annotations can be communicated bidirectionally between the participating room terminal and the teaching room terminal, annotations can be communicated bidirectionally between the participating room terminal and the operating room terminal, and annotations can be communicated bidirectionally between the operating room terminal and the teaching room terminal, An information processing program that enables bidirectional communication of video between the operating room terminal and the instruction room terminal, bidirectional communication of video between the operating room terminal and the participating room terminal, and bidirectional communication of video between the instruction room terminal and the participating room terminal, regardless of whether the first mode, the second mode, or the third mode is activated.

Images