Information processing device, information processing system, and information processing program

Abstract

Provided is an information processing device capable of switching between an operating room terminal, a guidance room terminal, and a participation room terminal between opposite parties capable of mutually communicating information including video, sound, and annotations. The server (11) is provided with a control circuit (17) which is connected to the operating room terminal (13), the guidance room terminal (14), and the participation room terminal (43) via the network (12), and executes processing, the control circuit (17) executes processing of the acquired image, processing of the acquired sound, processing of the acquired annotation, and processing of switching the transmission state of the information, and the operation room terminal (13), the guidance room terminal (14), and the participation room terminal (43) are connected via the network (12). The control circuit (17) is configured so as to switch between a first mode, a second mode, and a third mode in which information transmission states are different, and to activate the control circuit (17).

Description

Technical Field

[0001] The present disclosure relates to an information processing device, an information processing system, and an information processing program connected to an operating room terminal, an instruction room terminal, and a participant room terminal via a network. Background Art

[0002] A doctor who has learned the technique performs the following operations: videotaping the surgery and recording the captured images in a recording device such as an HDD (Hard Disk Drive) or a DVD (Digital Versatile Disc). A doctor who has learned the technique is called an expert doctor. A novice doctor performs the following operations: browsing the images of the surgery recorded in these recording devices to hone his surgical skills. An example of such a technology is recorded in Patent Document 1.

[0003] Patent Document 1 states:

[0004] “A medical imaging system comprising:

[0005] A photographing unit configured to control photographing conditions and photograph the affected part;

[0006] an acquisition unit, which acquires information related to the shooting conditions set in the shooting unit; and

[0007] The control unit controls, based on the imaging conditions, communication conditions related to transmission of image data corresponding to the imaging result of the affected part by the imaging unit via a network. ".

[0008] Prior Art Literature

[0009] Patent Literature

[0010] Patent Document 1: Japanese Patent Application Laid-Open No. 2021-40874 Summary of the invention

[0011] -Problems to be solved by the invention-

[0012] The inventors of the present application have recognized the following problem: According to the technology described in Patent Document 1, it is not possible to switch between the operating room terminal, the instruction room terminal, and the participant room terminal to communicate information including images, sounds, and annotations with each other.

[0013] The present disclosure aims to provide an information processing device, an information processing system and an information processing program capable of switching between operating room terminals, instruction room terminals and participant room terminals with which information including images, sounds and annotations can be communicated.

[0014] -Methods for solving problems-

[0015] The information processing device disclosed in the present invention comprises: a communication unit connected to the following terminals via a network: an operating room terminal, which is arranged in the operating room and has a display unit for displaying images and annotations, a microphone for acquiring sounds in the operating room, and a speaker for outputting sounds acquired from outside the operating room; an instruction room terminal, which is arranged in an instruction room separated from the operating room and has a display unit for displaying images and annotations, a microphone for acquiring sounds in the instruction room, and a speaker for outputting sounds acquired from outside the instruction room; a participant room terminal, which is arranged in a participant room separated from the operating room and the instruction room and has a display unit for displaying images and annotations, a microphone for acquiring sounds in the instruction room, and a speaker for outputting sounds acquired from outside the instruction room. a display unit for displaying images and annotations, a microphone for acquiring sounds in the participant room, and a speaker for outputting sounds acquired from outside the participant room; a storage unit storing a non-transitory program; and a control circuit connected to the communication unit and the storage unit, and starting the non-transitory program connected to the storage unit to perform processing, the control circuit performing: processing of images acquired from the operating room terminal, the instruction room terminal, and the participant room terminal; processing of sounds acquired from the operating room terminal, the instruction room terminal, and the participant room terminal; processing of instructions and annotations acquired from the instruction room terminal;and switching the transmission state of information including images, sounds, and annotations between the operating room terminal, the instruction room terminal, and the participant room terminal and the communication unit, the control circuit switching and starting the first mode, the second mode, and the third mode in which the transmission state of the information is different, when the first mode is started, sound can be communicated bidirectionally between the operating room terminal and the instruction room terminal, and sound can be sent unidirectionally from the operating room terminal to the participant room terminal, and sound can be sent unidirectionally from the instruction room terminal to the participant room terminal, and annotations can be sent unidirectionally from the instruction room terminal to the operating room terminal, and annotations can be sent unidirectionally from the instruction room terminal to the participant room terminal, when the second mode is started, sound can be communicated bidirectionally between the participant room terminal and the instruction room terminal, and sound can be sent unidirectionally from the operating room terminal to the instruction room terminal, and sound cannot be communicated between the participant room terminal and the operating room terminal, and sound can be sent unidirectionally between the instruction room terminal and the participant room terminal bidirectionally communicate annotations, and annotations cannot be communicated between the participant room terminal and the operating room terminal, and annotations cannot be communicated between the instruction room terminal and the operating room terminal. When the third mode is activated, voice can be communicated bidirectionally between the participant room terminal and the instruction room terminal, and voice can be communicated bidirectionally between the participant room terminal and the operating room terminal, and voice can be communicated bidirectionally between the operating room terminal and the instruction room terminal, and annotations can be communicated bidirectionally between the participant room terminal and the instruction room terminal, and annotations can be communicated bidirectionally between the participant room terminal and the operating room terminal, and annotations can be communicated bidirectionally between the operating room terminal and the instruction room terminal. When any of the first mode, the second mode, or the third mode is activated, images can be communicated bidirectionally between the operating room terminal and the instruction room terminal, and images can be communicated bidirectionally between the operating room terminal and the participant room terminal, and images can be communicated bidirectionally between the instruction room terminal and the participant room terminal. ;

[0016] -Effects of the Invention-

[0017] According to the present disclosure, during surgery, it is possible to switch between the operating room terminal, the instruction room terminal, and the participant room terminal, with each other being able to communicate information including images, sounds, and annotations. For example, it is possible to suppress the pressure on the surgeon using the operating room terminal due to the situation in the operating room being seen and heard by the outsiders, that is, the instructor using the instruction room terminal and the participant using the participant room terminal. In addition, it is possible to provide appropriate instructions and annotations from the instruction room terminal to the operating room terminal according to the surgery performed by the surgeon. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1A This is a block diagram showing a first specific example of a remote guidance system which is an example of the information processing system of the present disclosure.

[0019] Figure 1B This is a block diagram showing an example of a server included in the remote guidance system.

[0020] Figure 1C : is a block diagram showing an example of an operating room included in the remote guidance system.

[0021] Figure 1D This is a block diagram showing an example of a guidance room included in the remote guidance system.

[0022] Figure 1E This is a block diagram showing an example of a participant room included in the remote guidance system.

[0023] Figure 2A It is a schematic diagram showing a display example of a monitor installed in an operating room.

[0024] Figure 2B It is a schematic diagram showing a display example of a monitor installed in a guidance room.

[0025] Figure 3 This is a block diagram showing a second specific example of the remote guidance system of the present disclosure.

[0026] Figure 4 1 is a flowchart showing an example of a remote guidance method performed using the remote guidance system of the present disclosure.

[0027] Figure 5 This is an example of a screen displayed on the monitor of the instruction room terminal.

[0028] Figure 6 is a schematic diagram illustrating the function of mode switching in the present disclosure.

[0029] Figure 7 This is a conceptual diagram showing an API when the monitoring mode is selected.

[0030] Figure 8 This is a conceptual diagram showing an API when the demonstration mode is selected.

[0031] Fig. 9 This is a conceptual diagram showing an API when the full mode is selected.

[0032] Fig.10 This is a conceptual diagram showing a sequence process performed when an icon of the guidance room terminal is operated in the present disclosure. DETAILED DESCRIPTION

[0033] (Summary of the present disclosure)

[0034] The information processing system disclosed in the present invention is connected via a network to an operating room terminal used by a subject who receives guidance related to medical treatment, specifically surgery, and a guidance room terminal used by a guide who guides the subject. As an example of an information processing system, a remote guidance system is disclosed. As an example of an information processing program, a remote guidance program is disclosed. The main purpose of the present invention is that the remote guidance system is used for so-called remote monitoring in which the surgeon performs surgery on a patient in the operating room and guides the surgery in real time from a place away from the operating room. The images used by the remote guidance system include both 2D images and 3D images. In addition, the images include annotations and pointers. 3D images are, for example, stereoscopic types. The remote guidance system disclosed in the present invention includes the following examples of use.

[0035] The remote guidance system disclosed in the present invention can be used in the first scenario, such as real-time guidance in a "one-to-one" situation. Here, the first "one person" is a novice surgeon, and the second "one person" is an instructing doctor. In order to obtain certification as a doctor based on various societies, novice surgeons submit images during surgery as proof of their skills to various societies. When preparing images to be submitted to various societies, novice surgeons can use the remote guidance system in actual surgery and receive guidance remotely from the instructor. Novice surgeons can use the images actually obtained during surgery as proof of their skills.

[0036] The remote guidance system disclosed in the present invention can be used in the second scenario, such as a video clinic in a "1 person vs. n person" situation. Here, "1 person" is a novice surgeon. "n people" are instructors other than the novice surgeon, and "n" is an integer greater than 1. For example, a novice surgeon presents a video image as technical proof for becoming a certified professional doctor by a society. The remote guidance system records the image of the novice surgeon performing surgery without the guidance of an instructor, and shares the image with the supervising doctor, colleagues, surgeons affiliated with other medical facilities, etc., so that the novice surgeon can receive guidance and have discussions.

[0037] The remote guidance system disclosed in the present invention can be used in the third scenario, such as a public live broadcast of a society in a "1 person vs. 1 person vs. n people" situation. Here, the first "1 person" is the surgeon, the second "1 person" is the supervising doctor, and the "n people" are the participants. While the surgeon is performing the operation, the remote guidance system provides information including the images of the operation to places such as societies and public lectures. The remote guidance system has the function of the instructor providing guidance to the surgeon during the operation performed by the surgeon, as well as displaying annotations and pointers. The remote guidance system disclosed in the present invention has the function of delivering guidance, annotations and pointers to the surgeon and participants. The remote guidance system disclosed in the present invention has the function of not delivering the images and sounds of the discussion between the supervising doctor and the participants to the surgeon.

[0038] The remote guidance system disclosed in the present invention has a function that can be used in a video on demand (VOD) situation. The remote guidance system disclosed in the present invention has, for example, a function of automatically recording and uploading the information of images, guidance, annotations, and pointers obtained in the first scene, the second scene, and the third scene. The remote guidance system disclosed in the present invention has a function of uploading files containing various information. The remote guidance system disclosed in the present invention has a function of managing uploaded animations. The remote guidance system disclosed in the present invention has a function of managing the viewing and listening of uploaded animations. The remote guidance system disclosed in the present invention has a function of managing the sharing of uploaded animations.

[0039] The remote guidance system disclosed in the present invention has a function of broadcasting information including images, guidance, comments, and pointers obtained in the third scene to a plurality of users in one direction at the same time.

[0040] The present disclosure can be used for the training of professional doctors, elimination of the shortage of surgeons, research and mastery of advanced surgical cases, local medical training, situations where instructors provide remote guidance to subjects, etc. In addition, the present disclosure can be used when medical institutions actually perform surgery. In addition, it is also effective for the career path of female surgeons and the basis for providing medical services with less regional differences.

[0041] Next, at least one specific example of a remote guidance system including the remote guidance apparatus, the remote guidance method, and the remote guidance program of the present disclosure will be described using the drawings.

[0042] (First specific example)

[0043] The first specific example is an example in which a surgery is performed on a patient in an operating room and the surgery is guided in real time from a place away from the operating room. More specifically, it is an example in which a person involved in the surgery performs the surgery on the patient manually while the patient is lying on the operating table. The first specific example is shown in FIG. Figure 1A , Figure 1B , Figure 1C , Figure 1D , Figure 1E , Figure 2A as well as Figure 2B .

[0044] (Description of the overall structure)

[0045] Figure 1A The remote guidance system 10 shown includes a server 11, an operating room terminal 13, a guidance room terminal 14, and a participant room terminal 43. The server 11, the operating room terminal 13, the guidance room terminal 14, and the participant room terminal 43 are connected to each other via a network 12. In addition, the server 11 is connected to a processing terminal 81 via the network 12. The network 12 includes the Internet, an intranet, a wireless LAN (Local Area Network), and a mobile phone system network such as 4G and 5G.

[0046] The operating room terminal 13 and the instruction room terminal 14 can be indirectly connected via the server 11. In addition, when the operating room terminal 13 and the instruction room terminal 14 are authenticated by the server 11, the operating room terminal 13 and the instruction room terminal 14 can also bypass the server 11 and directly perform two-way communication. Figure 1A , an example is shown in which a single operating room terminal 13 is connected to the server 11, a single instruction room terminal 14 is connected to the server 11, and a single participant room terminal 43 is connected to the server 11, but a plurality of operating room terminals 13 may be connected to the server 11. In addition, a plurality of instruction room terminals 14 may be connected to the server 11. In addition, a plurality of participant room terminals 43 may be connected to the server 11. The network 12 is constructed by at least one of wireless communication or wired communication. In addition, wireless communication includes Wi-Fi (registered trademark).

[0047] In the network 12 and the server 11, the communication circuit of the signal including information such as images, pictures, sounds, and data is constructed by independent interfaces and compression transmission technology. The independent interface includes an encoder for signal processing, a browser and an application for connecting to the server 11 using the Internet line. In addition, by compressing high-quality images, two-way communication can be performed in real time between the guidance room terminal 14, the operating room terminal 13, and the participant room terminal 43.

[0048] The operating room terminal 13 is used by the subject receiving the guidance related to the surgery. The subject includes the surgeon, nurse, assistant, etc. The operating room terminal 13 is set in the operating room 15, for example. The guidance room terminal 14 is a terminal used by the instructor who gives the subject guidance related to the surgery. The guidance room terminal 14 is set in the guidance room 16. The operating room 15 and the guidance room 16 are, for example, located in different places. The different places can be any of different countries, different prefectures, different towns and villages, different buildings, different rooms in the same building, etc. The participant room terminal 43 is used by the trainee who observes the situation of the operating room 15 and learns the guidance of the instructor. The participant room terminal 43 is used in places other than the operating room 15 and the guidance room 16. An example of the structure of the server 11, the structure of the operating room 15, the structure of the guidance room 16, and the structure of the participant room terminal 43 is as follows.

[0049] (Server description)

[0050] like Figure 1B As shown, the server 11 includes a control circuit 17, an operation unit 18, a monitor 19, a communication unit 20, and a storage unit 22. The server 11 is set in a place different from the operating room 15 and the instruction room 16. The control circuit 17 includes an image processing unit 82, a sound processing unit 83, an annotation processing unit 84, a robot information processing unit 85, an auxiliary information processing unit 86, a switching unit 105, and a mode selection support unit 92. The control circuit 17 can perform various processing and judgment based on various information obtained from the operating room terminal 13, various information obtained from the instruction room terminal 14, and various information obtained from the participant room terminal 43, and store the processing results and judgment results in the database of the storage unit 22. Various information includes, for example, images, sounds, annotations, pointers, folders, files, and data. The control circuit 17 can also convert sound information into text in real time and translate it into a foreign language such as English. The translation method can also use artificial intelligence technology such as deep learning to automatically translate.

[0051] The image processing unit 82 has a function of processing images acquired from the operating room terminal 13, the instruction room terminal 14, and the participant room terminal 43, and images read from the storage unit 22. The sound processing unit 83 has a function of processing sounds acquired from the operating room terminal 13, the instruction room terminal 14, and the participant room terminal 43, and sounds read from the storage unit 22. The annotation processing unit 84 has a function of processing annotations acquired from the operating room terminal 13, the instruction room terminal 14, and the participant room terminal 43, and annotations read from the storage unit 22. In addition, the annotation processing unit 84 has a function of associating information including the movement of the robot main body 48 acquired by the robot information processing unit 85 with information of annotations generated by the instruction room terminal 14 and storing them in the storage unit 22.

[0052] The robot information processing unit 85 has a function of acquiring information including the operation of the main controller 49 and the movement of the robot body 48 from the operating room terminal 13 and processing the acquired information. The auxiliary information processing unit 86 can generate auxiliary information of characteristics that make the medical treatment remuneration different according to the type of mode activated by the switching unit 105. The medical treatment remuneration will be described later. The communication unit 20 is connected to the network 12. The signal received by the communication unit 20 via the network 12 is input to the control circuit 17. In addition, the signal output from the communication unit 20 is sent to the operating room terminal 13, the instruction room terminal 14, and the participation room terminal 43 via the network 12.

[0053] The operation unit 18 is constructed by at least one element of a touch switch, a display screen such as a liquid crystal panel, a keyboard, a mouse, etc. When the administrator of the server 11 operates the operation unit 18, a signal corresponding to the operation content is input to the control circuit 17. The monitor 19 has a display screen. The display screen of the monitor 19 is constructed by a touch panel, a liquid crystal panel, etc. The content displayed on the display screen of the monitor 19 is controlled by the control circuit 17. The display screen of the operation unit 18 and the display screen of the monitor 19 can be set separately or combined.

[0054] The storage unit 22 may be a built-in type provided inside the main body constituting the server 11, or an external type provided outside the main body. In addition, the external storage unit 22 may also be a hard disk, a DVD, a stick memory, etc. In the storage unit 22, non-temporary programs, data, information, etc. used in the processing and judgment performed by the control circuit 17 are stored. The learning data for machine learning stored in the storage unit 22 may also use data in the natural language processing previously performed. Thus, foreign surgeons can also participate and improve their surgical skills. In the storage unit 22, applications activated in the processing, judgment, control, etc. performed by the control circuit 17 are stored, and various information, data, etc. used in the processing, judgment, control, etc. performed by the control circuit 17 are stored. In addition, in the storage unit 22, various information and data, etc. sent to the operating room terminal 13, the instruction room terminal 14, and the participant room terminal 43 are stored. In the storage unit 22, evaluation criteria sent to the participant room terminal 43 are also stored.

[0055] The mode selection support unit 92 has a function of supporting the selection operation of the mode by the guidance room terminal 14. The mode selection support unit 92 can perform the following support processing, for example.

[0056] (First support process)

[0057] The mode selection support unit 92 stores the mode selected by the instruction room terminal 14 as data in the storage unit 22 for each disease condition, each surgical content, each surgeon, and each participant. In addition, the mode selection support unit 92 stores the number of instructions and the required time for instructions from the instruction room terminal 14 to the operating room terminal 13 and the participant room terminal 43, the number of times and the required time for annotations, etc., for each surgical content, each disease condition, each surgeon, and each participant in the storage unit 22. Thereafter, when the operation is performed in the operating room 15, the mode selection support unit 92 recommends the mode in which the operation is performed most smoothly from the data stored in the storage unit 22 to the instruction room terminal 14 from the server 11, based on the surgical content, the case, the surgeon, etc., as a suitable mode. For example, the mode selection support unit 92 can determine that the mode selected in the following cases is the "mode in which the operation is performed most smoothly": the time required from the start of the operation to the end of the operation is the shortest, the number of times and the required time for instructions from the instruction room terminal 14 to the operating room terminal 13 during the operation is the shortest, and the number of times and the required time for annotations are the shortest.

[0058] (Second support processing)

[0059] The mode selection support unit 92 stores the relationship between each surgical content, each disease condition, each surgeon, participant, etc. and the mode selected by the instruction room terminal 14 in the storage unit 22 as data. In addition, the mode selection support unit 92 obtains the evaluation of the instructions and comments sent from the instruction room terminal 14 to the participant room terminal 43 from the participant room terminal 43, and stores the obtained evaluation in the storage unit 22. The evaluation obtained from the participant room terminal 43 will be described later. In addition, the mode selection support unit 92 performs a process of recommending the mode selected at the time of performing the operation with the highest evaluation from the server 11 to the instruction room terminal 14 when the operation is performed after the evaluation is stored in the storage unit 22.

[0060] (Third support processing)

[0061] Before the instructor operates the instruction room terminal 14 and selects a mode, the mode selection support unit 92 acquires the desired mode from at least one of the operating room terminal 13 and the attending room terminal 43. The mode selection support unit 92 also recommends the acquired desired mode to the instruction room terminal 14.

[0062] The instructor can operate the operation unit 18 of the instruction room terminal 14 according to the aforementioned first to third scenes to select a plurality of modes, for example, any one of the first to third modes. In addition, the instructor can also select any one of the first to third modes based on the information recommended by the mode selection support unit 92.

[0063] The switching unit 105 has a function of starting the selected mode. The transmission state of various information between the operating room terminal 13 and the instruction room terminal 14, the transmission state of various information between the instruction room terminal 14 and the participant room terminal 43, and the transmission state of various information between the operating room terminal 13 and the participant room terminal 43 are different in the first to third modes. The switching unit 105 includes a monitoring mode circuit 120, a demonstration mode circuit 121, and a full mode circuit 122. In addition, the specific functions of the monitoring mode circuit 120, the demonstration mode circuit 121, and the full mode circuit 122 and specific examples of the first to third modes will be described later.

[0064] (Instructions for the operating room)

[0065] like Figure 1C As shown, in the operating room 15, an operating room terminal 13, a surgical field camera 23, a first camera 24, a second camera 25, a monitor 27, a speaker 28, a microphone 29, and an operation unit 30 are provided. The surgical field camera 23 includes an endoscope, a laparoscope camera, and a robot camera. The operating room terminal 13 is a computer used by a subject who receives guidance from a guide. The subject includes a surgeon, a trainee, a nurse, an assistant of the surgery, and the like. The operating room terminal 13 controls and manages the monitor 27, the surgical field camera 23, the first camera 24, the second camera 25, the microphone 29, the speaker 28, and the like.

[0066] The operating room terminal 13 has a control circuit 31, a storage unit 32, and a communication unit 33. The control circuit 31 is mainly composed of, for example, a video card (also called a graphic board), and has an encoder and a decoder for performing data conversion of signals, a calculation circuit for performing various processing, comparison, and calculation, and a memory for temporarily storing information. The operating room terminal 13 is connected to the operation unit 30, the monitor 27, the surgical field camera 23, the first camera 24, the second camera 25, the microphone 29, and the speaker 28 so that signals can be sent and received. The communication unit 33 receives a signal sent from the server 11 via the network 12. The signal received by the communication unit 33 is processed by the control circuit 31. In addition, the signal output from the communication unit 33 is sent to the server 11 via the network 12. The signal input to the communication unit 33 and the signal output from the communication unit 33 include information such as video, image, sound, data, annotation, and pointer.

[0067] The monitor 27 constructs a display screen through a touch panel. The touch panel can be any of an electrostatic capacitance type, a resistive film type, and an infrared type. The monitor 27 and the operating room terminal 13 are connected through a communication line such as HDMI (registered trademark (High-Definition Multimedia Interface: registered trademark)), USB (Universal Serial Bus), etc. Between the monitor 27 and the operating room terminal 13, signals containing information such as images, pictures, and sounds are communicated bidirectionally. Figure 2A 2 shows an example of a display screen 34 of the monitor 27. The display screen 34 can display: a display unit 34A displays an image captured by the surgical field camera 23; a display unit 34B displays an image captured by the first camera 24; a display unit 34C displays an image captured by the second camera 25; and a display unit 34D displays an image sent from the guidance room terminal 14. In addition, the display screen 34 can also display an operation menu. The display screen 34 supports high-resolution images.

[0068] The operation unit 30 is constructed by at least one element among a keyboard, a mouse, and a display screen of a touch panel. The operation unit 30 is connected to a control circuit 31 of the operating room terminal 13. When the subject operates the operation unit 30, a signal corresponding to the operation is input to the control circuit 31. The display screen of the operation unit 30 and the display screen 34 of the monitor 27 can be provided separately or combined. Figure 2A The display screen 34 of the monitor 27 shown is an example of integration with the display screen of the operation unit 30. The surgeon or nurse can also operate the operation unit 30 to select a desired mode for each surgery, each disease condition, each surgical content, each surgeon, and each participant, and send it from the operating room terminal 13 to the server 11.

[0069] The storage unit 32 may be a built-in type provided inside the main body constituting the operating room terminal 13, or an external type provided outside the main body. In addition, the external storage unit 32 may also be a hard disk, a DVD (Digital Versatile Disc), a stick memory, etc. The storage unit 32 stores non-temporary programs, data, information, etc. used in processing, judgment, control, etc. performed by the control circuit 31.

[0070] The endoscope (laparoscope) 23 and the robot body 48 are medical devices for the purpose of observing the inside of a patient, that is, a human body, which is a surgical object. In addition to the function of observing the inside of a human body, the surgical field camera 23 may also have a structure and function for performing a given surgery and collecting specimens. The surgical field camera 23 converts the image captured of the inside of the human body, specifically the surgical field, into an image signal and outputs it. In addition, the surgical field camera 23 may also have a microphone. The surgical field camera 23 and the operating room terminal 13 are connected via a communication line such as an SDI (Serial Digital Interface), and digital image signals and digital sound signals are sent from the surgical field camera 23 to the operating room terminal 13.

[0071] The first camera 24 and the second camera 25 convert the images captured in the operating room 15 into signals and output them. The first camera 24 and the second camera 25 are cameras that overlook the operating room 15. The first camera 24 captures, for example, the entire image in the operating room 15. The capturing range (space) of the second camera 25 is narrower than that of the first camera 24. The first camera 24 and the second camera 25 are different in the installation location, the captured location, the captured direction, etc. The first camera 24 and the second camera 25 are connected in parallel to each other, for example, connected to the operating room terminal 13 via HDMI. The first camera 24 and the second camera 25 capture the overall situation of the team including surgical assistants, nurses, etc., that is, images of the surgeon's standing position, the assistant's standing position, the assistant's forceps movements, etc.

[0072] The first camera 24 and the second camera 25 may be installed in the operating room 15 in any manner such as a fixed type, a movable type, or a person holding the camera by hand. In addition, although not shown in the figure, a signal including information of an electronic medical record, a signal including a CT (Computed Tomography) image, a signal including an MRI (Magnetic Resonance Imaging) image, a signal including an echo examination (ultrasound examination) image, etc. may be input to the operating room terminal 13, and the signal may be replaced with the signal of the first camera 24 or the signal of the second camera 25.

[0073] The microphone 29 is installed in the operating room 15, and converts and outputs the sound of the operating room 15 into a signal. The signal output from the microphone 29 is sent to the operating room terminal 13. In addition, the microphone 29 may be installed in either the first camera 24 or the second camera 25, or may be installed independently of the first camera 24 and the second camera 25.

[0074] The speaker 28 is connected to the operating room terminal 13. When a signal including sound is input from the instruction room terminal 14 to the operating room terminal 13 via the network 12, the speaker 28 can output the sound to the operating room 15. The speaker 28 may be provided in either the monitor 27 or the main body of the operating room terminal 13, or may be provided independently of the monitor 27 and the main body of the operating room terminal 13.

[0075] (Explanation from the Guidance Office)

[0076] like Figure 1D As shown, in the instruction room 16, an instruction room terminal 14, an operation unit 35, a monitor 36, a camera 37, a speaker 38, and a microphone 44 are provided. The instruction room terminal 14 is a computer used by an instructor who instructs the subject. The instructor includes surgeons, university professors, authorities in the medical field, and scholars. The instruction room terminal 14 is connected to the server 11 via the network 12, and is connected to the operating room terminal 13 via the network 12. Two-way communication is possible between the instruction room terminal 14 and the server 11, two-way communication is possible between the instruction room terminal 14 and the operating room terminal 13, and two-way communication is possible between the server 11 and the instruction room terminal 14. The instruction room terminal 14 is connected to the operation unit 35, the monitor 36, the camera 37, the speaker 38, and the microphone 44.

[0077] The guidance room terminal 14 may be any of a fixed terminal, a mobile terminal, or a portable terminal, or may be a unit formed by combining a plurality of these terminals. The guidance room terminal 14 includes a control circuit 39, a storage unit 40, and a communication unit 41. The control circuit 39 is mainly composed of a video card (also called a graphic board), and includes an encoder and a decoder for performing data conversion of signals, an operation circuit for performing various processing, comparison, and operation, and a memory for temporarily storing information.

[0078] The storage unit 40 may be a built-in type installed inside the main body constituting the guidance room terminal 14, or an external type installed outside the main body. In addition, the external storage unit 40 may also be a hard disk, a DVD (Digital Versatile Disc), a stick memory, etc. The storage unit 40 stores non-temporary programs, data, information, etc. used in the processing and judgment performed by the control circuit 39.

[0079] The instruction room terminal 14 is connected to the operation unit 35, the monitor 36, the camera 37, the microphone 44, and the speaker 38 in a manner that enables signal transmission and reception. The communication unit 41 receives a signal transmitted from the server 11 via the network 12, and receives a signal transmitted from the operating room terminal 13 via the network 12. The signal received by the communication unit 41 is input to the control circuit 39. In addition, the signal output from the communication unit 41 is transmitted to the server 11 via the network 12. The signal input to the communication unit 41 and the signal output from the communication unit 41 include information such as video, image, sound, and data.

[0080] like Figure 2B As shown, the monitor 36 constructs a display screen 42 through a touch panel. The touch panel can be any of an electrostatic capacitance method, a resistive film method, and an infrared method. The monitor 36 and the guidance room terminal 14 are connected through a communication line such as HDMI. Signals containing information such as images, pictures, and sounds are exchanged between the monitor 36 and the guidance room terminal 14. The display screen 42 shows: a display unit 42A, which displays the image of the surgical field camera 23 sent via the network 12; a display unit 42B, which displays the image shot by the first camera 24 sent via the network 12; a display unit 42C, which displays the image shot by the second camera 25 sent via the network 12; a display unit 42D, which displays the image shot by the camera 37; a dedicated menu 42E for giving annotations and pointers; and other operation menus, etc. The display screen 42 corresponds to a high-resolution image. In addition, the monitor 36 can also display the evaluation of the instructions and annotations sent from the server 11, the hope of the mode, etc.

[0081] The operation unit 35 includes a keyboard, a mouse, a display screen of a touch panel, a touch pen, etc. The operation unit 35 is connected to the instruction room terminal 14. When the instructor operates the operation unit 35, a signal corresponding to the operation is input to the instruction room terminal 14. The display screen of the operation unit 35 and the display screen of the monitor 36 can be provided separately or combined. Figure 2B This is an example of combining the display screen of the operation unit 35 and the display screen 42 of the monitor 36. The instructor can use a touch pen or a mouse to operate the operation menu and the dedicated menu 42E of the display screen 42. In addition, the instructor can add annotations based on the drawing to the display unit 42A. The technical meaning of annotations will be described later.

[0082] The camera 37 converts the image captured in the instruction room 16 into a signal and outputs it. The camera 37 is connected to the instruction room terminal 14, for example, via HDMI. The camera 37 may be installed in the instruction room in any manner, such as a fixed type, a movable type, or a hand-held type by a staff member.

[0083] Microphone 44 is installed in guidance room 16, and converts and outputs the sound of guidance room 16. The signal output from microphone 44 is sent to guidance room terminal 14. Microphone 44 may be installed in camera 37 or installed independently of camera 37.

[0084] The speaker 38 is connected to the instruction room terminal 14. When a signal including sound is output from the operating room terminal 13 and the signal is input to the instruction room terminal 14 via the server 11, the speaker 38 can output the sound of the operating room 15 to the instruction room 16. In addition, the speaker 38 may be provided in either the monitor 36 or the main body of the instruction room terminal 14, or may be provided independently of the monitor 36 and the main body of the instruction room terminal 14. The sound of the speaker 38 can be switched by operating the operation unit 35.

[0085] (Instructions for participating room terminals)

[0086] like Figure 1E As shown, the participant room terminal 43 is used in the participant room 116. The participant room terminal 43 is a computer used by participants, such as trainees who learn the entire medical practice, novice surgeons, etc. The medical practice has nothing to do with whether or not the operation is in progress. Figure 1A As shown, the participant room terminal 43 is connected to the operating room terminal 13 and the instruction room terminal 14 via the server 11. The participant room terminal 43 can send a signal containing information to the operating room terminal 13, and can receive a signal containing information from the operating room terminal 13. The participant room terminal 43 can send a signal containing information to the instruction room terminal 14, and can receive a signal containing information from the instruction room terminal 14. The participant room terminal 43 can be any one of a fixed terminal, a mobile terminal, and a portable terminal, or a unit composed of a plurality of these terminals.

[0087] The room terminal 43 includes a control circuit 139, a storage unit 140, and a communication unit 141. The control circuit 139 includes a calculation circuit for performing various processes, comparisons, and calculations, a memory for temporarily storing information, and the like. The storage unit 140 stores non-temporary programs, data, information, and the like used in the processes and judgments performed by the control circuit 139.

[0088] The participant room terminal 43 is connected to the operation unit 135, the monitor 136, the camera 137, the microphone 144, and the speaker 138 in a manner that enables signal transmission and reception. The communication unit 141 receives a signal transmitted from the server 11 via the network 12. The signal received by the communication unit 141 is input to the control circuit 139. In addition, the signal output from the communication unit 141 is transmitted to the server 11 via the network 12. The signal input to the communication unit 141 and the signal output from the communication unit 141 include various information, such as video, image, sound, data, folder, and file.

[0089] The monitor 136 constructs a display screen through a touch panel. The touch panel may be any of an electrostatic capacitance method, a resistive film method, and an infrared method. The monitor 136 and the participant room terminal 43 are connected through a communication line such as HDMI. Signals containing various information are exchanged between the monitor 136 and the participant room terminal 43.

[0090] The operation unit 135 includes a keyboard, a mouse, a display screen of a touch panel, a touch pen, etc. The operation unit 135 is connected to the participant room terminal 43. When the participant operates the operation unit 135, a signal corresponding to the operation is input to the participant room terminal 43. The display screen of the operation unit 135 and the display screen of the monitor 136 can be set separately or combined. The participant room terminal 43 obtains information containing evaluation criteria required for evaluation from the server 11. By operating the operation unit 135, the participant can send the evaluation of the guidance and annotations obtained from the guidance room terminal 14 to the server 11. The evaluation can be sent for each operation, each surgical content, each disease condition, each surgeon, and each participant. In addition, the evaluation criteria can be any of a predetermined stage-by-stage evaluation, an evaluation based on a predetermined standard text, an evaluation set by the participant alone, etc. Furthermore, the participant can also operate the operation unit 135 to send a desired pattern to the server 11 for each surgery, each disease condition, each surgical content, each operating doctor, or each participant.

[0091] The camera 137 converts the image captured in the participant room 116 into a signal and outputs it. The camera 137 is connected to the participant room terminal 43, for example, via HDMI. The camera 137 can also be set in the guidance room in any manner such as a fixed type, a movable type, or a staff member holding it by hand. The microphone 144 is set in the participant room 116 to obtain the sound of the participant room 116. The sound obtained by the microphone 144 is processed by the participant room terminal 43. The speaker 138 is connected to the participant room terminal 43. A signal containing sound is output from the participant room terminal 43, and the speaker 38 outputs the sound in the participant room 116. In addition, the speaker 138 can be set in any of the monitor 136 and the main body of the participant room terminal 43, or can be set independently from the monitor 136 and the main body of the participant room terminal 43.

[0092] In addition, the participant room terminal 43 sends a signal including information about the participant room terminal 43 to the server 11. The information about the participant room terminal 43 includes an identification number inherent to the participant room terminal 43, a password of the participant, an authentication ID of the participant, and the like. The server 11 processes the identification number inherent to the participant room terminal 43, the password of the participant, the authentication ID of the participant, and the like. However, when the server 11 processes the identification number inherent to the participant room terminal 43, the password of the participant, the authentication ID of the participant, and the like, information can be mutually communicated between the server 11 and the participant room terminal 43.

[0093] (Description of the processing terminal)

[0094] The processing terminal 81 is an example of a computer having a function of processing the medical treatment remuneration of the instructor when the instructor's behavior is recognized as a medical behavior. The processing terminal 81 has: a control circuit 87, an operating unit 88, a display unit 89, a storage unit 90, and a communication unit 91. The control circuit 87 has a function of processing the operation content of the operating unit 88 and a function of controlling the information displayed by the display unit 89. The operating unit 88, the display unit 89, the storage unit 90, and the communication unit 91 are connected to the control circuit 87. The display unit 89 is, for example, a monitor. The operating unit 88 includes a keyboard, a mouse, a touch panel, etc. In the storage unit 90, software, programs, data, etc. used in the processing and judgment performed by the control circuit 87 are stored. The communication unit 91 is connected to the server 11 via the network 12.

[0095] In the present disclosure, the processing terminal 81 can process the medical treatment remuneration of the instructor based on the instructor's behavior of remotely instructing the operating room terminal 13 using the instruction room terminal 14. The processing terminal 81 can obtain the auxiliary information generated by the auxiliary information processing unit 86 of the server 11 from the server 11. In addition, the processing terminal 81 may also have a function of databaseizing the completion history of the seminars that the novice surgeons attended in order to obtain the certification doctor.

[0096] (Functions and usage examples of remote guidance system)

[0097] When the patient lies on the operating table in the operating room 15, the operation personnel basically perform the operation on the patient manually. The image and sound of the operating room 15 are processed by the operating room terminal 13, such as Figure 2A As shown, a plurality of display units 34A, 34B, 34C, and 34D can be divided and displayed on the display screen 34 of the monitor 27. The video of the first camera 24 displayed on the display unit 34A is a 2D video and a 3D video.

[0098] The images displayed on the display units 34A, 34B, 34C, and 34D include the standing position of the operation personnel, the posture of the operation personnel, the hand movements of the operation personnel, the placement position of the instruments, the direction of the placement of the instruments, the operation direction of the instruments relative to the patient's affected part, the operation position of the instruments relative to the patient's affected part, etc. The operation personnel include the surgeon, nurses, assistants, etc. The instruments include the operating table, the imaging device, the image diagnosis device, the measuring device, the endoscope, the forceps, the tweezers, the scissors, the cutters, the needle holders, etc.

[0099] The size and display position of the display units 34A, 34B, 34C, and 34D can be arbitrarily set by operating the operation menu. In addition, when the operation unit 30 is operated, the images captured by the surgical field camera 23, the first camera 24, and the second camera 25 and the sound of the microphone 29 can be sent to the guidance room terminal 14 via the network 12.

[0100] The guidance room terminal 14 receives information including video, image, sound, and data. Display units 42A, 42B, 42C, and 42D can be displayed on the display screen 42 of the monitor 36. The video of the display unit 42A is the same as the video of the display unit 34A, the video of the display unit 42B is the same as the video of the display unit 34B, the video of the display unit 42C is the same as the video of the display unit 34C, and the video of the display unit 42D is the same as the video of the display unit 34D. The video displayed on the display screen 42 is a 2D video and a 3D video.

[0101] The presence, size, and display position of the display units 42A, 42B, 42C, and 42D can be set by operating the operation menu of the display screen 42 or the operation of the operation unit 35. In addition, the speaker 38 of the instruction room 16 can output the sound sent from the operating room 15. In addition, for the image of the display unit 42A displayed on the display screen 42, the instructor can add and display annotations and pointers overlapping the image of the display unit 42A by bringing the front end of the touch pen into contact with the display screen 42. Annotations are instructions, methods, sequences, positions, etc. related to the surgery, and they can be mainly represented by line-based drawing B1. In addition, a pointer without drawing B1 can also be displayed.

[0102] The dedicated menu 42E is operated when the annotation and pointer of the drawing B1 are displayed in the image displayed on the display screen. The dedicated menu 42E includes a timer (delayed display), line density, rectangle, circle, snapshot, recording, chat, whiteboard, hand raising, drawing priority control, line color, line thickness, type of fixed pattern represented by line, partial erase, full erase, etc. The instructor can give the image of the display unit 42A a drawing B1, which is a drawing B1 that gives a curve or a straight line, a drawing B1 that gives an arrow, or a drawing B1 based on a circle that surrounds a given part. In addition, in addition to displaying the annotation and pointer of the drawing B1 in the image of the display unit 42A, the instructor can also provide guidance based on voice. The guidance based on voice is picked up by the microphone 44.

[0103] The depiction B1 on the display unit 42A includes the incision site of the patient's affected part, the incision method, the incision direction, the position of the forceps to stretch the tissue, the stretching direction, the exposure method of the peeling surface based on the stretching, and various treatments, etc. The voice-based guidance can provide the operator with an explanation of important guidance content for the surgery in real time through verbal explanations corresponding to the depiction B1, explanations of technical meanings and reasons, and comprehensive knowledge and thoughts including the patient's QOL (Quality of life).

[0104] The instruction room terminal 14 transmits the drawing B1 given on the display screen 42, the sound picked up by the microphone 44, and the image captured by the camera 37 to the operating room terminal 13 via the network 12. The operating room terminal 13 displays the annotations and pointers of the drawing B1 sent from the instruction room terminal 14, for example, Figure 2A In this way, the image B2 is displayed on the display unit 34A in an overlapping manner. In addition, the operating room terminal 13 outputs the voice guidance sent from the guidance room terminal 14 from the speaker 28. Therefore, the subject in the operating room 15 can receive guidance related to the surgery from the instructor in real time while the surgery is being performed.

[0105] The interface and compression transmission technology in the remote guidance system 10 of the present disclosure can realize two-way communication. Therefore, appropriate guidance and instructions related to the treatment performed in the operating room 15 can be sent in real time from the guidance room 16. Therefore, the treatment in the operating room 15 can proceed smoothly. Specifically, the knowledge and experience of experienced surgeons can be applied to the actual cases of novice surgeons for guidance.

[0106] Furthermore, when the operating room 15 and the instruction room 16 are separated in different places, the instructor who lives in the local area can instruct the subject without visiting the place (for example, the urban area) where the operating room 15 is located. In contrast to this environment, when the person who receives instruction in the operating room 15 lives in a medically sparse area such as a remote area or an island area, and is a novice surgeon in charge of medical treatment in the area, he or she can also receive instruction from the instructor in the instruction room 16.

[0107] Furthermore, the operator can give annotations based on the drawing to the monitor 27. The operation of the operator giving annotations to the monitor 27 is the same as the operation of the instructor giving annotations to the monitor 36. A signal including the annotations given to the monitor 27 is transmitted from the operating room terminal 13 to the server 11. The server 11 can transmit the annotations given to the monitor 27 to the instruction room terminal 14 and the participant room terminal 43.

[0108] Furthermore, the participant can give annotations based on the drawing to the monitor 136. The operation of the participant giving annotations to the monitor 136 is the same as the operation of the instructor giving annotations to the monitor 36. A signal including the annotations given to the monitor 136 is transmitted from the participant room terminal 43 to the server 11. The server 11 can transmit the annotations given to the monitor 136 to the instruction room terminal 14 and the operating room terminal 13.

[0109] (Second specific example)

[0110] The second specific example is an example of performing surgery on a patient in an operating room and providing surgical guidance in real time from a location away from the operating room. More specifically, an example of performing surgery on a patient using a medical robot system while the patient is lying on an operating table.

[0111] (structure)

[0112] like Figure 3As shown, in the operating room 15, a medical robot system 45, a first camera 24 and a second camera 25, a microphone 29, an operating room terminal 13, a monitor 27, a speaker 28, and an operation unit 30 are provided. The medical robot system 45 includes an operation unit (console) 46, a video cart 47, a robot body (bedside robot arm system: Patient Cart) 48, etc. The operation unit 46 includes a main controller 49, a monitor 50, a microphone 51, a speaker 52, a chair, and operation buttons, etc. When the surgeon operates the main controller 49, the display screens of the monitors 50 and 55 can be switched between display and non-display, the display content of the display screen can be controlled, and the motion / function of the robot body 48 can be controlled. In accordance with the motion of the surgeon's hand operating the main controller 49, the motion arm of the robot body 48 moves. The medical robot system 45 and the operating room terminal 13 are connected to enable bidirectional communication.

[0113] The operating unit 46 is operated by the surgeon sitting on the chair. The monitor 50 can display 3D images, 2D images, cross-sectional images, etc. The cross-sectional images include CT (Computed Tomography) images and MRI (Magnetic Resonance Imaging) images acquired before the operation. In addition, the sound data of the microphone 51 is sent to the operating room terminal 13. Furthermore, the speaker 52 outputs the sound sent from the operating room terminal 13. The robot body 48 has multiple action arms and a surgical field camera 53. Surgical instruments are respectively installed in the multiple action arms.

[0114] When the surgeon sits on the chair and operates the main controller 49 and various operation buttons, the operation signal is output from the operating unit 46, and the robot body 48 functions. The image of the patient's affected part taken by the surgical field camera 53 is sent to the operating room terminal 13. In addition, the signal of the sound picked up by the microphone 51 is sent to the operating room terminal 13. The sound sent from the operating room terminal 13 is output from the speaker 38 via the network 12.

[0115] The vision cart 47 has a monitor 55 and a computer 56. The computer 56 is connected to the operating room terminal 13, the operating unit 46, and the robot body 48 in a bidirectional communication manner, and is connected to the monitor 55. The computer 56 processes the signal sent from the operating room terminal 13, the signal sent from the operating unit 46, and the signal sent from the robot body 48. The signal sent from the operating room terminal 13 to the computer 56 includes information such as video and audio guidance, and annotations depicting B1.

[0116] In addition, the computer 56 outputs a signal of an image displayed by the monitor 50 of the operation unit 46, a signal of a sound output by the speaker 52 of the operation unit 46, and a signal of an image displayed by the monitor 55 of the visual cart 47. The monitor 50 can display an image of the surgical field camera 53, an image of the first camera 24, and an image of the second camera 25. In addition, the monitor 50 can display an image of the camera 37 transmitted from the server 11. The monitor 50 can switch between 3D images and 2D images for display. Therefore, when the surgeon wants to visually view the 3D image, there is no need to move the line of sight.

[0117] In addition, Figure 3 In the embodiment, the monitor 27 may not be provided. Figure 3 The other structures shown in FIG. 5 are the same as those shown in FIG. 1 . In addition, the computer 56 processes the signal including the operation content and the motion content of the operating unit 46, and sends the control signal to the robot body 48. Furthermore, the computer 56 processes the signal including the motion content of the robot body 48 and the signal including the operation content and the motion content of the operating unit 46, and sends the processing result to the server 11 via the operating room terminal 13.

[0118] (Operation Example of the Second Specific Example)

[0119] When performing surgery using the medical robot system 45, the image and sound of the operating room 15 are transmitted to the guidance room terminal 14 via the operating room terminal 13 and the network 12. Then, the image of the surgical field camera 53 is displayed on the Figure 2B The display unit 42A displays the image of the first camera 24, and the display unit 42B displays the image of the second camera 25. In the instruction room 16, an annotation depicting B1 is added to the image of the display unit 42A, and audio instructions are given.

[0120] The instruction room terminal 14 outputs a signal including annotations of the drawing B1 and a signal including voice instructions, and the signal is sent to the medical robot system 45 via the network 12 and the operating room terminal 13. Therefore, when performing surgery using the medical robot system 45, the drawing B1 sent from the instruction room terminal 14 can be processed as described above. Figure 2A Thus, the image B2 is displayed on the display unit 34A of the monitor 50. Figure 2A The example in which the display units 34A, 34B, 34C, and 34D are displayed on the display screen 34 of the monitor 50 is shown. In addition, the sound sent from the guidance room terminal 14 is output from the speaker 52 of the operation unit 46. Therefore, the second specific example can achieve the same effect as the first specific example.

[0121] Furthermore, the operator can give annotations based on the drawing to the monitors 50 and 55. The operation of the operator giving annotations to the monitors 50 and 55 is the same as the operation of the instructor giving annotations to the monitor 36. The signal containing the annotations given to the monitors 50 and 55 is transmitted to the server 11 via the visual cart 47 and the operating room terminal 13. The server 11 can transmit the annotations given to the monitors 50 and 55 to the instruction room terminal 14 and the participant room terminal 43. Figure 2A As shown, the monitors 50 and 55 can display the same display screen 34 as the display screen 34 of the monitor 27. That is, the display screen 34 of the monitors 50 and 55 can switch between 2D images and 3D images. In addition, the display screen 34 of the monitors 50 and 55 can also display only annotations.

[0122] (Application example)

[0123] The processing such as coordinate conversion that causes the drawing B1 and the pointer given to the 2D image or 3D image of the display unit 42A of the monitor 36 of the instruction room 16 to be displayed on the 3D image displayed on the monitor 27, 55 of the operating room 15 through the drawing B2 and the pointer can also be performed by any one of the control circuit 39 of the instruction room terminal 14 and the control circuit 31 of the operating room terminal 13. The size of each display unit 42A, 42B, 42C, 42D displayed on the display screen 42 of the monitor 36 and the position of each display unit 42A, 42B, 42C, 42D can be arbitrarily changed by operating the operation unit 35 or the operation menu. For example, when the image of the operating room 15 is divided into a plurality of parts at a given time interval in a time series, the size of the display unit 42A, 42B can be made different according to each division.

[0124] Furthermore, it is also possible to have an environment in which a plurality of instruction room terminals 14 are installed in a single instruction room 16, or an environment in which instruction room terminals 14 are installed in each of the plurality of instruction rooms 16. In this case, the plurality of instruction rooms 16 may be located in different places. Furthermore, it is also possible to make the processing that can be performed by a specific instruction room terminal 14 among the plurality of instruction room terminals 14 different from the processing that can be performed by other instruction room terminals 14. For example, it is also possible to perform processing such as playback, stop, temporary stop, fast forward, and fast rewind of the recorded image performed in the operating room 15 only by a specific instruction room terminal 14.

[0125] Furthermore, when the monitors 27, 50, and 55 installed in the operating room 15 display the images of the monitors 36 connected to each of the plurality of instruction room terminals 14, the plurality of monitors 36 may be distinguished from each other by using identification marks or color distinctions for each different monitor 36. Furthermore, when sound is input to the microphone 44 connected to the instruction room terminal 14 to which the monitor 36 is connected, display processing such as surrounding the display image of the corresponding monitor 36 with a colored frame may be performed so that any instruction room terminal 14 can be identified from the plurality of instruction room terminals 14. Furthermore, the annotation depicting B1 may be assigned through the monitor 36 of any one of the plurality of instruction room terminals 14, and the monitors 36 of the other instruction room terminals 14 may only display images.

[0126] The process of making the color of the drawing displayed on the monitors 27, 50, 55 of the operating room 15 different for each of the plurality of monitors 36 can be performed by either the control circuit 17 of the server 11 or the control circuit 31 of the operating room terminal 13. In addition, when a plurality of monitors 36 are provided, the control circuit 17 of the server 11 can also perform a process of limiting the selection of the same drawing color on each monitor 36. Furthermore, the control circuit 31 of the operating room terminal 13 can also perform the following process: displaying which of the plurality of monitors 36 the drawing displayed on the monitors 27, 50, 55 of the operating room terminal 13 is performed.

[0127] Furthermore, the control circuit 17 of the server 11 can also perform the following processing: the images displayed on the monitors 27, 50, 55 of the operating room 15 and the sounds output by the speakers 28, 52 of the operating room 15 are distributed in real time or recorded to the participating room terminal 43 different from the operating room terminal 13 and the instruction room terminal 14 via the network 12. The participating room terminal 43 can receive 2D images, 3D images and sounds from the server and reproduce the images and sounds. For convenience, the participating room terminal 43 is set not to send images and sounds to the server 11. The participating room terminal 43 includes a fixed terminal, a mobile terminal, a portable terminal, a goggle-type terminal, and a projector.

[0128] Furthermore, if Figure 2BAs shown, the display portion T1 showing the time when the signal depicting the annotation of B1 is transmitted from the instruction room terminal 14, the display portion T2 showing the time when the signal is transmitted to the operating room terminal 13 via the server 11 and then returns to the instruction room terminal 14 via the server 11, and the display portion T3 showing the time required for one-way signal transmission between the instruction room terminal 14 and the operating room terminal 13 may be displayed at the corners of the display screen 42 of the monitor 36. Here, there is a time difference between the time when the signal depicting the annotation of B1 is transmitted from the instruction room terminal 14 and the time when the signal is transmitted to the operating room terminal 13 via the server 11 and then returns to the instruction room terminal 14 via the server 11.

[0129] Furthermore, the subject can also send the video images of the actions related to the surgery from the operating room terminal 13 to the instruction room terminal 14 without performing any actions related to the surgery. In this case, the operating room terminal 13 may not be set in the operating room 15. In addition, the video images are stored in the folder of the storage unit 32 of the operating room terminal 13. The folder storing the video images is classified according to the body part, case, surgeon, operation method, date and time, operating room number (operation instruction), etc. and stored in the storage unit 32. After that, the video images are sent from the operating room terminal 13 to the instruction room terminal 14 via the server 11, and the video images are displayed on the display screen 42 of the monitor 36, and the subject can also receive instructions from the instructor. That is, the video images can be shared by both the operating room terminal 13 and the instruction room terminal 14.

[0130] The instructor gives the same annotations describing B1 and voice guidance to the display screen 42 of the monitor 36 as in the case of a real-time operation. In this case, an operation menu (operation button) such as playback, stop, temporary stop, fast forward, and fast rewind of the recorded video is displayed on the monitor 27 of the operating room 15. In addition, it may be configured so that these operations can be performed on at least one of the monitors 27, 50, and 55 and the monitor 36. Furthermore, the operating room terminal 13 may also have permission to grant permission for the above-mentioned operations from the operating room terminal 13 to the instruction room terminal 14.

[0131] (Overview of control example)

[0132] Figure 4 1 is a flowchart that generally shows the control example described in the first specific example, the second specific example, and the application example. Figure 4 , the steps including processing and judgment performed by the server 11, the operating room terminal 13, the instruction room terminal 14, and the participant room terminal 43 are shown. The server 11, the operating room terminal 13, the instruction room terminal 14, and the participant room terminal 43 can also be connected to the server 11, the operating room terminal 13, the instruction room terminal 14, and the participant room terminal 43. Figure 4 The steps may be performed in a different order than the order shown.

[0133] In step S10, the operating room terminal 13 displays an image of an action related to the surgery on at least one of the monitors 27, 50, and 55. In step S10, the operating room terminal 13 transmits a signal including information of the operating room terminal 13 to the server 11. The information of the operating room terminal 13 includes an identification number inherent to the operating room terminal 13, a password of the operating doctor, and an authentication ID of the operating doctor.

[0134] On the other hand, in step S11, the instruction room terminal 14 causes the monitor 36 to display the image captured by the camera 37. In addition, in step S11, the instruction room terminal 14 transmits a signal including information of the instruction room terminal 14 to the server 11. The information of the instruction room terminal 14 includes an identification number inherent to the instruction room terminal 14, a password of the instructor, and an authentication ID of the instructor. In step S12, the server 11 receives a signal transmitted from the operating room terminal 13 and a signal transmitted from the instruction room terminal 14, and performs authentication of the operating room terminal 13 and authentication of the instruction room terminal 14. In step S13, the server 11 performs a process of transmitting a signal received from the operating room terminal 13 to the instruction room terminal 14, and a process of transmitting a signal received from the instruction room terminal 14 to the operating room terminal 13.

[0135] In step S14, the operating room terminal 13 obtains information sent from the server 11. In step S15, the instruction room terminal 14 obtains information sent from the server 11. When the processing of step S14 is performed by the operating room terminal 13 and the processing of step S15 is performed by the instruction room terminal 14, bidirectional communication can be directly performed between the operating room terminal 13 authenticated by the server 11 and the instruction room terminal 14, bypassing the server 11.

[0136] In step S18, the operating room terminal 13 displays the image sent from the instruction room terminal 14 on at least one of the monitors 27, 50, and 55, and transmits information including the image of the operating room 15 and information including sound to the server 11. In step S19, the instruction room terminal 14 displays the image sent from the operating room terminal 13 on the monitor 36, and transmits information including the image of the instruction room 16 and information including sound to the server 11. In step S20, the server 11 receives the signal sent from the operating room terminal 13 and stores it in the storage unit 22, and receives the signal sent from the instruction room terminal 14 and stores it in the storage unit 22. Furthermore, in step S21, the server 11 transmits the signal including the image and sound to the operating room terminal 13, and transmits the signal including the image and sound to the instruction room terminal 14.

[0137] In step S22, the operating room terminal 13 receives the image sent from the server 11 and displays it on at least one of the monitors 27, 50, and 55. In step S23, the instruction room terminal 14 receives the image sent from the server 11 and displays it on the monitor 36. In step S24, the instructor adds annotations to the image displayed on the monitor 36, for example, the drawing B1 of FIG. 2 (B). In step S24, the instructor gives voice guidance from the microphone 44. Then, the instruction room terminal 14 processes the signal of the annotation of the drawing B1 added in step S24, and sends a signal corresponding to the annotation of the drawing B1 added and a signal corresponding to the voice guidance to the server 11 in step S25.

[0138] The server 11 receives the signal from the instruction room terminal 14 in step S26, and transmits the signal to the operating room terminal 13 in step S27. In step S28, the operating room terminal 13 receives the signal from the server 11. The operating room terminal 13 processes the received signal in step S29, and transmits the annotation of the description B1 to the operating room terminal 13. Figure 2A Thus, it is displayed on at least one of the monitors 27, 50, and 55. In addition, the guidance by voice is output from at least one of the speakers 28 and 52. In addition, in step S50, the operating room terminal 13 can give the annotation without acquiring the information of the guidance room terminal 14.

[0139] Furthermore, the server 11 receives a signal including the video and sound of the operating room 15 and a signal including the video and sound of the instruction room 16 in step S20, so when the participant room terminal 43 is directly connected to the server 11 via the network 12 at the time of step S20, it is possible to publish in real time to the participant room terminal 43. In addition, the server 11 can store the information received in step S20 in the storage unit 22. Therefore, after the server 11 performs the processing of step S20, when the participant room terminal 43 is connected to the server 11, the user of the participant room terminal 43 can use the information stored in the storage unit 22 of the server 11 through video on demand. In addition, even if the operation is not performed in real time, the operating room terminal 13 can be connected to the server 11 to use the information stored in the storage unit 22 through video on demand.

[0140] Furthermore, when recording the video of the operating room 15 in the storage unit 22 of the server 11 or broadcasting the video of the operating room 15 to the participant room terminal 43 at the same time, the second control can be selected. Figure 4Although not shown in the figure, the first control can also be selected to perform the processing of steps S18 to S28 between the operating room terminal 13 and the guidance room terminal 14, bypassing the server 11. In this case, the processing of steps S18 to S28 is skipped. Figure 4 The program for switching the first control and the second control is stored in the storage unit 32 of the operating room terminal 13, the storage unit 22 of the control circuit 17 of the server 11, and the storage unit 40 of the guidance room terminal 14, respectively.

[0141] Further, in Figure 4 In step S27, the server 11 can send various information to the participant room terminal 43. The various information sent by the server 11 to the participant room terminal 43 includes information obtained from the operating room terminal 13 and information obtained from the instruction room terminal 14. In step S30, the participant room terminal 43 can receive various information from the server 11. The participant room terminal 43 can cause the monitor 136 to display the information received from the server 11 in step S30, such as video and image. In addition, the participant room terminal 43 can cause the speaker 138 to output the information received from the server 11 in step S30, such as sound.

[0142] Furthermore, in step S31, the participant room terminal 43 can send various information to the server 11. The server 11 can receive various information from the participant room terminal 43 in step S26, and store the processing results of the various information in the storage unit 22. The server 11 can send the various information obtained from the participant room terminal 43 in step S13 to the operating room terminal 13 and the instruction room terminal 14.

[0143] Server 11 is in progress Figure 4 The processing process of each step shown in Fig.10 In the process of the processing of each step shown, the following processing is performed: various information obtained from the operating room terminal 13, the instruction room terminal 14, and the participant room terminal 43, and information sent to the operating room terminal 13, the instruction room terminal 14, and the participant room terminal 43 are stored in the database of the storage unit 22, and the history of updating the database is created. For example, necessary information including the year, month, day and date and time of the operation, the name of the person responsible for the operation, the name of the surgeon, the name of the assistant in the operating room 15, the name of the instructor, the name of the participant, the patient's disease name and medical records, the patient's surgical site, the surgical method, the equipment used, etc. are databased, and a summary form is created, and the summary form is stored in the storage unit 22. The necessary information stored in the storage unit 22 is distinguished by the distinguishing mark given to each operation.

[0144] These information are input to the operating room terminal 13 or the instruction room terminal 14, and are transmitted from the operating room terminal 13 or the instruction room terminal 14 to the server 11. In addition, various information generated during the operation, such as images, pictures, instructions, comments, sounds, the movement of the robot body 48, and the operation content of the main controller 49 are stored in the storage unit 22 in association with the aforementioned distinguishing marks. Therefore, by inputting the distinguishing mark or inputting a part of the necessary information in any one of the operating room terminal 13, the instruction room terminal 14, and the participant room terminal 43, the target information can be retrieved. Furthermore, the information including the operation content of the main controller 49 and the movement of the robot body 48 is stored in the storage unit 22 in association with the time code encoding the time information of the time when the annotation was generated.

[0145] In addition, the control circuit 17 of the server 11 can classify various information according to each different instructor, each different surgeon, each different disease, each different surgical method, each different case, and each different part of the patient's body, and store it in a storage area such as the storage unit 22. In addition, the control circuit 17 of the server 11 has a function of generating learning data for guidance based on the information and data stored in the storage unit 22. The generated learning data is stored in the storage unit 22. The learning data for guidance is used for guidance for performing surgery using the medical robot system 45.

[0146] Furthermore, the auxiliary information processing unit 86 of the control circuit 17 has a function of generating auxiliary information associated with the medical treatment remuneration of the instructor based on the information and data stored in the storage unit 22. The auxiliary information of the medical treatment remuneration includes the year, month, and day of the guidance behavior performed remotely by the instructor, the elapsed time from the start to the end of the guidance, and the content of the guidance behavior performed remotely by the instructor. In addition, the auxiliary information includes scores and points calculated based on the elapsed time from the start to the end of the guidance, the content of the guidance behavior performed remotely by the instructor, etc. In addition to the patient's disease name and case, the patient's surgical site, the surgical method, etc. in the actual operation, the guidance behavior also includes explanations to trainees and novice surgeons. A third party can use the auxiliary information generated by the server 11. In the case where remote guidance becomes the subject of insurance coverage, the certification information of novice doctors generated by the server 11 can be used for the calculation of medical treatment remuneration received by the instructor.

[0147] (Supplementary explanation)

[0148] The remote guidance device, remote guidance method, and remote guidance program described in this embodiment are not limited to the above-mentioned embodiments. For example, each camera can also use a 360-degree camera. The monitor can also be set in VR (Virtual Reality) goggles mounted on the face, or HMD (Head Mounted Display) mounted on the head. Mobile terminals and portable terminals include smart phones and tablet terminals.

[0149] The computer disclosed herein includes personal computers, workstations, mainframe computers, mainframe racks, supercomputers, etc. In addition, the network connecting the server, operating room terminal, and instruction room terminal in a bidirectional communication manner may be any one of the Internet, intranet, and extranet.

[0150] The first place is a place where the operating room terminal 13 is set, and the first place can be, for example, any one of an operating room, a lecture room, a hall, a space, a room, etc. The second place is a place where the guidance room terminal 14 is set, and the second place can be, for example, any one of an operating room, a lecture room, a hall, a waiting room, a remote guidance room, a space, a room, etc. In addition, the second place is not limited to any one place, and can also be multiple places. The computer possessed by the server can be either a single electrical component or an electronic component, or a unit having multiple electrical components or multiple electronic components. The electrical component or the electronic component includes a processor, a control circuit, and a module. The computer possessed by the server can be either singular or plural. In addition, the communication unit and the storage unit can also be another component or another device that can be connected to the computer in a communicative manner.

[0151] In addition, the remote guidance system 10 can also skip the following configuration: Figure 4 Steps S18, S19, S20, S21, S22, and S23 are shown in both an environment where the participant room terminal 43 is not connected to the server 11 to receive real-time delivery, and an environment where the participant room terminal 43 is connected to the server 11 to use the information stored in the storage unit 22 through video on demand.

[0152] (3rd specific example)

[0153] like Figure 1B As shown, the switching unit 105 includes a proctoring mode circuit 120, a demonstration mode circuit 121, and a full mode circuit 122. The proctoring mode circuit 120 activates the proctoring mode, the demonstration mode circuit 121 activates the demonstration mode, and the full mode circuit 122 activates the full mode circuit. The activation and release of the proctoring mode circuit 120, the demonstration mode circuit 121, and the full mode circuit 122 are switched by the operation of the instruction room terminal 14.

[0154] An example of a screen displayed on the monitor 36 of the guidance room terminal 14 is shown in FIG. Figure 5 The monitor 36 displays a monitoring mode icon 201, a demonstration mode icon 202, a full mode icon 203, a participant screen 204, a main screen 205, and a sub-screen 206. The monitoring mode icon 201, the demonstration mode icon 202, and the full mode icon 203 generate a mode selection unit. When the monitoring mode icon 201, the demonstration mode icon 202, and the full mode icon 203 are operated, the auxiliary information processing unit 86 generates a mode start request.

[0155] When the monitor mode icon 201 is operated, the monitor mode circuit 120 is activated. When the demo mode icon 202 is operated, the demo mode circuit 121 is activated. When the full mode icon 203 is operated, the full mode circuit 122 is activated.

[0156] The color of the icon of the activated mode, that is, the active mode, is displayed in a darker color than the icon of the inactive mode. In addition, the active mode is displayed in a design that allows you to know that it is selected. This is to clearly indicate the currently activated mode.

[0157] The main screen 205 is a screen that displays images and pictures acquired from the operating room 15. The main screen 205 mainly displays images and pictures captured by the surgical field camera 23, or images of the body cut by a cutting knife, etc. The main screen 205 may also display academic materials or explanation materials stored in the storage unit 32 of the operating room terminal 13. In addition, the main screen 205 may also display materials stored in the cloud. The sub-screen 206 is a screen that displays images and pictures captured by the first camera 24 and the second camera 25.

[0158] The monitoring mode circuit 120 starts the monitoring mode. The monitoring mode is a mode selected when, for example, an instructor located in the instruction room 16, such as a certified physician (expert physician (surgeon)) provides surgical guidance to a surgeon located in the operating room 15 and a participant located in the participant room 116. The demonstration mode circuit 121 starts the demonstration mode. The demonstration mode is a mode for, for example, a certified physician (expert physician (surgeon)) as an instructor to guide the surgeon to improve his / her skills. The full mode circuit 122 starts the full mode. The full mode is a mode in which, for example, all persons including an instructor, a surgeon, and a participant can share a conversation.

[0159] When the monitoring mode is activated, bidirectional communication of information including sound can be performed between the operating room terminal 13 and the instruction room terminal 14. When the monitoring mode is activated, unidirectional communication of information including sound can be performed from the operating room terminal 13 to the participant room terminal 43. When the monitoring mode is activated, unidirectional communication of information including sound can be performed from the instruction room terminal 14 to the participant room terminal 43. When the monitoring mode is activated, information including sound cannot be sent from the participant room terminal 43 to the instruction room terminal 14, and information including sound cannot be sent from the participant room terminal 43 to the operating room terminal 13.

[0160] Furthermore, when the monitoring mode is activated, information including annotations can be sent from the instruction room terminal 14 to the operating room terminal 13, and information including annotations can be sent from the instruction room terminal 14 to the participant room terminal 43. When the monitoring mode is activated, information including annotations cannot be sent from the participant room terminal 43 to the operating room terminal 13, and information including annotations cannot be sent from the participant room terminal 43 to the instruction room terminal 14.

[0161] Therefore, when the monitoring mode is activated, the surgeon cannot see the exchange of comments made by the instructor and the participant. That is, the surgeon in the operating room 15 cannot hear the voice information between the instructor in the instruction room 16 and the participant in the participant room 116. Therefore, the surgeon can concentrate on the operation. On the other hand, the instructor in the instruction room 16 and the participant in the participant room 116 can freely talk about the advantages and disadvantages of the current operation without putting pressure on the surgeon.

[0162] When the demonstration mode is activated, two-way communication of information including sound is possible between the participant room terminal 43 and the instruction room terminal 14. When the demonstration mode is activated, one-way communication of information including sound is possible from the operating room terminal 13 to the instruction room terminal 14. When the demonstration mode is activated, information including sound cannot be sent from the instruction room terminal 14 to the operating room terminal 13.

[0163] Furthermore, when the demonstration mode is activated, bidirectional communication of information including annotations can be performed between the instruction room terminal 14 and the participant room terminal 43. When the demonstration mode is activated, information including annotations cannot be sent from the participant room terminal 43 to the operating room terminal 13, and information including annotations cannot be sent from the operating room terminal 13 to the participant room terminal 43.

[0164] When the full mode is activated, two-way communication of information including sound can be performed between the participant room terminal 43 and the instruction room terminal 14, and two-way communication of information including sound can be performed between the participant room terminal 43 and the operating room terminal 13. When the full mode is activated, two-way communication of information including annotation can be performed between the participant room terminal 43 and the instruction room terminal 14, and two-way communication of information including annotation can be performed between the participant room terminal 43 and the operating room terminal 13.

[0165] In addition, when any mode is activated, bidirectional communication of information including images can be performed between the operating room terminal 13 and the instruction room terminal 14. In addition, when any mode is activated, bidirectional communication of information including images can be performed between the operating room terminal 13 and the participant room terminal 43. Furthermore, when any mode is activated, bidirectional communication of information including images can be performed between the instruction room terminal 14 and the participant room terminal 43.

[0166] Figure 6 1 is a schematic diagram of mode switching. When the monitoring mode is activated, the participant P1 participates by watching the monitor 136. When the demonstration mode is selected, the participant P1 can receive instructions related to the surgery from the instructor P2. When the full mode is activated, the participant P1 can participate in the surgery while watching the exchange between the surgeon and the instructor.

[0167] Figure 7 This is a conceptual diagram of the API (Application Programming Interface) when the monitoring mode is activated. Figure 8 This is a conceptual diagram of the API when the demo mode is enabled. Fig. 9 This is a conceptual diagram of the API when full mode is enabled.

[0168] Fig.10 This is a sequence of mode changes in one embodiment of the present disclosure. In the instruction room terminal 14, in step S40, the mode icon of the monitor 36 is operated. The instructor can operate the instruction room terminal 14 in step S40 to select a mode. The instructor can select a mode according to the first to third scenes. For example, the instructor can select a demonstration mode for the first scene. For example, the instructor can select a full mode for the second scene. For example, the instructor can select a monitoring mode for the third scene.

[0169] Then, the selected mode is transmitted from the instruction room terminal 14 to the server 11. The server 11 receives the signal of the selected mode from the instruction room terminal 14 in step S41 and stores it in the storage unit 22. Then, the switch unit 105 of the server 11 performs a process of starting the selected mode in step S42.

[0170] Then, the server 11 sends the command for controlling the activated mode to the operating room terminal 13, the instruction room terminal 14, and the participant room terminal 43. In step S43, the operating room terminal 13 displays the activated mode on the monitor 27. In addition, the monitor 55 of the visual cart 47 also displays the activated mode. Furthermore, the monitor 50 of the operation unit 46 also displays the activated mode. In step S44, the instruction room terminal 14 displays the activated mode on the monitor 36. In step S45, the participant room terminal 43 displays the activated mode on the monitor 136.

[0171] The instructor can also select each mode based on conditions different from the first scene, the second scene, and the third scene. An example is as follows.

[0172] After performing steps S41 and S42, the server 11 obtains the mode desire from the operating room terminal 13 and the participant room terminal 43 in step S38 again before performing the operation of step S40, and obtains the evaluation from the participant room terminal 43. In addition, in step S39 performed after step S38 and before step S40, the server 11 can recommend to the guidance room terminal 14 the mode in which the operation is performed most smoothly among the data stored in the storage unit 22 in the above-mentioned first support process, based on the operation content, case, surgeon, etc.

[0173] In addition, in step S39, the server 11 can recommend to the instruction room terminal 14 the mode selected at the time of performing the operation with the highest evaluation in the aforementioned second support process based on the surgical content, the case, each surgeon, the participants, etc. Further, in step S39, the server 11 can recommend the mode request received from the operating room terminal 13 or the participant room terminal 43 to the instruction room terminal 14. Therefore, in step S40, the instructor can display the evaluation and the mode request on the monitor 36 of the instruction room terminal 14. In addition, in step S40, the instructor can operate the mode icon with reference to the evaluation and the mode request to select the mode.

[0174] (Effects of Implementation Methods)

[0175] The remote guidance system 10 disclosed in the present invention can enable a novice surgeon or the like who is a participant to browse the real-time images captured during the operation, and the certified technical doctor (expert physician) can explain the points to pay attention to in the operation and provide specific suggestions for the qualified certified doctor test. Therefore, the surgical skills of novice surgeons or the like with a small number of cases can be improved. In addition, after receiving real-time guidance from the instructor one-on-one through remote control, the surgeon can reconfirm various information stored in the storage unit 22 of the server 11, such as images, pictures, and sounds, through the operating room terminal 13. In addition, if the user logs in with a permitted login ID and password, reconfirmation can be made regardless of which terminal is used. Therefore, the surgeon can obtain valuable images that can be used to take individual countermeasures for the technical certification test to examine his or her strengths and weaknesses.

[0176] However, it is painful to watch surgical images for a long time, and it is not easy to grasp the important points of the surgery. Therefore, the training of novice surgeons has become a problem. In this embodiment, when novice surgeons who are participants browse the real-time images taken during the operation, the technical certification doctors can explain the points to pay attention to in the operation and summarize the key points. Therefore, it is beneficial to improve the surgical skills of novice surgeons with a small number of cases.

[0177] In addition, the so-called summary points refer to the general points of attention in surgery. The part where the remote guidance system 10 exerts its ability means all special examples of individual patients in individual cases of individual surgeons. Such as the habits and ideas of each surgeon, it contains individual elements that cannot be corrected by passively viewing and listening to images.

[0178] According to the present disclosure, during surgery, between the operating room terminal 13, the instruction room terminal 14, and the participant room terminal 43, it is possible to switch between the other parties with which information including images, sounds, and annotations can be communicated. Therefore, it is possible to select a mode that complies with the laws and regulations for live surgery agreed upon in various societies. In addition, it is possible to suppress the pressure on the surgeon using the operating room terminal 13 from viewing and listening to the conditions of the operating room from the outside, that is, the instructor using the instruction room terminal 14 and the participant using the participant room terminal 43. In addition, according to the surgery performed by the surgeon, appropriate instructions and annotations can be provided from the instruction room terminal 14 to the operating room terminal 13.

[0179] When the switching unit 105 of the server 11 of the present invention receives a mode start request for which mode to start from the instruction room terminal 14, it starts the requested mode. Therefore, the surgeon in the operating room can concentrate on the operation, and the instructor can switch the mode according to the operation situation. The annotation processing unit 84 of the server 11 of the present invention associates the information including the movement of the robot body obtained by the robot information processing unit and the information of the annotation generated by the instruction room terminal 14, and stores them in the storage unit 22. Therefore, it is possible to generate learning data used in the case of guiding novice surgeons or less experienced surgeons to perform operations using the robot body 48.

[0180] The auxiliary information processing unit 86 of the present disclosure can generate auxiliary information for processing the medical treatment remuneration of the instructor through the processing terminal 81. In addition, the auxiliary information processing unit 86 can generate auxiliary information with characteristics that make the medical treatment remuneration different according to the type of mode activated by the switching unit 105. Therefore, in the case where the instructor's remote guidance becomes an insurance coverage object, the auxiliary information can be used, and the convenience of the user of the processing terminal 81 is improved.

[0181] An example of the technical meaning of the matters disclosed in this embodiment is as follows. The remote guidance system 10 is an example of an information processing system. The server 11 is an example of an information processing device. Monitors 27, 50, and 55 are examples of display units provided in an operating room. Microphones 29 and 51 are examples of microphones for acquiring sounds in the operating room. Speakers 28 and 52 are examples of speakers for outputting sounds acquired from outside the operating room. The operating room terminal 13, the operating unit 46, and the computer 56 are examples of operating room terminals.

[0182] The monitor 36 is an example of a display unit provided in the instruction room. The microphone 44 is an example of a microphone for acquiring sounds in the instruction room. The speaker 38 is an example of a speaker for outputting sounds acquired from outside the instruction room. The instruction room terminal 14 is an example of an instruction room terminal. The monitor 136 is an example of a display unit provided in the participant room. The microphone 144 is an example of a microphone for acquiring sounds in the participant room. The speaker 138 is an example of a speaker for outputting sounds acquired from outside the participant room. The communication unit 20 is an example of a communication unit of an information processing device. The storage unit 22 is an example of a storage unit. The robot information processing unit 85 is an example of a robot information processing unit.

[0183] The monitoring mode is an example of the first mode. The demonstration mode is an example of the second mode. The full mode is an example of the third mode. In addition, the switching unit 105 can be defined as a switch or a switching circuit. In addition, the image processing unit 82 can be defined as an image processor or an image processing circuit. Further, the sound processing unit 83 can be defined as a sound processor or a sound processing circuit. Further, the annotation processing unit 84 can be defined as an annotation processor or an annotation processing circuit. Further, the robot information processing unit 85 can be defined as a robot information processor or a robot information processing circuit. Further, the auxiliary information processing unit 86 can be defined as an auxiliary information processor or an auxiliary information processing circuit.

[0184] Furthermore, the present embodiment is not limited to the method disclosed using the drawings, and various changes can be made without departing from the scope of the present invention. For example, Figure 3 The illustrated manipulation unit 46 may be installed at a location different from the operating room 15 where the robot body 48 is installed. In this case, the manipulation unit 46 is connected to the computer 56 of the vision cart 47 via a network. The network may be either wireless or wired.

[0185] In addition, the control circuit of the information processing device disclosed in the present embodiment performs the following processing. The control circuit performs: processing of starting the mode selected by the guidance room terminal; processing of storing the started mode in the storage unit; processing of storing the number of times and the required time of annotations made from the guidance room terminal to the operating room terminal and the participant room terminal according to each surgical content, each disease condition, each surgeon, and each participant in the operating room in the storage unit; and processing of judging the mode in which the surgery in the operating room is most smoothly performed based on the number of times and the required time of annotations among the started modes, and recommending the judged mode to the guidance room terminal.

[0186] Furthermore, the control circuit performs: processing for starting a mode selected by the guidance room terminal; processing for obtaining, from the participating room terminal, evaluations of annotations sent from the guidance room terminal to the participating room terminal and storing the obtained evaluations in a storage unit; and processing for recommending to the guidance room terminal a mode selected at a time point when a surgery with the highest evaluation was performed.

[0187] Furthermore, the control circuit performs: starting the processing of the mode selected by the guidance room terminal; obtaining the desired mode from at least one of the operating room terminal or the participating room terminal before the guidance room terminal selects the mode, and recommending the obtained desired mode to the guidance room terminal.

[0188] Industrial Applicability

[0189] The present disclosure can be used as an information processing device, an information processing system, and an information processing program connected to an operating room terminal, an instruction room terminal, and a participant room terminal via a network.

[0190] -Explanation of symbols-

[0191] 10…remote guidance system; 11…server; 27, 36, 50, 51, 136…monitor; 29, 44, 51, 144…microphone; 28, 38, 52, 138…speaker; 13…operating room terminal; 46…control unit; 56…computer; 20…communication unit; 82…image processing unit; 83…sound processing unit; 84…annotation processing unit; 105…switching unit; 22…storage unit; 85…robot information processing unit; 86…auxiliary information processing unit.

Claims

1. An information processing device, characterized in that: have: The communication unit is connected to the following terminals via a network: an operating room terminal, which is arranged in the operating room and has a display unit for displaying images and annotations, a microphone for acquiring sounds in the operating room, and a speaker for outputting sounds acquired from outside the operating room; an instruction room terminal, which is arranged in an instruction room separated from the operating room and has a display unit for displaying images and annotations, a microphone for acquiring sounds in the instruction room, and a speaker for outputting sounds acquired from outside the instruction room; a participant room terminal, which is arranged in a participant room separated from the operating room and the instruction room and has a display unit for displaying images and annotations, a microphone for acquiring sounds in the participant room, and a speaker for outputting sounds acquired from outside the participant room; A storage unit storing a non-transitory program; and a control circuit connected to the communication unit and the storage unit, and activating a non-transitory program connected to the storage unit to execute processing, The control circuit performs: Processing of images acquired from the operating room terminal, the instruction room terminal, and the participant room terminal; Processing of sounds acquired from the operating room terminal, the instruction room terminal, and the participant room terminal; Processing of instructions and comments obtained from the instruction room terminal; and a process of switching the transmission state of information including images, sounds, and annotations between the operating room terminal, the instruction room terminal, and the participant room terminal and the communication unit, The control circuit switches and starts the first mode, the second mode, and the third mode in which the information transmission states are different. When the first mode is activated, voice can be communicated bidirectionally between the operating room terminal and the instruction room terminal, and voice can be sent unidirectionally from the operating room terminal to the participant room terminal, and voice can be sent unidirectionally from the instruction room terminal to the participant room terminal, and annotations can be sent unidirectionally from the instruction room terminal to the operating room terminal, and annotations can be sent unidirectionally from the instruction room terminal to the participant room terminal, When the second mode is activated, voice can be communicated bidirectionally between the participant room terminal and the instruction room terminal, and voice can be sent unidirectionally from the operating room terminal to the instruction room terminal, and voice communication cannot be performed between the participant room terminal and the operating room terminal, and annotations can be communicated bidirectionally between the instruction room terminal and the participant room terminal, and annotations cannot be communicated between the participant room terminal and the operating room terminal, and annotations cannot be communicated between the instruction room terminal and the operating room terminal, When the third mode is activated, voice can be communicated bidirectionally between the participant room terminal and the instruction room terminal, and voice can be communicated bidirectionally between the participant room terminal and the operating room terminal, and voice can be communicated bidirectionally between the operating room terminal and the instruction room terminal, and annotations can be communicated bidirectionally between the participant room terminal and the instruction room terminal, and annotations can be communicated bidirectionally between the participant room terminal and the operating room terminal, and annotations can be communicated bidirectionally between the operating room terminal and the instruction room terminal, When any one of the first mode, the second mode or the third mode is activated, images can be communicated bidirectionally between the operating room terminal and the guidance room terminal, and between the operating room terminal and the participant room terminal, and between the guidance room terminal and the participant room terminal.

2. The information processing device according to claim 1, wherein: The control circuit stores the processing result of the video, the processing result of the sound, and the processing result of the annotation in the storage unit.

3. The information processing device according to claim 1 or 2, wherein: A robot body is provided in the operating room. The robot body is connected to the operating room terminal and is a structure that operates based on the operation of the main controller operated by the surgeon. The control circuit includes a robot information processing unit that acquires information including an operation of the main controller and a motion of the robot body from the operating room terminal.

4. The information processing device according to claim 3, wherein: The control circuit associates information including the motion of the robot main body acquired by the robot information processing unit with information of comments generated by the instruction room, and stores the information in the storage unit.

5. The information processing device according to claim 1, wherein: The communication unit is connected to the processing terminal via a network, and the processing terminal processes the medical treatment remuneration of the instructor based on the instructor's behavior of remotely guiding the operating room terminal using the guidance room terminal. The control circuit generates auxiliary information for the processing terminal to process the medical treatment remuneration, The auxiliary information generated by the control circuit has a characteristic of making the medical treatment remuneration different depending on the activated mode.

6. An information processing system, characterized in that: have: An operating room terminal is provided in the operating room and has a display unit for displaying images and annotations, a microphone for acquiring sounds in the operating room, and a speaker for outputting sounds acquired from outside the operating room; an instruction room terminal, which is provided in an instruction room separated from the operating room, and has a display unit for displaying images and annotations, a microphone for acquiring sounds in the instruction room, and a speaker for outputting sounds acquired from outside the instruction room; a participant room terminal, which is provided in a participant room separated from the operating room and the instruction room, and has a display unit for displaying images and annotations, a microphone for acquiring sounds in the participant room, and a speaker for outputting sounds acquired from outside the participant room; and The information processing device is connected to the operating room terminal, the instruction room terminal, and the participant room terminal via a network. The information processing device comprises: a storage unit storing a non-transitory program; and The control circuit starts the non-transitory program stored in the storage unit and executes the processing, The control circuit performs: Processing of images acquired from the operating room terminal, the instruction room terminal, and the participant room terminal; Processing of sounds acquired from the operating room terminal, the instruction room terminal, and the participant room terminal; processing of annotations obtained from the instruction room terminal; and a process of switching the transmission state of information including images, sounds, and annotations between the operating room terminal, the instruction room terminal, and the participant room terminal; The control circuit switches and starts the first mode, the second mode, and the third mode in which the information transmission states are different. When the first mode is activated, voice can be communicated bidirectionally between the operating room terminal and the instruction room terminal, and voice can be sent unidirectionally from the operating room terminal to the participant room terminal, and voice can be sent unidirectionally from the instruction room terminal to the participant room terminal, and annotations can be sent unidirectionally from the instruction room terminal to the operating room terminal, and annotations can be sent unidirectionally from the instruction room terminal to the participant room terminal, When the second mode is activated, voice can be communicated bidirectionally between the participant room terminal and the instruction room terminal, and voice can be sent unidirectionally from the operating room terminal to the instruction room terminal, and voice communication cannot be performed between the participant room terminal and the operating room terminal, and annotations can be communicated bidirectionally between the instruction room terminal and the participant room terminal, and annotations cannot be communicated between the participant room terminal and the operating room terminal, and annotations cannot be communicated between the instruction room terminal and the operating room terminal, When the third mode is activated, voice can be communicated bidirectionally between the participant room terminal and the instruction room terminal, and voice can be communicated bidirectionally between the participant room terminal and the operating room terminal, and voice can be communicated bidirectionally between the operating room terminal and the instruction room terminal, and annotations can be communicated bidirectionally between the participant room terminal and the instruction room terminal, and annotations can be communicated bidirectionally between the participant room terminal and the operating room terminal, and annotations can be communicated bidirectionally between the operating room terminal and the instruction room terminal, When any one of the first mode, the second mode or the third mode is activated, images can be communicated bidirectionally between the operating room terminal and the guidance room terminal, and between the operating room terminal and the participant room terminal, and between the guidance room terminal and the participant room terminal.

7. An information processing program, characterized in that A control circuit of an information processing device is caused to execute processing, wherein the information processing device is connected to each of the following terminals via a network: An operating room terminal is provided in the operating room and has a display unit for displaying images and annotations, a microphone for acquiring sounds in the operating room, and a speaker for outputting sounds acquired from outside the operating room; an instruction room terminal, which is provided in an instruction room separated from the operating room, and has a display unit for displaying images and annotations, a microphone for acquiring sounds in the instruction room, and a speaker for outputting sounds acquired from outside the instruction room; a participant room terminal, which is provided in a participant room separated from the operating room and the instruction room, and has a display unit for displaying images and annotations, a microphone for acquiring sounds in the participant room, and a speaker for outputting sounds acquired from outside the participant room, The control circuit performs the following processing: Processing of images acquired from the operating room terminal, the instruction room terminal, and the participant room terminal; Processing of sounds acquired from the operating room terminal, the instruction room terminal, and the participant room terminal; processing of annotations obtained from the instruction room terminal; and a process of switching the transmission state of information including images, sounds, and annotations between the operating room terminal, the instruction room terminal, and the participant room terminal and the information processing device; The control circuit switches and starts the first mode, the second mode, and the third mode in which the information transmission states are different. When the first mode is activated, voice can be communicated bidirectionally between the operating room terminal and the instruction room terminal, and voice can be sent unidirectionally from the operating room terminal to the participant room terminal, and voice can be sent unidirectionally from the instruction room terminal to the participant room terminal, and annotations can be sent unidirectionally from the instruction room terminal to the operating room terminal, and annotations can be sent unidirectionally from the instruction room terminal to the participant room terminal, When the second mode is activated, voice can be communicated bidirectionally between the participant room terminal and the instruction room terminal, and voice can be sent unidirectionally from the operating room terminal to the instruction room terminal, and voice communication cannot be performed between the participant room terminal and the operating room terminal, and annotations can be communicated bidirectionally between the instruction room terminal and the participant room terminal, and annotations cannot be communicated between the participant room terminal and the operating room terminal, and annotations cannot be communicated between the instruction room terminal and the operating room terminal, When the third mode is activated, voice can be communicated bidirectionally between the participant room terminal and the instruction room terminal, and voice can be communicated bidirectionally between the participant room terminal and the operating room terminal, and voice can be communicated bidirectionally between the operating room terminal and the instruction room terminal, and annotations can be communicated bidirectionally between the participant room terminal and the instruction room terminal, and annotations can be communicated bidirectionally between the participant room terminal and the operating room terminal, and annotations can be communicated bidirectionally between the operating room terminal and the instruction room terminal, When any one of the first mode, the second mode or the third mode is activated, images can be communicated bidirectionally between the operating room terminal and the guidance room terminal, and between the operating room terminal and the participant room terminal, and between the guidance room terminal and the participant room terminal.

Images