Remote support method and system for surgical support robot

The remote support method for surgical robots allows for rapid issue resolution by enabling bidirectional communication and processed image transmission, addressing delays and ensuring efficient support without compromising privacy.

JP7578853B2Active Publication Date: 2024-11-06SYSMEX CORP
View PDF 12 Cites 0 Cited by

Patent Information

Application Number
JP2024018793
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2024-02-09
Publication Date
2024-11-06
Estimated Expiration
2038-10-06

AI Technical Summary

Technical Problem

Conventional remote support methods for surgical support robots face delays in identifying and addressing issues due to the need for on-site presence when detailed information about the robot's status is unclear, leading to inefficiencies in problem resolution.

Method used

A remote support method and system that enables bidirectional communication between a surgical support robot and a call center, using image processing to obscure personal information and transmitting processed endoscopic images to facilitate prompt and appropriate remote support.

Benefits of technology

Enables rapid identification and resolution of issues at the surgical site while protecting personal information, providing effective remote support comparable to on-site assistance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007578853000001
    Figure 0007578853000001
  • Figure 0007578853000002
    Figure 0007578853000002
  • Figure 0007578853000003
    Figure 0007578853000003
Patent Text Reader

Abstract

To grasp details of trouble in a short time in a site where an operation using an operation support robot is carried out, and perform quick and appropriate remote support.SOLUTION: A remote support method for an operation support robot includes: transmitting, to a server device (40), a request for requesting two-way communications transmitted from an operation support robot (20) and a terminal device (25) in a medical-related facility (2) where the operation support robot (20) is installed; performing, by the server device (40), processing of establishing two-way communications between the operation support robot (20) and terminal device (25) and a terminal device (42) in a call center (4); performing, by a communication controller (261), image processing of visually obscuring identification information with which a person can be identified, on an endoscope image photographed by an endoscope (201b) attached to the operation support robot (20); transmitting the endoscope image on which the image processing is performed to the server device (40) from the communication controller (261); and providing, by the server device (40), the endoscope image on which the image processing is performed to the terminal device (42).SELECTED DRAWING: Figure 6
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a remote support method for a surgical support robot. [Background technology]

[0002] In recent years, surgical support robots have been used in laparoscopic surgery and the like, and there is an increasing need to provide precise support for the surgical support robots. Patent Document 1 discloses a method for monitoring a medical robot for preventive maintenance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] US Patent Application Publication No. 2012 / 0330613 Summary of the Invention [Problem to be solved by the invention]

[0004] In conventional remote support for a surgical support robot, including the method disclosed in Patent Document 1, log data (text data) related to the operating status of the surgical support robot is collected and analyzed by a remote server. When a problem occurs with the surgical support robot, a call center staff member receives a call from the medical institution and grasps the details of the problem while looking at the analysis results of the surgical support robot's log data. However, there are cases where the details of the problem cannot be grasped, in which case a service staff member S must be present at the site (for example, the operating room where the surgical support robot is installed). In this case, it takes time from the occurrence of the problem to its resolution, which is problematic in terms of speed.

[0005] The present invention has been made in consideration of the above problems, and its purpose is to provide a remote support method and system that can provide rapid and appropriate remote support at a site where surgery or the like is performed using a surgical support robot by enabling a call center to grasp details of a problem in a short period of time while protecting personal information. [Means for solving the problem]

[0006] A method for remotely supporting a surgical support robot according to one aspect of the present invention includes a communication control device (261) that transmits a request for two-way communication, the request being transmitted from at least one of a surgical support robot (20) and a first terminal device (25) used in a medical facility (2) in which the surgical support robot (20) is installed, to a server device (40), and the server device (40) performs processing for establishing the two-way communication between at least one of the surgical support robot (20) and the first terminal device (25) and a second terminal device (42) installed in a call center (4) based on reception of the request, the method comprising the steps of: The communication control device (261) performs image processing on the endoscopic image taken by the medical support robot (20) to visually obscure specific information that can identify a person, and the endoscopic image after the image processing is transmitted from the communication control device (261) to the server device (40). The server device (40) provides the endoscopic image after the image processing to the second terminal device (42). The second terminal device (42) is equipped with a monitor for displaying the endoscopic image after the image processing provided by the server device (40). The call center (4) is a facility different from the medical-related facility (2) and is a facility for performing remote support work to deal with trouble occurring in the surgical support robot (20) using the second terminal device (42) with which the two-way communication has been established.

[0007] Also, a remote support system according to another aspect of the present invention includes a first terminal device (25) used in a medical facility (2) in which a surgical support robot (20) is installed, a second terminal device (42) installed in a call center (4), a server device (40), and a communication control device (261) that transmits a request for two-way communication transmitted from at least one of the surgical support robot (20) and the first terminal device (25) to the server device (40), wherein the server device (40) is configured to perform processing for establishing the two-way communication between at least one of the surgical support robot (20) and the first terminal device (25) and the second terminal device (42) based on reception of the request, and the communication control device (261) controls the surgical support robot (20) to perform processing for establishing the two-way communication between the second terminal device (42) and at least one of the surgical support robot (20) and the first terminal device (25) based on reception of the request, and The server device (40) is configured to perform image processing on an endoscopic image captured by an endoscope (201b) attached to a bot (20) to visually obscure specific information that can identify a person, the server device (40) is configured to receive the endoscopic image that has been subjected to the image processing from the communication control device (261) and provide the endoscopic image that has been subjected to the image processing to the second terminal device (42), the second terminal device (42) has a monitor for displaying the endoscopic image that has been subjected to the image processing provided by the server device (40), and the call center (4) is a facility different from the medical-related facility (2) and is a facility for performing remote support work to deal with trouble that occurs in the surgical support robot (20) using the second terminal device (42) with which the two-way communication has been established. Effect of the Invention

[0008] According to the present invention, at a site where surgery or the like is being performed using a surgical support robot, details of the problem can be grasped in a short time at a call center while protecting personal information, and rapid and appropriate remote support can be provided. [Brief description of the drawings]

[0009] [Figure 1] 1 is a diagram showing a configuration including a remote support system for implementing a remote support method according to a first embodiment. [Diagram 2] FIG. 2 is a diagram showing an outline of a process flow until two-way communication becomes possible in the remote support method according to the first embodiment. [Diagram 3] 1 is a diagram showing an overview of a patient-side device, a remote control device, an image processing device, and a system linkage device that make up a surgical support robot. [Figure 4] FIG. 2 is a diagram illustrating a schematic configuration of a communication control device. [Diagram 5] FIG. 2 is a diagram illustrating an overview of a process performed by a data acquisition unit. [Figure 6] FIG. 2 is a diagram showing an overview of a process performed by a data processing unit. [Figure 7] FIG. 13 is a diagram illustrating an overview of a process performed by a data transmission unit. [Figure 8] FIG. 2 is a diagram illustrating a schematic configuration of a server device. [Figure 9] FIG. 13 is a diagram illustrating an overview of a process performed by a data storage unit. [Figure 10] 11 is a diagram showing an overview of a process for establishing two-way communication by a communication control unit. FIG. [Figure 11] FIG. 11 is a diagram showing an example of a process flow for distributing data inputted in a terminal device with which bidirectional communication has been established upon detection of an abnormality in an image captured by a network camera. [Figure 12] 11 is a diagram showing an example of a process flow for distributing images in a state where two-way communication is established between terminal devices; FIG. [Figure 13] FIG. 13 is a diagram illustrating an overview of a process performed by a data providing unit. [Figure 14] FIG. 2 is a diagram showing an overview of a process performed by a data analysis unit. [Figure 15] FIG. 2 is a schematic diagram showing an example of a display screen of a terminal device of a call center. [Figure 16] FIG. 13 is a schematic diagram showing an example of a display screen of a terminal device in a medical facility. [Figure 17] FIG. 17 is a schematic diagram showing an example of a state in which data is superimposed and displayed on the display screen shown in FIGS. 15 and 16. [Figure 18] FIG. 11 is a diagram showing a schematic configuration of components that realize a remote support method according to a second embodiment. [Figure 19] FIG. 11 is a diagram showing an outline of a surgery support robot according to a third embodiment. [Figure 20] FIG. 13 is a diagram showing an outline of a surgery support robot according to a fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] [Embodiment 1] An embodiment of the present invention will be described below with reference to Figs. 1 to 10.

[0011] This embodiment will be described with reference to Figures 1 and 2. An employee of a call center 4 (hereinafter referred to as "service staff S") who performs remote support work for a medical-related facility 2, such as a hospital or medical research institute, where surgery or training using a surgical support robot 20 is performed, provides rapid and appropriate remote support to doctors and medical workers (hereinafter referred to as "medical staff M") at the medical-related facility 2.

[0012] The call center 4 is assumed to perform remote support operations for a plurality of medical-related facilities 2, and in Fig. 1, the plurality of medical-related facilities 2 are represented as medical-related facility 2A, medical-related facility 2B, ... When there is no need to distinguish between the medical-related facility 2A, the medical-related facility 2B, ..., they will be simply referred to as medical-related facility 2.

[0013] The main feature of this embodiment is that under normal circumstances, only one-way communication is performed in which operation information (described later) of the surgical support robot 20 in the medical-related facility 2 and at least one of the images captured by the network camera 24 are sent to the server device 40 of the call center 4, but when a specified event (described later) occurs, two-way communication becomes possible between the terminal device 25 of the call center 4 and the devices of the medical-related facility 2 (at least one of the surgical support robot 20, the terminal device 25, and the network camera 24).

[0014] The flow of processing until two-way communication becomes possible will be described with reference to Fig. 2. First, under normal circumstances, operation information of the surgery support robot 20 is transmitted to the server device 40 via the communication control device 261 and stored in the storage device 41 (S11 to S13). Similarly, images captured by the network camera 24 are transmitted to the server device 40 via the communication control device 261 and stored in the storage device 41 (S14 to S16).

[0015] Next, as an example of an event, when the surgical support robot 20 receives an operation from the medical staff M requesting two-way communication (S17), a request for establishing two-way communication is transmitted from the surgical support robot 20 to the server device 40 via the communication control device 261 (S18-S19). When the server device 40 receives the request (S20), the server device 40 transmits an inquiry about whether two-way communication is possible to the terminal device 42 of the call center 4 (S21). The terminal device 42 outputs the received inquiry on a monitor (S22). When a response to the inquiry is input from the service staff S using the terminal device 42, the terminal device 42 transmits the response to the server device 40 (S23-S24). If the response is to permit two-way communication (YES in S25), the server device 40 transmits a notification that the two-way communication is permitted (hereinafter referred to as a "permission notification") to the surgical support robot 20 that transmitted the request (S26-S28). In this manner, a request for establishing two-way communication is transmitted (S18), and two-way communication is established when a permission notice is received as a response thereto (S28).

[0016] After the two-way communication is established, data (e.g., at least one of voice, image, and text) inputted to the terminal device 42 and transmitted to the server device 40 is transmitted to the surgery support robot 20 via the server device 40 and the communication control device 261 (S29-S32). In addition, data received by the server device 40 during the period when the two-way communication is established is stored in the storage device 41 as a log of the two-way communication (S33).

[0017] In addition, Figure 2 shows an example in which the surgical support robot 20 receives an operation requesting two-way communication, and as a result, two-way communication is established between the surgical support robot 20 and the terminal device 42. However, if an operation requesting two-way communication is received by the terminal device 25, two-way communication is established between the terminal device 25 and the terminal device 42 through similar processing.

[0018] Next, each of the components that realize the remote support method according to this embodiment will be described with reference to FIGS. 1 and 3 to 14.

[0019] [Medical facilities 2] A LAN (local area network) (hereinafter referred to as "intra-facility LAN") is provided within the medical facility 2. The intra-facility LAN is communicatively connected to an external network 3. A surgical support robot 20, a terminal device 25, a network camera 24, and a communication control device 261 are communicatively connected to the intra-facility LAN. The communication control device 261 is a main device constituting a network service system 26.

[0020] [Surgical Assistance Robot 20] The surgery support robot 20 is installed in an operating room (room) of the medical facility 2. The surgery support robot 20 includes a patient-side device 21 to which medical instruments are attached, a remote control device 22 that remotely controls the medical instruments attached to the patient-side device 21, and an image processing device 23 that processes images captured by an endoscope 201b, which is one of the medical instruments, as well as a system linking device 27 that mediates communication between the patient-side device 21, the remote control device 22, and the image processing device 23 and a communication control device 261.

[0021] When an operation command to be executed by the patient-side device 21 is input to the remote control device 22 by the medical staff M, the remote control device 22 transmits the operation command to the controller 206 of the patient-side device 21. The controller 206 is a computer that controls the operation of the patient-side device 21. Then, in response to the operation command transmitted from the remote control device 22, the patient-side device 21 operates medical instruments such as surgical instruments and endoscopes respectively held by the multiple surgical manipulators 201. In this way, minimally invasive surgery is performed.

[0022] The patient-side device 21 is placed beside the operating table 300 on which the patient P lies. One of the surgical manipulators 201 (first manipulator) of the patient-side device 21 holds an endoscope 201b, and the other surgical manipulator 201 (second manipulator) holds an instrument 201a, which is a surgical tool. The surgical manipulator 201 holding the instrument 201a functions as an instrument arm 201A, and the surgical manipulator 201 holding the endoscope 201b functions as a camera arm 201B. Each of the instrument arms 201A and the camera arm 201B is supported by a platform 203. The multiple surgical manipulators 201 have multiple joints, and each joint is provided with a drive unit including a servo motor and a position detector such as an encoder. The surgical manipulator 201 is configured such that a medical instrument attached to the surgical manipulator 201 is controlled to perform a desired operation by a drive signal provided via the controller 206 .

[0023] The platform 203 is supported by a positioner 202 placed on the floor of a room (typically an operating room) in which the surgical support robot 20 is installed. The positioner 202 is configured to move the platform 203 and the multiple surgical manipulators 201 to positions for preparing for surgery after being positioned relative to the operating table 300 on which the patient P lies. The positioner 202, the platform 203 and the surgical manipulators 201 are moved by inputting from a touch panel 207 provided on the patient-side device 21.

[0024] An instrument 201a is detachably attached to the tip of the instrument arm 201A. The instrument 201a includes a housing attached to the instrument arm 201A and an end effector provided at the tip of an elongated shaft. Examples of the end effector include, but are not limited to, grasping forceps, scissors, hooks, high-frequency knives, snare wires, clamps, and staplers, and various treatment tools can be used. The instrument arm 201A is introduced into the body of the patient P through a trocar placed on the body surface of the patient P, and the end effector of the instrument 201a is placed near the surgical site.

[0025] An endoscope 201b is detachably attached to the tip of the camera arm 201B. The camera arm 201B is introduced into the body of the patient P via a trocar placed on the body surface of the patient P, whereby the endoscope 201b is positioned near the surgical site and captures images of the inside of the body cavity of the patient P. Images captured by the endoscope 201b are processed by the image processing device 23 and then displayed on the monitor 221 of the remote control device 22. The medical staff M inputs operation commands to the remote control device 22 while viewing the images displayed on the monitor 221.

[0026] Moreover, the image captured by the endoscope 201b is subjected to image processing by the image processing device 23, and then transmitted from the system linkage device 27 to the server device 40 via the communication control device 261. As will be described later, in terms of protecting personal information, when transmitting to the server device 40, it is preferable to perform image processing in the communication control device 261 to mask a part of the image area, or to restrict transmission as necessary.

[0027] Furthermore, the image captured by the endoscope 201b may be subjected to image processing by the image processing device 23 and then displayed on the monitor 231 connected to the image processing device 23. This allows the medical staff M who is not operating the remote control device 22 to view the image captured by the endoscope 201b.

[0028] The remote control device 22 is a device that allows the medical staff M to operate the medical instruments attached to the surgical manipulator 201. The remote control device 22 is provided with an operation handle 233, an operation pedal unit 234, and a monitor 221. The operation handle 233 is provided so that the medical staff M can remotely operate the medical instruments supported by the surgical manipulator 201. The operation handle 233 is composed of a right-hand handle that is operated with the right hand of the medical staff M, and a left-hand handle that is operated with the left hand. For example, the instrument 201a of the instrument arm 201A can be operated with the right-hand handle, and the instrument 201a of another instrument arm 201A can be operated with the left-hand handle. The operation pedal section 234 is provided with a plurality of operation pedals, including an operation pedal for executing a function of the instrument 201a operated with the right-hand handle, an operation pedal for executing a function of the instrument 201a operated with the left-hand handle, and an operation pedal for operating the camera arm 201B with the operation handle, etc. That is, the remote operation device 22 is configured to be able to transmit, to the controller 206 of the patient-side device 21, operation commands to be executed by the instrument 201a and the endoscope 201b, which are input by the medical staff M using the operation handle 233 and the operation pedal section 234.

[0029] The system linking device 27 is a computer and has functions that a computer normally has. The system linking device 27 is communicably connected to the facility LAN. The system linking device 27 mediates communication between the patient side device 21, the remote control device 22, and the image processing device 23 and the communication control device 261. The system linking device 27 is equipped with an API (application program interface), and by using this API, in response to a request from the communication control device 261, information regarding the operation of the surgery support robot 20 (hereinafter referred to as "operation information") is obtained from the patient side device 21, the remote control device 22, and the image processing device 23, and the obtained operation information is transmitted to the communication control device 261. The timing at which the system linking device 27 transmits the operation information to the communication control device 261 may be immediately after the information is obtained (in real time), or may be after a certain amount of data has been accumulated. Typical examples of operation information are shown in the following (1) to (13).

[0030] (1) Information about the error that occurred (error code, location of error, date and time of error, etc.) (2) Information regarding operation commands input to the remote control device 22 (3) Output values ​​of various sensors attached to the remote control device 22 (4) Output values ​​of various sensors attached to the patient-side device 21 (5) Axial values ​​and current values ​​of each drive unit of the patient-side device 21 (6) Image captured by endoscope 201b (7) Image processed by the image processing device 23 (8) Results of operation checks performed at system startup (9) Version information of installed software (10) Operation mode (procedure / site) (11) Status information (illumination status of the light-emitting part) (12) Information about consumables and their operation count (13) Information regarding charging of the power-assisted device for moving the patient-side device 21 and the remote control device 22 The above (1) to (13) are merely examples, and the operation information is not limited to these.

[0031] In addition, when a specified event has not been detected by the server device 40, one-way communication is established between the surgical support robot 20 and the server device 40 when transmitting operation information from the surgical support robot 20 to the server device 40.

[0032] [Network Service System 26] The network service system 26 provides various network services to the devices in the medical facility 2. To that end, the network service system 26 is provided with various APIs.

[0033] The network service system 26 includes a communication control device 261. The communication control device 261 is a computer and includes functions that a computer normally has. As shown in Fig. 4, the communication control device 261 includes a control unit 262, which is a central processing unit (CPU), and includes a data acquisition unit 263, a data processing unit 264, and a data transmission unit 265 as functions realized by software that runs on the control unit 262.

[0034] The data acquisition unit 263 acquires operation information of the surgery support robot 20. This function is realized by calling an API installed in the system linkage device 27. In addition, the data acquisition unit 263 acquires images captured by the network camera 24.

[0035] An overview of the processing performed by the data acquisition unit 263 will be described with reference to Fig. 5. Every time a predetermined time has elapsed (YES in S111), the data acquisition unit 263 calls an API for data acquisition mounted in the system linkage device 27 (S112). Through this API, the data acquisition unit 263 acquires operation information of the surgery support robot 20 (S113).

[0036] The data processing unit 264 has the following functions.

[0037] (Feature 1) Masking personal information When an image captured by the network camera 24 or an image captured by the endoscope 201b is distributed via the communication control device 261, if information that can identify a person (such as a person's face) (hereinafter referred to as "identifiable information") is shown, it is desirable to perform image processing to mask the identifiable information (processing to make it visually unclear by applying a mosaic, etc.) from the viewpoint of protecting personal information. Therefore, the data processing unit 264 checks whether or not identifiable information is shown in the image to be distributed, and if identifiable information is shown, performs image processing on the image to mask the identifiable information.

[0038] (Feature 2) Frame rate adjustment The larger the data size of the video delivered to the server device 40 via the communication control device 261 is compared to the line speed of the in-facility LAN, the longer it takes to deliver the video, causing delays. Therefore, the data processing unit 264 adjusts the number of frames per second (frame rate) of the video to be delivered to be smaller (i.e., thins out frames). Specifically, the number of frames per second is set to 6. The data processing unit 264 may monitor the communication load of the in-facility LAN and the external network 3, and adjust the number of frames per second of the video to be smaller depending on the communication load.

[0039] (Function 3) Video analysis and distribution of analysis results When the video captured by the network camera 24 is distributed to the server device 40 via the communication control device 261, if a communication load is placed on the facility LAN, it may affect the operation of various medical systems connected to the facility LAN. Therefore, the data processing unit 264 analyzes the movement of people appearing in the video to be distributed. Then, the data transmission unit 265 transmits only text data obtained as a result of the analysis to the server device 40. This reduces the communication load on the facility LAN compared to the case of distributing video. Here, "analysis of human movement" refers to, for example, an AI (artificial intelligence) installed in the communication control device 261 replacing the movement of a person appearing in the video to be analyzed with a skeleton (rod-shaped model) and quantifying the movement of the joints.

[0040] (Function 4) Processing (aggregation) of text data such as output values ​​of various sensors The data processing unit 264 may aggregate text data such as output values ​​of various sensors acquired at one-second intervals from the surgical support robot 20 into an hourly average value or a daily average value. This reduces the amount of data sent per transmission and the frequency of data sent by the data sending unit 265 to the server device 40, thereby reducing the communication load on the external network 3.

[0041] An overview of the process performed by the data processing unit 264 will be described with reference to FIG. 6. When a video is distributed to the server device 40 (YES in S211), the data processing unit 264 adjusts the frame rate of the video to be distributed (S212). Next, when identifiable information is included in the image to be distributed (YES in S213), the data processing unit 264 applies image processing to the image to obscure the identifiable information (S214). Next, when a person is included in the image to be distributed (YES in S215), the data processing unit 264 analyzes the movement of the person (S216). Next, when it is set to aggregate the data to be transmitted (YES in S217), the data processing unit 264 aggregates the data to be transmitted (S218).

[0042] The data transmission unit 265 transmits the data acquired by the data acquisition unit 263 and the data processed by the data processing unit 264 to the server device 40. Furthermore, the data transmission unit 265 may be configured to be able to restrict the transmission of data. Specifically, the data transmission unit 265 is configured to prevent only the audio included in the video from being transmitted, or to prevent both the video and the audio from being transmitted. This makes it possible to avoid unnecessary data transmission.

[0043] An overview of the processing performed by data transmission unit 265 will be described with reference to Fig. 7. When data is acquired by data acquisition unit 263 or data is processed by data processing unit 264 (YES in S311), if the data is not subject to transmission restrictions (NO in S312), the data is transmitted as is (S313). On the other hand, if the data is subject to transmission restrictions (YES in S312), the transmission of the data is restricted (S314).

[0044] [Terminal device 25] The terminal device 25 is a computer terminal used by the medical staff M in the room where the surgery support robot 20 is installed. A typical example of the terminal device 25 is a personal computer (PC). The terminal device 25 is a mobile phone, tablet terminal, smartphone, etc. The terminal device 25 has functions that a computer terminal usually has, such as a communication function with other devices, a data input function such as a keyboard, a data display function such as a monitor, a voice input function such as a microphone, and a voice output function such as a speaker.

[0045] The terminal device 25 is communicably connected to the facility LAN and accesses the network service system 26 via a Web application. For example, the terminal device 25 acquires data (e.g., operation information of the surgery support robot 20) stored in the storage device 41 of the call center 4 via the communication control device 261, and displays the acquired data on a monitor.

[0046] The terminal device 25 can receive an operation from the medical staff M requesting two-way communication with the terminal device 42. The terminal device 25 transmits a request corresponding to the operation to the server device 40. An example of the operation is clicking a button displayed on a monitor or inputting voice into a microphone, but is not limited to these.

[0047] The terminal device 25 can execute various functions for performing two-way communication (for example, voice call, video call, chat, etc.) with the terminal device 42 with which two-way communication with the server device 40 has been established. That is, the terminal device 25 transmits data input for performing two-way communication (hereinafter referred to as "communication data") to the server device 40 via the communication control device 261 to share it with the terminal device 42, while receiving and outputting communication data input to the terminal device 42 from the server device 40 via the communication control device 261. Examples of communication data include voice in the case of a voice call, video with voice in the case of a video call, text data in the case of a chat, graphics drawn with a paint tool in the case of data sharing using a whiteboard function, etc., but are not limited to these.

[0048] [Network Camera 24] The network camera 24 is installed in the room where the surgery support robot 20 is installed, and is a photographing device that photographs the situation in the room as video or still images. The video preferably includes audio. The subject of the photographing may be the entire room, the patient-side device 21, or a specific medical staff member M. It is preferable that the network camera 24 is capable of arbitrarily adjusting the tilt and zoom according to an external instruction.

[0049] The image captured by the network camera 24 is transmitted to the server device 40 via the communication control device 261. As described above, when transmitting an image to the server device 40, it is preferable that the communication control device 261 performs image processing such as masking a part of the image area (for example, the face of the patient P) or restricts transmission as necessary, from the viewpoint of protecting personal information. Furthermore, when transmitting the video captured by the network camera 24 to the server device 40, the communication control device 261 may adjust the frame rate of the video.

[0050] [Call Center 4] The call center 4 is a facility separate from the medical-related facility 2, and is a facility for performing remote support operations for the medical-related facility 2. A LAN (hereinafter referred to as the "call center LAN") is provided within the call center 4. The call center LAN is communicatively connected to the external network 3. A server device 40, a storage device 41, and a terminal device 42 are communicatively connected to the call center LAN.

[0051] [Server device 40] The server device 40 is a computer and has the functions that a computer normally has. The server device 40 is communicably connected to the LAN in the call center. As shown in Fig. 8, the server device 40 has a control unit 400 which is a CPU, and includes a data storage unit 401, a communication control unit 402, a data providing unit 403, and a data analysis unit 404 as functions realized by software running on the control unit 400.

[0052] [Data storage unit 401] The data storage unit 401 stores data transmitted from the surgery support robot 20, the network camera 24, the terminal device 25, and the terminal device 42 to the server device 40 in the storage device 41. Typical examples of data stored in the storage device 41 by the data storage unit 401 are the following (1) to (5).

[0053] (1) Operational information of Surgical Assist Robot 20 (2) Image captured by endoscope 201b (3) Images taken by Network Camera 24 (4) Data received from the terminal device 25 (various data such as voice, image, text, and graphics) (5) Data received from the terminal device 42 (various data such as voice, image, text, and graphics) Furthermore, the data storage unit 401 stores, during the period in which the two-way communication is established, data transmitted to the server device 40 from a device with which the two-way communication is established, in the storage device 41 as a log of the two-way communication.

[0054] An overview of the process performed by data storage unit 401 will be described with reference to Fig. 9. When server device 40 receives data (YES in S411), data storage unit 401 stores the received data in storage device 41 (S412). If two-way communication is still established (YES in S413), the data is stored in storage device 41 as a log of the two-way communication (S414).

[0055] [Communication control unit 402] (Communication establishment process) The communication control unit 402 controls communication between the server device 40 and devices communicably connected to the server device 40, and performs processing to establish one-way communication and two-way communication.

[0056] An overview of the process of establishing bidirectional communication by the communication control unit 402 will be described with reference to FIG. 10. The process of establishing bidirectional communication by the communication control unit 402 is started by receiving a request for bidirectional communication. When the request is received (YES in S511), an inquiry is sent to the request destination of the bidirectional communication (S512). If there is a response to the inquiry that allows the bidirectional communication (YES in S513), an approval notice is sent to the request source (S514). This establishes bidirectional communication. On the other hand, if there is a response to the inquiry that rejects the bidirectional communication (NO in S513), a notice that the bidirectional communication has been rejected (hereinafter referred to as a "rejection notice") is sent to the request source (S515). In this case, bidirectional communication is not established. Note that a well-known technique is used for the process of establishing bidirectional communication, so details will be omitted. Furthermore, it is preferable that the bidirectional communication is full-duplex communication in order to enable real-time data sharing.

[0057] The communication control unit 402 can establish multiple two-way communications in parallel. Specifically, it is possible to establish two-way communications between multiple surgery support robots 20 and the terminal device 42 installed in the same medical facility 2. It is also possible to establish two-way communications between multiple surgery support robots 20 and the terminal device 42 installed in different medical facilities 2.

[0058] The detection of a predetermined event triggers the communication control unit 402 to perform a process of establishing two-way communication. Typical examples of the predetermined event are shown below as (Event Example 1) to (Event Example 5).

[0059] (Event Example 1) A request transmitted as a result of an operation requesting two-way communication being performed from the surgical support robot 20 or the terminal device 25 is received by the server device 40.

[0060] This request is transmitted to the server device 40 from the surgery support robot 20 or the terminal device 25 where an operation requesting two-way communication has been performed by the medical staff M. If an abnormality such as a breakdown or error occurs in the surgery support robot 20, it is highly likely that the medical staff M will require the assistance of the service staff S. Therefore, it is useful to establish two-way communication between the server device 40 and at least one of the surgery support robot 20 and the terminal device 25, and between the server device 40 and the terminal device 42.

[0061] An example of an operation performed by the medical staff M is an operation on an operation unit communicably connected to the surgery support robot 20 or an operation unit provided on the surgery support robot 20. For example, pressing a switch 222 provided near the remote operation device 22 (for example, at the hand of the medical staff M who operates the remote operation device 22) or touching a button displayed on a touch panel 207 provided on the patient-side device 21. The former is mainly performed by the medical staff M who operates the remote operation device 22, and the latter is mainly performed by the medical staff M who is near the patient-side device 21. With this operation as a trigger, the controller 206 of the surgery support robot 20 transmits a request for bidirectional communication to the server device 40. The communication control unit 402 that detects the request performs a process of establishing bidirectional communication (1) between the server device 40 and at least one of the surgery support robot 20 and the terminal device 25 that transmitted the request, and (2) between the server device 40 and the terminal device 42.

[0062] Another example of an operation performed by the medical staff M is clicking a button displayed on the monitor of the terminal device 25. In this case, the request is transmitted from the terminal device 25 to the server device 40. The communication control unit 402, which detects the request, performs a process of establishing two-way communication (1) between the server device 40 and at least one of the terminal device 25 and the surgery support robot 20 that transmitted the request, and (2) between the server device 40 and the terminal device 42.

[0063] Yet another example of an operation performed by the medical staff M is a voice input to the surgery support robot 20 or the terminal device 25 requesting the establishment of two-way communication.

[0064] (Event Example 2) A request transmitted as a result of an operation for requesting two-way communication being performed from terminal device 42 is received by server device 40.

[0065] This request is transmitted to the server device 40 from the terminal device 42 in which the service staff S has performed an operation to request two-way communication. This operation is performed when the service staff S detects an abnormality or the like in the surgery support robot 20 and determines that the medical staff M should be assisted. For example, if the operation information of the surgery support robot 20 contains an abnormal value indicating an error or a malfunction, the medical staff M may require the assistance of the service staff S. Therefore, it is useful to establish two-way communication between the server device 40 and the terminal device 42, and between the server device 40 and at least one of the surgery support robot 20 and the terminal device 25.

[0066] Typical examples of abnormal values ​​contained in the operation information include axis values ​​and current values ​​that suggest that the instrument arm 201A and the camera arm 201B are interfering with each other and cannot move, and an error code indicating that the surgical support robot 20 needs to be restarted.

[0067] One example of an operation performed by the service staff S is to specify the surgery support robot 20 to be remotely operated on the terminal device 42. The request is transmitted from the terminal device 42 to the server device 40. The communication control unit 402, which detects the request, performs processing to establish two-way communication (1) between the server device 40 and the terminal device 42 that transmitted the request, and (2) between the server device 40 and at least one of the specified surgery support robot 20 and the terminal device 25.

[0068] Another example of an operation performed by the service staff S is to specify, on the terminal device 42, the terminal device 25 of the medical staff M to be provided with assistance. In this case as well, a request is transmitted from the terminal device 42 to the server device 40. The communication control unit 402, which detects the request, performs processing to establish two-way communication (1) between the server device 40 and the terminal device 42 that transmitted the request, and (2) between the server device 40 and at least one of the specified terminal device 25 and the surgery support robot 20.

[0069] (Event example 3) Detection of abnormal values ​​in operation information.

[0070] As described above, typical examples of abnormal values ​​included in the operation information include axis values ​​and current values ​​that suggest that the instrument arm 201A and the camera arm 201B are interfering with each other and are unable to move, and error codes indicating that the surgical support robot 20 needs to be restarted. In this case, the medical staff M may require assistance from the service personnel S. Therefore, it is preferable that the service personnel S be able to quickly share data with the medical staff M or remotely operate the surgical support robot 20.

[0071] In this case, the communication control unit 402 performs processing to establish two-way communication (1) between the server device 40 and at least one of the terminal device 25 and the surgical support robot 20 located in the same medical facility 2 as the network camera 24, and (2) between the server device 40 and the terminal device 42.

[0072] (Event Example 4) Detection of an abnormal situation based on image analysis of the video captured by the network camera 24 (specifically, detection of emergency behavior by medical staff member M).

[0073] An emergency action is, for example, an action different from normal (such as suddenly moving around in a hurry) or an action of operating emergency equipment (such as operating a specific telephone). In this case, the medical staff M may require the support of the service staff S. Therefore, it is preferable that the service staff S be able to quickly share data with the medical staff M or remotely operate the surgery support robot 20.

[0074] In this case, the communication control unit 402 performs processing to establish two-way communication (1) between the server device 40 and at least one of the terminal device 25 and the surgical support robot 20 located in the same medical facility 2 as the network camera 24, and (2) between the server device 40 and the terminal device 42.

[0075] (Event Example 5) Detection of an abnormal situation based on audio analysis of video captured by the network camera 24 (specifically, detection of a specific phrase, or a volume or frequency exceeding a threshold).

[0076] When a phrase requesting help from the call center 4 is detected, or when a volume or frequency exceeding a threshold is detected, it is highly likely that the speech was made due to some problem, and it is highly likely that the medical staff M will require assistance from the service staff S. Therefore, it is preferable that the service staff S be able to quickly share data with the medical staff M or remotely operate the surgery support robot 20.

[0077] In this case, the communication control unit 402 performs processing to establish two-way communication (1) between the server device 40 and at least one of the terminal device 25 and the surgical support robot 20 located in the same medical facility 2 as the network camera 24, and (2) between the server device 40 and the terminal device 42.

[0078] (Data distribution processing) The communication control unit 402 distributes various data transmitted from the device with which bidirectional communication has been established to the server device 40 to the other device with which bidirectional communication has been established. The following (Distribution Example 1) to (Distribution Example 7) are examples of distribution.

[0079] (Distribution Example 1) The communication control unit 402 distributes data (at least one of voice, image, and text) inputted into at least one of the surgical support robot 20 and the terminal device 25 with which two-way communication has been established, and the terminal device 42, and transmitted to the server device 40, to the other. This allows advice and instructions to be quickly shared between the medical staff M and the service staff S, so that the same effect as when the service staff S is present at the site to provide support can be obtained. If the data providing unit 403 constantly transmits operation information of the surgical support robot 20 to the terminal device 42, the service staff S can estimate the cause of the trouble and the method of dealing with it based on the operation information already provided before the two-way communication is established, so that the service staff S can provide support quickly and appropriately immediately after the two-way communication is established.

[0080] As an example, an example of a process flow in which bidirectional communication is established by the above-mentioned event example 3 or event example 4 and distribution example 1 is performed will be described with reference to FIG. 11. An image captured by the network camera 24 is transmitted to the server device 40 via the communication control device 261 (S41, S42). When the server device 40 receives the image (S43) and the communication control unit 402 detects an abnormal situation (YES in S44), a process of establishing bidirectional communication is performed between the server device 40 and at least one of the surgical support robot 20 and the terminal device 25, and between the server device 40 and the terminal device 42 (S45). The process of establishing bidirectional communication is as described above, so details are omitted. After the bidirectional communication is established, the data inputted by the terminal device 42 and transmitted to the server device 40 is transmitted from the communication control unit 402 to at least one of the surgical support robot 20 and the terminal device 25 (S46 to S47). At this time, the data storage unit 401 stores the transmitted data in the storage device 41 as a log of the bidirectional communication (S48). At least one of the surgery support robot 20 and the terminal device 25 that receives the data via the communication control device 261 outputs the data (S49 to S50). Similarly, data input to the surgery support robot 20 or the terminal device 25 and transmitted to the server device 40 is transmitted from the communication control unit 402 to the terminal device 42 (S51 to S53), and the terminal device 42 outputs the received data (S55). At this time, the data storage unit 401 stores the transmitted data in the storage device 41 as a log of the two-way communication (S54). Note that, if the two-way communication is full-duplex communication, S46 to S50 and S51 to S55 can be performed in parallel.

[0081] (Distribution Example 2) When two-way communication is established, it is preferable that the communication control unit 402 distributes the images captured by the network camera 24 and received by the server device 40 to at least the terminal device 42. The terminal device 42 displays the distributed images on a monitor. This allows the medical staff M and the service personnel S to visually share the current situation in the operating room to the same extent as or better than if the service personnel S were present on-site, allowing the service personnel S to provide appropriate support. The terminal device 25 may also be added as a distribution destination of the images.

[0082] (Distribution Example 3) When two-way communication is established, it is preferable that the communication control unit 402 distributes the images captured by the endoscope 201b and received by the server device 40 to at least the terminal device 42. The terminal device 42 displays the distributed images on a monitor. This allows the medical staff M and the service personnel S to visually share the status inside the body cavity of the patient P to the same extent as or better than if the service personnel S were present on-site, so that the service personnel S can provide appropriate support. The terminal device 25 may also be included as a distribution destination of the images.

[0083] (Distribution Example 4) It is preferable to simultaneously perform the distribution shown in Distribution Example 2 and the distribution shown in Distribution Example 3. In other words, it is preferable to distribute at least the image captured by the network camera 24 and received by the server device 40, and the image captured by the endoscope 201b and received by the server device 40 to the terminal device 42. The terminal device 42 simultaneously displays both of the distributed images on the monitor. This allows the service staff S to simultaneously check the current situation in the operating room and the situation inside the body cavity of the patient P, allowing the service staff S to provide appropriate support. The terminal device 25 may also be added as a destination of image distribution.

[0084] (Distribution Example 5) It is assumed that the communication control unit 402 distributes an image (at least one of an image taken by the network camera 24 and an image taken by the endoscope 201b) to the terminal device 42 with which bidirectional communication is established, and the image is displayed on the monitor of the terminal device 42. It is also assumed that the communication control unit 402 distributes the image to at least one of the surgery support robot 20 and the terminal device 25 with which bidirectional communication is established, and the image is displayed on a monitor provided in the surgery support robot 20 (the touch panel 207 provided in the patient-side device 21, the monitor 231 connected to the image processing device 23) or the monitor of the terminal device 42. In this case, when data to be superimposed on the image (for example, a graphic drawn by a paint tool in the display area of ​​the image) is input in the terminal device 42, the data is transmitted to the server device 40. Then, the communication control unit 402 distributes the data to at least one of the surgery support robot 20 and the terminal device 25 with which bidirectional communication is established. The surgery support robot 20, which has received the data, superimposes the data on the image distributed from the communication control unit 402 and displayed on the monitor. Similarly, the terminal device 25, which has received the data, superimposes the data on the image distributed from the communication control unit 402 and displayed on the monitor. This allows both the medical staff M and the service staff S to share the image on which the data is superimposed. Therefore, for example, by drawing a marking on a portion of the image that requires attention using the terminal device 42, both the medical staff M and the service staff S can easily share the portion that requires attention.

[0085] As an example, an example of a process flow in which distribution examples 2 to 5 are performed in a state where two-way communication is established between the server device 40 and the surgery support robot 20 and the terminal device 25, and between the server device 40 and the terminal device 42, will be described with reference to FIG. 12. First, an image captured by at least one of the network camera 24 and the endoscope 201b is transmitted to the server device 40 via the communication control device 261 (S61, S62). When the server device 40 receives the image (S63), the communication control unit 402 transmits the image to the terminal device 42 with which two-way communication has been established, and also transmits the image to the surgery support robot 20 and the terminal device 25 (S64). The surgery support robot 20 and the terminal device 25 display the image transmitted via the communication control device 261 on their monitors (S65, S66). The terminal device 42 similarly displays the image on its monitor (S67). As a result, the images are displayed on the monitors of the surgery support robot 20, the terminal device 25, and the terminal device 42. At this time, the data storage unit 401 stores the transmitted image in the storage device 41 as a log of the two-way communication (S68). Next, when data to be superimposed on the image is input by the service staff S of the terminal device 42 (S69), the data is received by the server device 40 (S70), and the communication control unit 402 distributes the data to the surgery support robot 20 and the terminal device 25 via the communication control device 261 (S71, S73). The surgery support robot 20 and the terminal device 25 that have received the data display the image with the data superimposed on the monitor (S74). The data storage unit 401 stores the image with the data superimposed on the storage device 41 as a log of the two-way communication (S72).

[0086] (Distribution Example 6) The communication control unit 402 distributes, to the surgery support robot 20, operation commands and alarm output commands for the surgery support robot 20 that are input to the terminal device 42 and transmitted to the server device 40. This allows the service staff S to quickly remotely operate the surgery support robot 20.

[0087] (Distribution Example 7) When the surgery support robot 20 is capable of inputting data such as voice, image, and text, the communication control unit 402 may distribute the data inputted by the surgery support robot 20 and transmitted to the server device 40 to the terminal device 42. Conversely, when the surgery support robot 20 is capable of outputting data such as voice, image, and text, the communication control unit 402 may distribute the data inputted by the terminal device 42 and transmitted to the server device 40 to the surgery support robot 20. This enables communication between the medical staff M and the service staff S without using the terminal device 25.

[0088] [Data providing unit 403] The data providing unit 403 transmits data in response to a request from the terminal device 25 to the requesting terminal device 25. Similarly, the data providing unit 403 provides data in response to a request from the terminal device 42 to the requesting terminal device 42. To this end, various APIs are prepared in the data providing unit 403. Typical examples of data provided to the terminal device 25 and the terminal device 42 are data stored in the storage device 41, and in particular, operation information of the surgery support robot 20, images captured by the network camera 24, and images captured by the endoscope 201b.

[0089] It is preferable that the data providing unit 403 constantly transmits the operation information received from the surgery support robot 20 to the terminal device 42. This allows the service staff S to constantly monitor the status of the surgery support robot 20 on the terminal device 42. Therefore, the service staff S can confirm the cause of a problem and how to deal with it in advance based on the operation information of the surgery support robot 20 before two-way communication is established, so that the service staff S can provide support quickly and appropriately immediately after starting two-way communication with the medical staff M.

[0090] If the data to be provided contains an error, the data providing unit 403 transmits a notification of the error to the requesting terminal device 25 or terminal device 42. The notification of the error may include a method for dealing with the error. Examples of the method for dealing with the error include, but are not limited to, a hyperlink that links to a corresponding section in an electronic instruction manual, or data that dynamically aggregates only necessary information from among stored information related to the method for dealing with the error.

[0091] The data providing unit 403 may transmit the provided data after statistically analyzing it using the data analyzing unit 404, or may transmit the provided data after adjusting its display format.

[0092] An overview of the processing performed by the data providing unit 403 will be described with reference to Fig. 13. When a data provision request is made from the terminal device 25 or the terminal device 42 (YES in S611), the data providing unit 403 acquires the requested data from the storage device 41 (S612). If the acquired data does not contain an error (NO in S613), the data is provided to the requester as is (S614). On the other hand, if the acquired data contains an error (YES in S613), a notification regarding the error is sent to the requester (S615). In Fig. 13, data is subsequently provided to the requester (S614), but providing the data to the requester may be omitted depending on the content of the error.

[0093] [Data Analysis Unit 404] The data analysis unit 404 analyzes various data in response to a request from the terminal device 25 or the terminal device 42, and transmits the analysis result to the request source. To this end, various APIs for analysis are prepared in the data providing unit 403. A typical example of data to be analyzed is data stored in the storage device 41 (typically, operation information of the surgery support robot 20).

[0094] An overview of the process performed by the data analysis unit 404 will be described with reference to Fig. 14. When a data analysis request is received from the terminal device 25 or the terminal device 42 (YES in S711), the data analysis unit 404 selects an API for analyzing the target data (S712) and executes the analysis (S713). Then, the analysis result is transmitted to the request source (S714).

[0095] [Terminal device 42] The terminal device 42 is a computer terminal used by the service representative S of the call center 4 within the call center 4. Typical examples of the terminal device 42 are a personal computer, a tablet terminal, a smartphone, etc. The terminal device 42 is connected to the server device 40 via a LAN within the call center 4 so as to be able to communicate with the server device 40.

[0096] In addition, the terminal device 42 has functions that a computer terminal typically has, such as a communication function with other devices, a data input function such as a keyboard, a data display function such as a monitor, an audio input function such as a microphone, and an audio output function such as a speaker.

[0097] The terminal device 42 acquires data (typically, operation information of the surgery support robot 20) stored in the storage device 41 via the server device 40, and displays the acquired data on a monitor.

[0098] The terminal device 42 can receive an operation from the service staff S requesting two-way communication with the surgery support robot 20 or the terminal device 25. The terminal device 42 transmits a request corresponding to the operation to the server device 40. An example of the operation is clicking a button displayed on a monitor or inputting voice into a microphone, but is not limited to these.

[0099] The terminal device 42 can execute various functions for performing two-way communication and data sharing with the terminal device 25 with which two-way communication has been established. That is, the terminal device 42 transmits communication data to the server device 40 to be shared with the terminal device 25, and also receives communication data input to the terminal device 25 from the server device 40.

[0100] The terminal device 42 can receive commands for the surgery support robot 20 with which bidirectional communication has been established. Examples of such commands include an alarm output command from the surgery support robot 20 and an operation command for remotely operating the surgery support robot 20.

[0101] The terminal device 42 can acquire a screen displayed on the monitor 221 of the remote control device 22 of the surgery support robot 20 with which bidirectional communication has been established, and display it on the monitor of the terminal device 42 .

[0102] If the surgical support robot 20 is equipped with an output section that outputs data such as audio, images, and text, the terminal device 42 may distribute the data to the surgical support robot 20 with which two-way communication has been established via the server device 40.

[0103] [Storage device 41] The storage device 41 is a device for storing various data, and is a magnetic disk such as a hard disk.

[0104] [Screen example] 15 to 17, an example of the display screen of each monitor of terminal device 42 and terminal device 25 will be described. Fig. 15 is a schematic diagram showing an example of the display screen of the monitor of terminal device 42. Fig. 16 is a schematic diagram showing an example of the display screen of the monitor of terminal device 25.

[0105] Here, it is assumed that two-way communication is established between the server device 40 and the terminal device 42 and the terminal device 25. It is also assumed that an image of the operating room taken by the network camera 24 and an image of the inside of the body cavity of the patient P taken by the endoscope 201b are distributed from the communication control unit 402 to the terminal device 42 and the terminal device 25.

[0106] The delivered image of the operating room is displayed in a display area A1 (see FIG. 15) of the monitor of the terminal device 42 and in a display area B1 (see FIG. 16) of the monitor of the terminal device 25. The patient-side device 21 and the remote control device 22 are shown in the image.

[0107] The delivered image of the inside of the body cavity of the patient P is displayed in a display area A2 (see FIG. 15) of the monitor of the terminal device 42 and in a display area B2 (see FIG. 16) of the monitor of the terminal device 25. The image shows the two instruments 201a introduced into the body of the patient P and the surgical site R.

[0108] In this way, the same image is simultaneously displayed on both the monitor of the terminal device 42 and the monitor of the terminal device 25. This allows both the medical staff M and the service personnel S to share the current status of the operating room and the current status inside the body cavity of the patient P to the same extent as or better than if the service personnel S were present on-site, so the service personnel S can provide appropriate support to the medical staff M.

[0109] In addition, in the display area A3 shown in FIG. 15, the operation information of each of the surgical support robots 20 acquired from the storage device 41 is displayed for each of the surgical support robots 20. Specifically, the status (Status), lab name (Lab Name), model (Model), serial number (Serial No.), last update date and time (Last Update), last error (Latest Error), and operation mode (OP Mode) are displayed, and a button (Browse) for displaying a detailed screen related to the action (Action), parts information (Parts Info.), supply information (Supply Info.), self check (Self Check), and remote view (Remote View) is displayed. In addition, in the display area A4 shown in FIG. 15, the information of each part (Parts) of the surgical support robot 20 installed in Room 1 of the A Hospital is displayed in detail (Command Value, Actual Value, Current, Difference). The service staff S can provide more accurate support by viewing this operation information.

[0110] FIG. 17 is a schematic diagram showing an example of a display screen on which data to be superimposed on an image is drawn on either one of the terminal devices 25 and 42 in the state of the display screen shown in FIG. 15 and FIG. 16. FIG. 17 shows a state in which a marking MK drawn using a paint tool or the like on either one of the terminal devices 25 and 42 is superimposed on an image on both display screens. In this way, when data to be superimposed on an image is drawn on either one of the terminal devices 25 and 42 with which bidirectional communication has been established, an image on which the data is superimposed is shared on the monitors of both the terminal devices 42 and 25 by distribution by the communication control unit 402. This allows the medical staff M and the service staff S to easily share, for example, a noteworthy part in an image.

[0111] [Variations] The patient-side device 21 may be provided with a voice input / output device and an image output device in addition to the touch panel 207. These devices enable an operation to request the establishment of two-way communication on the patient-side device 21, and after the two-way communication is established, two-way communication and data sharing between the medical staff M using the patient-side device 21 and the service staff S using the terminal device 42 are possible.

[0112] Similarly, the remote control device 22 may be provided with an operation device for receiving input operations, a voice input / output device, and an image output device in addition to the monitor 221. With these, an operation for requesting the establishment of two-way communication can be performed on the remote control device 22, and after the two-way communication is established, two-way communication and data sharing can be performed between the medical staff M using the remote control device 22 and the service staff S using the terminal device 42.

[0113] Similarly, an operation device for receiving input operations, an audio input / output device, and an image output device may be connected to the image processing device 23 in addition to the monitor 231. With these, an operation for requesting the establishment of two-way communication can be performed in the image processing device 23, and after the two-way communication is established, two-way communication and data sharing can be performed between the medical staff M using the image processing device 23 and the service staff S using the terminal device 42.

[0114] The terminal device 25 is usually connected to an in-facility LAN so as to be able to communicate with it, but may also be connected to an external network 3 so as to be able to communicate with it without going through the in-facility LAN.

[0115] [Embodiment 2] In the above-described embodiment, a configuration example in which the server device 40 is installed in the call center 4 has been described, but the location of the server device 40 is not limited to this. As shown in FIG. 18, the medical-related facility 2b according to this embodiment differs from the medical-related facility 2 of the first embodiment in that the server device 40 and the storage device 41 are installed. The server device 40 and the storage device 41 are communicably connected to a LAN arranged in the medical-related facility 2b. Also, the call center 4b according to this embodiment differs from the call center 4 of the first embodiment in that the server device 40 and the storage device 41 are not installed.

[0116] In this embodiment, data is exchanged between each device in the medical facility 2b and the server device 40 via a LAN. Data is exchanged between the storage device 41 and the terminal device 42 of the call center 4b and the server device 40 via an external network 3.

[0117] [Embodiment 3] In the above-described embodiment, the surgical support robot 20 is provided with one patient-side device 21 having a plurality of surgical manipulators 201, but the configuration of the surgical support robot 20 is not limited to this. For example, the surgical support robot 20 may be provided with a plurality of patient-side devices instead of the patient-side device 21, and each of the plurality of patient-side devices may be provided with a manipulator to which a surgical instrument such as an endoscope or forceps is attached.

[0118] A surgery support robot 20b according to this embodiment is shown in Fig. 19. The surgery support robot 20b is different from the surgery support robot 20 in that, instead of the patient-side device 21, the surgery support robot 20b includes three patient-side devices attached to a top plate 214 of a surgery table on which a patient P is placed. Specifically, the surgery support robot 20b includes a first patient-side device 211 having a surgery manipulator (first manipulator) to which an endoscope is attached, and a second patient-side device 212 and a third patient-side device 213 having a surgery manipulator (second manipulator) to which a surgical instrument such as forceps is attached. In the surgery support robot 20b, the remote control device 22 remotely controls the surgery manipulator of the first patient-side device 211 by transmitting an operation command to be executed by the endoscope held by the surgery manipulator of the first patient-side device 211 to a controller of the first patient-side device 211. Similarly, the remote control device 22 remotely controls the surgical manipulator of the second patient-side device 212 by transmitting an operation command to be executed by the surgical instrument held by the surgical manipulator of the second patient-side device 212 to the controller of the second patient-side device 212. Similarly, the remote control device 22 remotely controls the surgical manipulator of the third patient-side device 213 by transmitting an operation command to be executed by the surgical instrument held by the surgical manipulator of the third patient-side device 213 to the controller of the third patient-side device 213.

[0119] In this embodiment, the communication control unit 402 performs processing to establish bidirectional communication between the server device 40 and at least one of the surgical support robot 20b (the first patient side device 211, the second patient side device 212, the third patient side device 213, the remote control device 22, the image processing device 23), and the terminal device 25.

[0120] [Embodiment 4] In the above-described embodiment, the surgery support robots 20 and 20b are configured to include the patient side devices 21, 211, 212, 213, the remote control device 22, and the image processing device 23. However, the configuration of the surgery support robot 20 is not limited to this.

[0121] FIG. 20 shows the surgical support robot 20c according to this embodiment. The surgical support robot 20c includes a robotic surgical table 225 and an operating device 226. The robotic surgical table 225 includes a tabletop 223 on which a patient is placed, and a manipulator 224 for moving the tabletop 223. One end of the manipulator 224 is fixed to the floor, and the other end supports the tabletop 223. The operating device 226 is a mobile terminal for operating the robotic surgical table 225. The operating device 226 includes a button 227 for instructing the moving direction of the manipulator 224. The operating device 226 transmits the instruction inputted by the button 227 to the controller of the robotic surgical table 225. The manipulator 224 moves according to the instruction received by the controller, and positions the tabletop 223 at the desired position.

[0122] The operation device 226 is provided with a contact button 228 for requesting two-way communication. When the medical staff M presses the contact button 228, a request for two-way communication is transmitted from the operation device 226 to the server device 40. In this embodiment, the communication control unit 402 performs a process of establishing two-way communication between the server device 40 and at least one of the surgery support robot 20c (the robotic surgery table 225 and the operation device 226) and the terminal device 25.

[0123] The operation device 226 is capable of inputting and outputting data such as images, voice, and text, and after two-way communication is established, two-way communication and data sharing becomes possible between the medical staff M using the operation device 226 and the service staff S using the terminal device 42. Images and text are output to the monitor 229.

[0124] [Additional Note 1] The timing when the medical staff M needs the support of the service staff S may occur not only during the period when the surgery support robot 20 is in operation (i.e., during surgery or training), but also during the period when the surgery support robot 20 is not in operation. For example, when the patient side device 21, the remote control device 22, and the image processing device 23 are connected by wires, there is a case where the medical staff M needs the support of the service staff S regarding the wiring when preparing before the surgery or cleaning up after the surgery. Therefore, even during the period when the surgery support robot 20 is not in operation, two-way communication may be established to perform two-way communication between the medical staff M and the service staff S.

[0125] [Additional Note 2] The two-way communication may be performed between two devices, or between multiple devices. A typical example of the former is suitable for a case where a service staff S remotely supports one medical facility 2 (i.e., a one-to-one relationship), in which case two-way communication is performed between one terminal device 42 and the surgery support robot 20 or the terminal device 42. A typical example of the latter is suitable for a case where an instructor remotely trains multiple trainees in the training of the surgery support robot 20 (i.e., a one-to-many relationship), in which case two-way communication is performed between one terminal device 42 and multiple surgery support robots 20.

[0126] [Additional Note 3] The server device 40 does not necessarily have to be installed within the premises of the call center 4 as in embodiment 1, but may be installed outside the call center 4 and communicatively connected to the call center 4 via the external network 3.

[0127] [Additional Note 4] In the above-described embodiment, all of the functions of the communication control device 261 do not necessarily have to be provided in the communication control device 261. For example, the surgery support robot 20 or the server device 40 may have a function of performing image processing on an image captured by at least one of the endoscope 201b and the network camera 24 to visually obscure specific information that can identify a person, and a function of adjusting the frame rate of an image captured by at least one of the endoscope 201b and the network camera 24.

[0128] [Additional Note 5] In the above-described embodiment, it has been explained that communication from the call center 4 to the medical-related facility 2 is possible by receiving a request for two-way communication at the server device 40. However, it goes without saying that in an emergency, communication from the call center 4 to the medical-related facility 2 is possible without sending a request for two-way communication.

[0129] [Realization by software] The control block of the surgical support robots 20, 20b, and 20c (particularly, the system linkage device 27), the control block of the communication control device 261 (particularly, the control unit 262), and the control block of the server device 40 (particularly, the control unit 400) may be realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like, or may be realized by software.

[0130] In the latter case, the surgical support robots 20, 20b, and 20c, the communication control device 261, and the server device 40 are provided with a computer that executes instructions of a program, which is software that realizes each function. The computer is provided with, for example, one or more processors, and a computer-readable recording medium that stores the program. The processor in the computer reads the program from the recording medium and executes it, thereby achieving the object of the present invention. The processor may be, for example, a CPU or a GPU (Graphics Processing Unit). The recording medium may be a "non-transient tangible medium," such as a ROM (Read Only Memory), a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, or the like. The computer may further include a RAM (Random Access Memory) in which the program is expanded. The program may be supplied to the computer via any transmission medium (such as a communication network or a broadcast wave) that can transmit the program. Note that one aspect of the present invention may also be realized in the form of a data signal embedded in a carrier wave, in which the program is embodied by electronic transmission.

[0131] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention.

[0132] (Additional Notes) A remote support method for a surgical support robot according to one embodiment of the present invention includes a communication control device that transmits a request for two-way communication transmitted from at least one of the surgical support robot and a first terminal device used in a medical facility in which the surgical support robot is installed to a server device, and the server device performs processing to establish the two-way communication between the surgical support robot and at least one of the first terminal devices and a second terminal device installed in a call center based on reception of the request, wherein the communication control device performs image processing on an image captured by a camera installed in a room in which the surgical support robot is installed and that captures the interior of the room, to visually obscure specific information that can identify a person, and transmits the image that has been subjected to the image processing from the communication control device to the server device, and the server device provides the image that has been subjected to the image processing to the second terminal device, the second terminal device having a monitor for displaying the image that has been subjected to the image processing provided by the server device, and the call center may be a facility different from the medical facility and a facility for performing remote support work to deal with a problem that occurs in the surgical support robot using the second terminal device with which the two-way communication has been established.

[0133] In one embodiment of the present invention, a method for remote support of a surgical support robot includes the communication control device performing image processing on an endoscopic image captured by an endoscope attached to the surgical support robot to visually obscure the specific information, and transmitting the endoscopic image after the image processing to the server device, and the server device providing the endoscopic image after the image processing to the second terminal device in response to a request from the second terminal device.

[0134] In a remote support method for a surgical support robot according to one embodiment of the present invention, the communication control device may check whether the specific information is displayed in the image, and if the specific information is displayed, perform the image processing.

[0135] In one embodiment of the present invention, a method for remote assistance of a surgical support robot includes a server device configured to establish two-way communication between the surgical support robot and at least one of the first terminal device and the second terminal device when a specified event is detected, and the server device may detect the specified event when it receives a request requesting the two-way communication.

[0136] In a remote assistance method for a surgical assistance robot according to one aspect of the present invention, the server device may detect the specified event based on detection of an abnormality based on image analysis or audio analysis of the image.

[0137] In a remote assistance method for a surgical assistance robot according to one embodiment of the present invention, the server device may detect the specified event by receiving a request for two-way communication from the second terminal device.

[0138] In a remote assistance method for a surgical assistance robot according to one embodiment of the present invention, the server device may distribute the images received by the server device to at least the second terminal device when the two-way communication is established.

[0139] In one embodiment of the present invention, a method for remote support of a surgical support robot may be configured such that, when the two-way communication is established, the server device distributes to at least the second terminal device both images taken by the camera and received by the server device, and endoscopic images taken by an endoscope attached to the surgical support robot and received by the server device.

[0140] In one embodiment of the present invention, a method for remote support of a surgical support robot may, when the server device is distributing the image to the second terminal device with which the two-way communication has been established, and data to be superimposed on the image is input in the second terminal device, the data is distributed to at least one of the surgical support robot and the first terminal device.

[0141] A remote support system according to one aspect of the present invention includes a first terminal device used in a medical facility in which a surgical support robot is installed, a second terminal device installed in a call center, a server device, and a communication control device that transmits a request for two-way communication transmitted from at least one of the surgical support robot and the first terminal device to the server device, wherein the server device is configured to perform a process of establishing the two-way communication between the second terminal device and at least one of the surgical support robot and the first terminal device based on reception of the request, and the communication control device is installed in a room in which the surgical support robot is installed. and configured to perform image processing on an image captured by a camera placed in the living room for capturing images of the interior of the living room so as to visually obscure specific information that can identify a person, the server device being configured to receive the image that has been subjected to the image processing from the communication control device and to provide the image that has been subjected to the image processing to the second terminal device, the second terminal device being equipped with a monitor for displaying the image that has been subjected to the image processing provided by the server device, and the call center being a facility separate from the medical-related facility and being a facility for performing remote support work to deal with problems that occur with the surgical support robot using the second terminal device with which the two-way communication has been established.

[0142] In a remote support system according to one embodiment of the present invention, the communication control device may apply image processing to an endoscopic image captured by an endoscope attached to the surgical support robot to visually obscure the specific information, and transmit the endoscopic image after the image processing to the server device, and the server device may provide the endoscopic image after the image processing to the second terminal device in response to a request from the second terminal device.

[0143] In a remote support system according to one embodiment of the present invention, the communication control device may be configured to check whether the specific information is displayed in the image, and if the specific information is displayed, to perform the image processing.

[0144] In a remote assistance system according to one embodiment of the present invention, the server device is configured to perform a process of establishing two-way communication between at least one of the surgical assistance robot and the first terminal device and the second terminal device when a specified event is detected, and the server device may detect the specified event when it receives a request requesting the two-way communication.

[0145] In a remote assistance system according to one aspect of the present invention, the server device may be configured to detect the specified event based on detection of an abnormal situation based on image analysis or audio analysis of the image.

[0146] In a remote support system according to one embodiment of the present invention, the server device may be configured to detect the specified event by receiving a request for two-way communication from the second terminal device.

[0147] In a remote support system according to one embodiment of the present invention, the server device may be configured to distribute the image received by the server device to at least the second terminal device when the two-way communication is established.

[0148] In a remote assistance system according to one embodiment of the present invention, the server device may be configured to distribute, when the two-way communication is established, both images captured by the camera and received by the server device and endoscopic images captured by the endoscope and received by the server device to at least the second terminal device.

[0149] In a remote assistance system according to one embodiment of the present invention, the server device may be configured, when data to be superimposed on the image is input in the second terminal device with which the two-way communication is established, to distribute the data to at least one of the surgical assistance robot and the first terminal device.

[0150] In the remote support system according to an aspect of the present invention, the first terminal device and the second terminal device may be any one of a personal computer, a tablet terminal, and a smartphone. [Explanation of symbols]

[0151] 2 Medical facilities 3. External Network 4. Call Center 20, 20b, 20c Surgical robots 21 Patient side device 22 Remote control device 23 Image Processing Device 24 Network Camera (Camera) 25 Terminal device (first terminal device) 27 System Linkage Device 40 Server device 41 Storage device 42 Terminal device (second terminal device) 201 Surgical manipulator (first manipulator, second manipulator) 201b Endoscope 206 Controller 211 1st patient side device 212 2nd patient side device 223 Tabletop 224 Manipulator 225 Robotic Operating Table 226 Operating device 261 Communication Control Device 401 Data Storage Unit 402 Communication control section 403 Data Provision Department 404 Data Analysis Department

Claims

1. A request for bidirectional communication is transmitted from at least one of the surgical support robot and a first terminal device used in a medical facility in which the surgical support robot is installed to a server device by a communication control device; A method for remotely supporting a surgical support robot, comprising the steps of: in the server device, based on receipt of the request, establishing the two-way communication between the surgical support robot and at least one of the first terminal device and a second terminal device installed in a call center, the method comprising: The communication control device performs image processing on an endoscopic image captured by an endoscope attached to the surgical support robot to visually obscure specific information that can identify a person; transmitting the endoscopic image that has been subjected to the image processing from the communication control device to the server device; providing the endoscopic image having undergone the image processing to the second terminal device in the server device; the second terminal device is provided with a monitor for displaying the endoscopic image that has been subjected to the image processing and that is provided by the server device, A method for remote support of a surgical support robot, wherein the call center is a facility different from the medical-related facility and is a facility for performing remote support work to deal with problems that arise with the surgical support robot using the second terminal device with which the two-way communication has been established.

2. 2. The method for remotely assisting a surgery support robot according to claim 1, wherein the communication control device checks whether the specific information is present in the endoscopic image, and performs the image processing when the specific information is present.

3. The server device is configured to establish bidirectional communication between at least one of the surgical support robot and the first terminal device and the second terminal device when a predetermined event is detected; The method for remote assistance of a surgical assistance robot according to claim 1 , wherein the server device detects the predetermined event when a request for the two-way communication is received.

4. The method for remote assistance of a surgery support robot according to claim 3 , wherein the server device detects the predetermined event by receiving a request for bidirectional communication from the second terminal device.

5. The server device is configured to establish bidirectional communication between at least one of the surgical support robot and the first terminal device and the second terminal device when a predetermined event is detected; 3. The method for remote support of a surgical support robot according to claim 1 or 2, wherein, when the server device receives the request from the communication control device, the server device sends an inquiry to the second terminal device regarding whether the two-way communication is possible, and when the server device receives a response from the second terminal device permitting the two-way communication, the server device detects the specified event.

6. 6. The method for remote support of a surgical support robot according to claim 1, wherein the server device distributes the endoscopic image received by the server device to at least the second terminal device when the two-way communication is established.

7. 7. The method for remote support of a surgical support robot according to claim 1, wherein, when the server device is delivering the endoscopic image to the second terminal device with which the two-way communication has been established, and data to be superimposed on the endoscopic image is input into the second terminal device, the server device delivers the data to at least one of the surgical support robot and the first terminal device.

8. A first terminal device used in a medical facility in which a surgical support robot is installed; A second terminal device installed in a call center; A server device; A remote support system comprising: a communication control device that transmits a request for bidirectional communication transmitted from at least one of the surgical support robot and the first terminal device to the server device, The server device is configured to perform a process of establishing the bidirectional communication between at least one of the surgical support robot and the first terminal device and the second terminal device based on the reception of the request, the communication control device is configured to perform image processing for visually obscuring specific information that can identify a person on an endoscopic image captured by an endoscope attached to the surgical support robot; the server device is configured to receive the endoscopic image that has been subjected to the image processing from the communication control device, and to provide the endoscopic image that has been subjected to the image processing to the second terminal device; the second terminal device is provided with a monitor for displaying the endoscopic image that has been subjected to the image processing and that is provided by the server device, A remote support system, wherein the call center is a facility separate from the medical-related facility and is a facility for performing remote support work to deal with problems that arise with the surgical support robot using the second terminal device with which the two-way communication has been established.

9. The remote assistance system according to claim 8 , wherein the communication control device is configured to check whether the specific information is present in the endoscopic image, and to perform the image processing if the specific information is present.

10. the server device is configured to, when a predetermined event is detected, perform a process of establishing bidirectional communication between at least one of the surgical support robot and the first terminal device and the second terminal device; 10. The remote assistance system according to claim 8, wherein the server device detects the predetermined event when a request for the two-way communication is received.

11. The remote assistance system according to claim 10 , wherein the server device is configured to detect the predetermined event by receiving a request for bidirectional communication from the second terminal device.

12. The server device is configured to establish bidirectional communication between at least one of the surgical support robot and the first terminal device and the second terminal device when a predetermined event is detected; The remote support system according to claim 8 or 9, wherein when the server device receives the request from the communication control device, the server device sends an inquiry to the second terminal device regarding whether the two-way communication is possible, and when the server device receives a response from the second terminal device permitting the two-way communication, the server device detects the specified event.

13. The remote assistance system according to any one of claims 8 to 12, wherein the server device is configured to distribute the endoscopic image received by the server device to at least the second terminal device when the two-way communication is established.

14. The remote assistance system according to any one of claims 8 to 13, wherein when the server device is distributing the endoscopic image to the second terminal device with which the two-way communication has been established, when data to be superimposed on the endoscopic image is input in the second terminal device, the server device is configured to distribute the data to at least one of the surgical support robot and the first terminal device.

15. The remote assistance system according to claim 8 , wherein the first terminal device and the second terminal device are any one of a personal computer, a tablet terminal, and a smartphone.

Citation Information

Patent Citations

  • Management system for medical equipment

    JP2002092183A

  • System and method for supporting remote operation

    JP2004181229A

  • Operation system

    JP2005111082A

  • Remote operation support system

    JP2005143918A

  • Operation system

    JP2006218127A