Remote support method and remote support system of surgical support robot
By establishing two-way communication and image processing in the remote support system of the surgical support robot, the problem of extended fault support time of surgical support robots in the prior art is solved, and rapid remote fault diagnosis and resolution are achieved.
Patent Information
- Application Number
- CN201910916739.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-10-06
- Filing Date
- 2019-09-26
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2039-09-26
AI Technical Summary
The existing remote support method of surgical support robots. When a fault occurs, it is difficult for service personnel to quickly grasp the details of the fault, resulting in a longer support time and affecting the efficiency of fault resolution.
By receiving operation information and network camera images of surgical support robots on the server device, and establishing two-way communication when a predetermined event is detected, sending sound, image and text data in real time, adjusting image frame rate and implementing image processing to determine that character information is unclear. , reduce communication load and achieve fast remote support.
Master the details of the surgical support robot fault in a short time, achieve fast and suitable remote support, reduce the necessity of on-site access, and improve the efficiency of fault resolution.
Smart Images

Figure CN111000634B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a remote support method of a surgery support robot, etc. Background Art
[0002] In recent years, surgical support robots are used in laparoscopic surgeries and the like, and there is an increasing need for detailed support of surgical support robots. Patent Document 1 discloses a method of monitoring a medical robot for preventive maintenance.
[0003] Prior art literature
[0004] Patent Document 1: U.S. Patent Application Publication No. 2012 / 0330613 Summary of the invention
[0005] In the remote support for surgical support robots using conventional methods including the method disclosed in Patent Document 1, log data (text data) related to the working status of the surgical support robot is collected and analyzed by a remote server. In the event of a failure in the surgical support robot, the person in charge of the customer service center (call center) receives a telephone call from a medical institution and obtains the details of the failure while observing the analysis results of the log data of the surgical support robot. However, there are cases where the details of the failure cannot be obtained, in which case the service personnel S sometimes have to go to the site (for example, an operating room where the surgical support robot is installed). In this case, it takes time from the occurrence of the failure to its elimination, so there is a problem from the perspective of speed.
[0006] The present invention has been completed in view of the above-mentioned problems, and its purpose is to provide a remote support method and a remote support system, which can grasp the details of the failure in a short time at the scene of surgery using a surgical support robot, and can quickly perform appropriate remote support.
[0007] In a remote support method for a surgical support robot involved in one embodiment of the present invention, a server device (40) used to perform remote support services for the surgical support robot (20) receives work information related to the work of the surgical support robot (20), and when a predetermined event is detected, at least one of sound, image and text is delivered from the server device (40) to at least one of the surgical support robot (20) and the terminal device (25).
[0008] In addition, in a remote support method for a surgical support robot according to another embodiment of the present invention, the frame rate of at least one image captured by a camera (24) for photographing the surgical support robot (20) and a room in which the surgical support robot (20) is installed is adjusted, and the image with the adjusted frame rate is sent to a server device (40) for performing remote support services for the surgical support robot (20).
[0009] In addition, in a remote support method for a surgical support robot according to another embodiment of the present invention, image processing is performed on at least one image captured by a camera (24) that captures the surgical support robot (20) and the room in which the surgical support robot (20) is installed, so that information that can identify a person is visually unclear, and the image after the image processing is sent to a server device (40) used to perform remote support services for the surgical support robot (20).
[0010] In addition, the remote support system involved in other schemes of the present invention comprises: a server device (40) for performing remote support services for a surgical support robot (20); and a communication control device (261) for communicating with the server device (40) in order to send work information related to the work of the surgical support robot (20) to the server device (40) and receive information from the server device (40). When the server device (40) detects the predetermined event, it establishes a two-way communication process between at least one of the first terminal devices (25) used in the surgical support robot (20) and the facility where the surgical support robot (20) is installed, and a second terminal device (42) used for performing remote support services for the surgical support robot (20).
[0011] According to the present invention, at a site where a surgery or the like is performed using a surgery support robot, details of a failure can be grasped in a short time, and appropriate remote support can be promptly performed. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] Figure 1 This is a diagram showing a configuration including a remote support system that implements the remote support method according to the first embodiment.
[0013] Figure 2 This is a diagram showing an overview of the flow of processing until two-way communication becomes possible in the remote support method according to the first embodiment.
[0014] Figure 3 The diagram schematically shows a patient-side device, a remote operation device, an image processing device, and a system coordination device that constitute a surgical support robot.
[0015] Figure 4It is a diagram showing a schematic configuration of a communication control device.
[0016] Figure 5 This is a diagram showing an overview of processing performed by the data acquisition unit.
[0017] Figure 6 This is a diagram showing an overview of processing performed by the data processing unit.
[0018] Figure 7 This is a diagram showing an overview of processing performed by a data transmission unit.
[0019] Figure 8 It is a diagram showing a schematic configuration of a server device.
[0020] Fig. 9 This is a diagram showing an overview of processing performed by the data storage unit.
[0021] Fig.10 This is a diagram showing an overview of a process for establishing bidirectional communication performed by a communication control unit.
[0022] Fig.11 This is a diagram showing an example of a process flow in which data input by a terminal device with which bidirectional communication has been established after an abnormality has been detected in an image photographed by a network camera is distributed.
[0023] Fig.12 This is a diagram showing an example of a flow of processing for distributing an image in a state where bidirectional communication is established between terminal devices.
[0024] Fig.13 This is a diagram showing an overview of processing performed by the data providing unit.
[0025] Fig.14 It is a diagram showing an overview of processing performed by the data analysis unit.
[0026] Fig.15 This is a schematic diagram showing an example of a display screen of a terminal device in a customer service center.
[0027] Fig.16 This is a schematic diagram showing an example of a display screen of a terminal device of a medical-related facility.
[0028] Fig.17 It is shown in Fig.15 as well as Fig.16 FIG. 1 is a schematic diagram showing an example of a state in which data is superimposed and displayed on a display screen.
[0029] Fig.18 This is a diagram showing a schematic configuration of components for realizing the remote support method according to the second embodiment.
[0030] Fig.19 This is a diagram schematically showing a surgical support robot according to a third embodiment.
[0031] Fig. 20 This is a diagram schematically showing a surgical support robot according to a fourth embodiment.
[0032] (Explanation of symbols)
[0033] 2: Medical-related facilities; 3: External network; 4: Customer service center; 20, 20b, 20c: Surgical support robot; 21: Patient-side device; 22: Remote operation device; 23: Image processing device; 24: Network camera (camera); 25: Terminal device (1st terminal device); 27: System collaboration device; 40: Server device; 41: Storage device; 42: Terminal device (2nd terminal device); 201: Surgical manipulator (1st manipulator, 2nd manipulator); 201b: Endoscope; 206: Controller; 211: 1st patient-side device; 212: 2nd patient-side device; 223: Top plate; 224: Manipulator; 225: Robot operating table; 226: Operation device; 261: Communication control device; 401: Data storage unit; 402: Communication control unit; 403: Data providing unit; 404: Data analysis unit. DETAILED DESCRIPTION
[0034] [Implementation method 1]
[0035] In accordance with Figures 1 to 10 One embodiment of the present invention will be described below.
[0036] Reference Figure 1 as well as Figure 2 The present embodiment is described below. A salesperson (hereinafter referred to as "service personnel S") of a customer service center 4 that provides remote support services for a medical-related facility 2 such as a hospital or medical research institution that performs surgery or training using a surgical support robot 20 quickly and appropriately provides remote support to doctors and medical practitioners (hereinafter referred to as "medical staff M") of the medical-related facility 2.
[0037] The customer service center 4 is expected to provide remote support services to multiple medical related facilities 2. Figure 1 In the figure, each of the plurality of medical-related facilities 2 is described as a medical-related facility 2A, a medical-related facility 2B, ... When the medical-related facility 2A, the medical-related facility 2B, ... are not distinguished, they are simply referred to as the medical-related facility 2 .
[0038] The main feature of this embodiment is that, under normal circumstances, only one-way communication is performed, such as sending the working information (described later) of the surgical support robot 20 of the medical-related facility 2 and at least one of the images photographed by the network camera 24 to the server device 40 of the customer service center 4, but when a predetermined event (described later) occurs, two-way communication can be performed between the terminal device 25 of the customer service center 4 and the device of the medical-related facility 2 (at least one of the surgical support robot 20, the terminal device 25, and the network camera 24).
[0039] Reference Figure 2 , describing the process until two-way communication is possible. First, in normal times, the operation information of the surgical support robot 20 is sent to the server device 40 via the communication control device 261 and stored in the storage device 41 (S11 to S13). Similarly, the images photographed by the network camera 24 are sent to the server device 40 via the communication control device 261 and stored in the storage device 41 (S14 to S16).
[0040] Next, as an example of an event, when the surgical support robot 20 receives an operation requesting two-way communication from the medical staff M (S17), a request for establishing two-way communication is sent from the surgical support robot 20 to the server device 40 via the communication control device 261 (S18-S19). When the request is received by the server device 40 (S20), an inquiry about whether two-way communication is allowed is sent from the server device 40 to the terminal device 42 of the customer service center 4 (S21). In the terminal device 42, the received inquiry is output to the monitor (S22). When a response to the inquiry is input from the service staff S using the terminal device 42, the terminal device 42 sends a response to the server device 40 (S23-S24). In the case of a response to allow two-way communication ("Yes" in S25), the server device 40 sends a notification to the surgical support robot 20 that sent the request (hereinafter referred to as "permission notification") to indicate that two-way communication is allowed (S26-S28). In this way, when a request for establishing bidirectional communication is sent (S18) and a permission notification is received as a response (S28), bidirectional communication is established.
[0041] After the two-way communication is established, the data (e.g., at least one of sound, image, and text) inputted and transmitted to the server device 40 through the terminal device 42 is transmitted to the surgical support robot 20 via the server device 40 and the communication control device 261 (S29 to S32). In addition, the data received by the server device 40 during the establishment of the two-way communication is stored in the storage device 41 as a log during the two-way communication (S33).
[0042] In addition, Figure 2, an example is shown in which the surgical support robot 20 accepts an operation requesting two-way communication, and as a result, two-way communication is established between the surgical support robot 20 and the terminal device 42. However, in the case in which the terminal device 25 accepts an operation requesting two-way communication, two-way communication is established between the terminal device 25 and the terminal device 42 through the same processing.
[0043] Next, refer to Figure 1 , Figure 3 to Figure 14 , describing the various components for implementing the remote support method involved in this embodiment.
[0044] 〔Medical-related facilities 2〕
[0045] A LAN (Local Area Network) (hereinafter referred to as "LAN within the facility") is provided in the medical-related facility 2. The LAN within the facility is connected to the external network 3 in a communicable manner. The surgical support robot 20, the terminal device 25, the network camera 24, and the communication control device 261 are connected to the LAN within the facility in a communicable manner. The communication control device 261 is a main device constituting the network service system 26.
[0046] [Surgery support robot 20]
[0047] The surgical support robot 20 is installed in an operating room (room) of the medical-related facility 2. The surgical support robot 20 includes a patient-side device 21 on which a medical instrument is installed, a remote operation device 22 for remotely operating the medical instrument installed in the patient-side device 21, and an image processing device 23 for processing an image captured by an endoscope 201b as one of the medical instruments, and a system coordination device 27 for mediating communication between the patient-side device 21, the remote operation device 22, the image processing device 23, and the communication control device 261.
[0048] When the medical staff M inputs an action command to be executed by the patient-side device 21 to the remote operation device 22, the remote operation device 22 sends the action command to the controller 206 of the patient-side device 21. The controller 206 is a computer that performs action control on the patient-side device 21. In addition, the patient-side device 21 operates the surgical instruments and medical instruments such as endoscopes respectively held by the plurality of surgical manipulators 201 in response to the action command sent from the remote operation device 22. Thus, minimally invasive surgery is performed.
[0049] The patient-side device 21 is arranged next to the operating table 300 on which the patient P lies. One of the surgical manipulators 201 (the first manipulator) of the patient-side device 21 clamps an endoscope 201b, and the other surgical manipulator 201 (the second manipulator) clamps an instrument 201a as a surgical instrument. The surgical manipulator 201 clamping the instrument 201a functions as an instrument arm 201A, and the surgical manipulator 201 clamping the endoscope 201b functions as a camera arm 201B. Each instrument arm 201A and the camera arm 201B are supported by a platform 203. The plurality of surgical manipulators 201 have a plurality of joints, and a driving unit including a servo motor and a position detector such as an encoder are provided in each joint. The surgical manipulator 201 is configured to be controlled in such a manner that a medical instrument mounted on the surgical manipulator 201 performs a desired action by a driving signal provided via a controller 206. The platform 203 is supported by a positioner 202 placed on the floor of a room (typically an operating room) where the surgical support robot 20 is installed. The positioner 202 is configured to move the platform 203 and a plurality of surgical manipulators 201 to a position for surgical preparation after positioning the operating table 300 on which the patient P lies. The positioner 202, the platform 203, and the surgical manipulators 201 are moved by input from a touch panel 207 installed in the patient-side device 21.
[0050] On the instrument arm 201A, the instrument 201a is detachably mounted at its front end. The instrument 201a includes a housing mounted on the instrument arm 201A and an end effector provided at the front end of a slender rod shaft. As the end effector, for example, clamping forceps, scissors, hooks, high-frequency knives, gauze, clamps, cutting staplers can be cited, but it is not limited to these, and various treatment instruments can be applied. The instrument arm 201A is introduced into the body of the patient P via a puncture cannula indwelling on the body surface of the patient P, and the end effector of the instrument 201a is arranged near the surgical site.
[0051] An endoscope 201b is detachably mounted on the front end of the camera arm 201B. The camera arm 201B is introduced into the body of the patient P via a puncture cannula indwelling on the body surface of the patient P, whereby the endoscope 201b is arranged near the surgical site to photograph the body cavity of the patient P. The image photographed by the endoscope 201b is processed by the image processing device 23 and then displayed on the monitor 221 of the remote operation device 22. The medical staff M inputs an action command to the remote operation device 22 while observing the image displayed on the monitor 221.
[0052] In addition, the image captured by the endoscope 201b is processed by the image processing device 23 and then transmitted from the system coordination device 27 to the server device 40 via the communication control device 261. As described later, when transmitting to the server device 40, from the viewpoint of personal information protection, it is preferable to perform image processing such as masking a part of the image area by the communication control device 261, so that transmission can be limited as needed.
[0053] Alternatively, the image captured by the endoscope 201b may be processed by the image processing device 23 and then displayed on a monitor 231 connected to the image processing device 23. Thus, a medical staff M who does not operate the remote control device 22 can also observe the image captured by the endoscope 201b.
[0054] The remote operation device 22 is a device for the medical staff M to operate the medical instruments installed on the surgical manipulator 201. The remote operation device 22 is provided with an operation handle 233, an operation pedal unit 234, and a monitor 221. The operation handle 233 is provided for the medical staff M to remotely operate the medical instruments supported by the surgical manipulator 201. The operation handle 233 is composed of a right-hand handle operated by the right hand of the medical staff M and a left-hand handle operated by the left hand. For example, the instrument 201a of the instrument arm 201A can be operated with the right-hand handle, and the instrument 201a of another instrument arm 201A can be operated with the left-hand handle. The operation pedal unit 234 is provided with a plurality of operation pedals, including an operation pedal for executing the function of the instrument 201a operated with the right-hand handle, an operation pedal for executing the function of the instrument 201a operated with the left-hand handle, and an operation pedal for operating the camera arm 201B with the operation handle. That is, the remote operation device 22 is configured to be able to transmit the operation instructions to be executed by the instrument 201a and the endoscope 201b input by the medical staff M using the operation handle 233 and the operation pedal unit 234 to the controller 206 of the patient-side device 21 .
[0055] The system collaboration device 27 is a computer having functions that a computer generally has. The system collaboration device 27 is connected to the LAN within the facility in a communicative manner. The system collaboration device 27 mediates the communication between the patient-side device 21, the remote operation device 22, the image processing device 23, and the communication control device 261. The system collaboration device 27 is equipped with an API (application program interface). By using the API, the system collaboration device 27 obtains information related to the operation of the surgical support robot 20 (hereinafter referred to as "operation information") from the patient-side device 21, the remote operation device 22, and the image processing device 23 according to a request from the communication control device 261, and sends the obtained operation information to the communication control device 261. Regarding the timing of the system collaboration device 27 sending the operation information to the communication control device 261, it can be sent immediately (in real time) after acquisition, or it can be sent after accumulating a certain amount of data. Typical examples of operation information are shown in the following (1) to (13).
[0056] (1) Information about the error that occurred (error code, location, date and time of occurrence, etc.)
[0057] (2) Information about the operation command input to the remote operation device 22
[0058] (3) Output values of various sensors installed in the remote control device 22
[0059] (4) Output values of various sensors installed in the patient-side device 21
[0060] (5) Axis values and current values of each driving unit of the patient-side device 21
[0061] (6) Image taken by endoscope 201b
[0062] (7) Image processed by the image processing device 23
[0063] (8) Results of the operation check performed at system startup
[0064] (9) Version information of installed software
[0065] (10) Surgical mode (surgical method / site)
[0066] (11) Status information (lighting status of the light emitting unit)
[0067] (12) Information related to consumables and their operation times
[0068] (13) Information on charging of electric auxiliary equipment for moving the patient-side device 21 and the remote control device 22
[0069] The above (1) to (13) are examples, and the operation information is not limited to these.
[0070] Furthermore, when operation information is transmitted from the surgery support robot 20 to the server device 40 in a stage where the server device 40 has not detected a predetermined event, one-way communication is established between the surgery support robot 20 and the server device 40 .
[0071] 〔Network Service System 26〕
[0072] The network service system 26 provides various network services to the devices in the medical-related facility 2. For this purpose, the network service system 26 has various APIs prepared.
[0073] The network service system 26 includes a communication control device 261. The communication control device 261 is a computer and has functions that a computer generally has. Figure 4 As shown, the communication control device 261 includes a control unit 262 as a CPU (Central Processing Unit), and includes a data acquisition unit 263, a data processing unit 264, and a data transmission unit 265 as functions realized by software running in the control unit 262.
[0074] The data acquisition unit 263 acquires the operation information of the surgery support robot 20. This function is realized by calling an API installed in the system coordination device 27. In addition, the data acquisition unit 263 acquires an image photographed by the network camera 24.
[0075] Reference Figure 5 , the outline of the processing performed by the data acquisition unit 263 is described. The data acquisition unit 263 calls the data acquisition API installed in the system coordination device 27 every time a predetermined time has passed ("Yes" in S111) (S112). Through this API, the data acquisition unit 263 acquires the operation information of the surgery support robot 20 (S113).
[0076] The data processing unit 264 has the following functions.
[0077] (Function 1) Masking of personal information
[0078] When distributing images photographed by the network camera 24 or the endoscope 201b via the communication control device 261, if information that can identify a person (such as a person's face) (hereinafter referred to as "identifiable information") is captured, it is preferable to perform image processing to cover the identifiable information (processing to make it visually unclear by adding mosaics, etc.) from the perspective of personal information protection. Therefore, the data processing unit 264 checks whether the identifiable information is captured in the distributed image, and if the identifiable information is captured, the image processing to cover the identifiable information is performed on the image.
[0079] (Function 2) Frame rate adjustment
[0080] The larger the data size of the image delivered to the server device 40 via the communication control device 261 is relative to the line speed of the LAN in the facility, the more time is required for image delivery, which becomes a cause of delay. Therefore, the data processing unit 264 adjusts the number of frames per second (frame rate) of the delivered image (i.e., thins out the frames). Specifically, the number of frames per second is set to 6 frames. In addition, the data processing unit 264 can also monitor the communication load of the LAN in the facility and the external network 3, and adjust the number of frames per second of the image to reduce the number of frames per second according to the communication load.
[0081] (Function 3) Image analysis and delivery of analysis results
[0082] When the image captured by the network camera 24 is distributed to the server device 40 via the communication control device 261, when a communication load is applied to the LAN within the facility, it is possible to affect the operation of various medical systems connected to the LAN within the facility. Therefore, the data processing unit 264 analyzes the movement of the person captured in the distributed image. Then, the object sent to the server device 40 by the data sending unit 265 becomes only text data obtained as a result of the analysis. As a result, the communication load of the LAN within the facility can be reduced compared to the case of distributing images. Here, "analysis of human activities" means, for example, that the AI (artificial intelligence) installed in the communication control device 261 replaces the movement of the person captured in the image as the analysis object with a skeleton (a rod-shaped model), digitizes the movement of its joints, etc.
[0083] (Function 4) Processing (aggregation) of text data such as output values of various sensors
[0084] The data processing unit 264 may collect text data such as output values of various sensors obtained from the surgical support robot 20 at 1-second intervals into an average value for 1 hour or 1 day, etc. Thus, in the transmission from the data transmission unit 265 to the server device 40, the transmission amount and transmission frequency per time are reduced, so that the communication load of the external network 3 can be reduced.
[0085] Reference Figure 6 , an overview of the processing performed by the data processing unit 264 is described. When delivering an image to the server device 40 ("Yes" in S211), the data processing unit 264 adjusts the frame rate of the delivered image (S212). Next, when the data processing unit 264 captures identifiable information in the delivered image ("Yes" in S213), the data processing unit 264 performs image processing on the image to make the identifiable information unclear (S214). Next, when the data processing unit 264 captures a person in the delivered image ("Yes" in S215), the data processing unit 264 analyzes the person's activities (S216). Next, when the data processing unit 264 is set to collect the data of the sending object ("Yes" in S217), it collects the data of the sending object (S218).
[0086] The data transmission unit 265 transmits the data acquired by the data acquisition unit 263 and the data processed by the data processing unit 264 to the server device 40. Furthermore, the data transmission unit 265 can also be configured to limit the transmission of data. Specifically, only the sound included in the image is not transmitted, or both the image and the sound are not transmitted. In this way, unnecessary data transmission can be avoided.
[0087] Reference Figure 7 , the outline of the processing performed by the data transmission unit 265 is described. When the data acquisition unit 263 acquires data or the data processing unit 264 processes data ("Yes" in S311), if the data is not subject to transmission restriction ("No" in S312), the data is transmitted as it is (S313). On the other hand, if the data is subject to transmission restriction ("Yes" in S312), the transmission of the data is restricted (S314).
[0088] [Terminal device 25]
[0089] The terminal device 25 is a computer terminal used by the medical staff M in the room where the surgical support robot 20 is installed. Typical examples of the terminal device 25 are a personal computer, a tablet terminal, a smart phone, etc. The terminal device 25 has functions generally possessed by a computer terminal, such as a communication function with other devices, a data input function such as a keyboard, a data display function such as a monitor, a sound input function such as a microphone, and a sound output function such as a speaker.
[0090] The terminal device 25 is connected to the LAN in the facility in a communicable manner, and accesses the network service system 26 via the Web application. For example, the terminal device 25 obtains data (such as the operation information of the surgical support robot 20) stored in the storage device 41 of the customer service center 4 via the communication control device 261, and displays the obtained data on the monitor.
[0091] The terminal device 25 can receive an operation requesting two-way communication with the terminal device 42 from the medical staff M. The terminal device 25 transmits a request corresponding to the operation to the server device 40. An example of the operation is clicking a button displayed on a monitor or inputting a voice to a microphone, but the present invention is not limited to these.
[0092] The terminal device 25 can perform various functions for performing two-way communication (e.g., voice call, video call, chat, etc.) with the terminal device 42 that establishes two-way communication with the server device 40. That is, the terminal device 25 transmits data input for two-way communication (hereinafter referred to as "communication data") to the server device 40 via the communication control device 261 for sharing with the terminal device 42, and receives and outputs the communication data input to the terminal device 42 from the server device 40 via the communication control device 261. An example of the communication data is sound in the case of a voice call, an image with sound in the case of a video call, text data in the case of a chat, and a graphic drawn by a drawing tool in the case of data sharing using a whiteboard function, but is not limited to these.
[0093] 〔Network Camera 24〕
[0094] The network camera 24 is a photographic device installed in the room where the surgical support robot 20 is installed and photographs the conditions of the room as a video or a still image. The video is preferably accompanied by sound. The photographed object may be the entire room, the patient-side device 21, or a specific medical staff member M. The network camera 24 is preferably capable of adjusting the tilt and zoom as desired according to instructions from the outside.
[0095] The image captured by the network camera 24 is transmitted to the server device 40 via the communication control device 261. As described above, when transmitting the image to the server device 40, from the viewpoint of personal information protection, it is preferable that the communication control device 261 performs image processing such as masking a part of the image area (e.g., the face of the patient P), so that transmission can be limited as needed. In addition, when transmitting the image captured by the network camera 24 to the server device 40, the communication control device 261 may adjust the frame rate of the image.
[0096] 〔Customer Service Center 4〕
[0097] The customer service center 4 is a facility different from the medical-related facility 2 and is a facility for performing remote support services for the medical-related facility 2. A LAN (hereinafter referred to as "LAN in the customer service center") is provided in the customer service center 4. The LAN in the customer service center is connected to the external network 3 in a communicable manner. A server device 40, a storage device 41, and a terminal device 42 are connected to the LAN in the customer service center in a communicable manner.
[0098] [Server device 40]
[0099] The server device 40 is a computer having the functions that a computer generally has. The server device 40 is connected to the LAN in the customer service center in a manner that enables communication. Figure 8 As shown, the server device 40 includes a control unit 400 as a CPU, and includes a data storage unit 401, a communication control unit 402, a data providing unit 403, and a data analyzing unit 404 as functions realized by software running in the control unit 400.
[0100] [Data storage unit 401]
[0101] The data storage unit 401 stores data transmitted from the surgery support robot 20, the web camera 24, the terminal device 25, and the terminal device 42 to the server device 40 in the storage device 41. Typical examples of the data stored in the storage device 41 by the data storage unit 401 are the following (1) to (5).
[0102] (1) Operation information of the surgery support robot 20
[0103] (2) Image taken by endoscope 201b
[0104] (3) Image captured by the network camera 24
[0105] (4) Data received from the terminal device 25 (various data such as sound, image, text, and graphics)
[0106] (5) Data received from the terminal device 42 (various data such as sound, image, text, and graphics)
[0107] Furthermore, while the bidirectional communication is established, the data storage unit 401 stores data transmitted from the device in which the bidirectional communication is established to the server device 40 in the storage device 41 as a log of the bidirectional communication.
[0108] Reference Fig. 9, describing the outline of the processing performed by the data storage unit 401. When the server device 40 receives data ("Yes" in S411), the data storage unit 401 stores the received data in the storage device 41 (S412). In addition, if the two-way communication is established ("Yes" in S413), the two-way communication log is stored in the storage device 41 (S414).
[0109] [Communication control unit 402]
[0110] (Communication establishment process)
[0111] The communication control unit 402 controls communication between the server device 40 and a device connected to the server device 40 in a communicable manner, and performs processing to establish unidirectional communication and bidirectional communication.
[0112] Reference Fig.10 , an overview of the establishment process of two-way communication implemented by the communication control unit 402 is described. The establishment process of two-way communication implemented by the communication control unit 402 is started by receiving a request for two-way communication. When the request is received ("Yes" in S511), an inquiry is sent to the destination of the two-way communication request (S512). In the case where there is a response to the inquiry that permits two-way communication ("Yes" in S513), a permission notification is sent to the request source (S514). Thus, two-way communication is established. On the other hand, in the case where there is a response to the inquiry that rejects two-way communication ("No" in S513), a notification that the two-way communication is rejected (hereinafter referred to as "rejection notification") is sent to the request source (S515). In this case, two-way communication is not established. In addition, well-known techniques are used in the establishment process of two-way communication, so the details are omitted. In addition, in order to enable real-time data sharing, the two-way communication is preferably full-duplex communication.
[0113] In addition, the communication control unit 402 can establish multiple two-way communications in parallel. Specifically, it is possible to establish two-way communications between multiple surgical support robots 20 and the terminal device 42 installed in the same medical-related facility 2. In addition, it is also possible to establish two-way communications between multiple surgical support robots 20 and the terminal device 42 installed in different medical-related facilities 2.
[0114] The trigger for the communication control unit 402 to perform the process of establishing bidirectional communication is the detection of a predetermined event. Typical examples of the predetermined event are shown in the following (Event Example 1) to (Event Example 5).
[0115] (Event Example 1) The server device 40 receives a request transmitted by an operation requesting bidirectional communication from the surgery support robot 20 or the terminal device 25 .
[0116] This request is sent to the server device 40 from the surgical support robot 20 or the terminal device 25 in which the medical staff M has performed an operation requesting two-way communication. When an abnormality such as a failure or error occurs in the surgical support robot 20, the medical staff M is likely to need support from the service staff S. Therefore, it is useful to establish two-way communication between the server device 40 and at least one of the surgical support robot 20 and the terminal device 25, and between the server device 40 and the terminal device 42.
[0117] An example of the operation performed by the medical staff M is an operation on an operation unit connected to the surgical support robot 20 in a communicable manner or an operation unit provided on the surgical support robot 20. For example, it is the pressing of a switch 222 provided near the remote operation device 22 (for example, near the medical staff M who operates the remote operation device 22) or the touching of a button displayed on the touch panel 207 provided on the patient-side device 21. The former is mainly performed by the medical staff M who operates the remote operation device 22, and the latter is mainly performed by the medical staff M who is near the patient-side device 21. In response to this operation, the controller 206 of the surgical support robot 20 sends a request for two-way communication to the server device 40. The communication control unit 402 that detects the request performs processing to establish two-way communication (1) between the server device 40 and at least one of the surgical support robot 20 and the terminal device 25 that sent the request, and (2) between the server device 40 and the terminal device 42.
[0118] Another example of the operation performed by the medical staff member M is clicking a button displayed on the monitor of the terminal device 25. In this case, a request is sent from the terminal device 25 to the server device 40. The communication control unit 402, which detects the request, performs processing to establish two-way communication (1) between the server device 40 and at least one of the terminal device 25 that sent the request and the surgical support robot 20, and (2) between the server device 40 and the terminal device 42.
[0119] Another example of the operation performed by the medical staff M is inputting a voice requesting establishment of two-way communication to the surgery support robot 20 or the terminal device 25 .
[0120] (Event Example 2) The server device 40 receives a request sent by the terminal device 42 through an operation requesting two-way communication.
[0121] This request is sent to the server device 40 from the terminal device 42 in which the service personnel S has performed an operation requesting two-way communication. This operation is performed when the service personnel S, who has detected an abnormality of the surgical support robot 20, determines that the medical staff M should be supported. For example, when the operation information of the surgical support robot 20 includes an abnormal value indicating an error, a failure, etc., the medical staff M may need the support of the service personnel S. Therefore, it is useful to establish two-way communication between the server device 40 and the terminal device 42, and between the server device 40 and at least one of the surgical support robot 20 and the terminal device 25.
[0122] Typical examples of abnormal values included in the operation information include axis values and current values that indicate that the instrument arm 201A and the camera arm 201B interfere with each other and cannot move, and error codes indicating that the surgery support robot 20 needs to be restarted.
[0123] An example of an operation performed by the service personnel S is to designate a surgical support robot 20 to be remotely operated in the terminal device 42. A request is sent from the terminal device 42 to the server device 40. The communication control unit 402, which detects the request, performs processing to establish two-way communication (1) between the server device 40 and the terminal device 42 that sent the request, and (2) between the server device 40 and at least one of the designated surgical support robot 20 and the terminal device 25.
[0124] Another example of the operation performed by the service personnel S is to designate the terminal device 25 of the medical staff member M to be supported by the terminal device 42. In this case, a request is also sent from the terminal device 42 to the server device 40. The communication control unit 402, which detects the request, performs processing to establish two-way communication (1) between the server device 40 and the terminal device 42 that sent the request, and (2) between the server device 40 and at least one of the designated terminal device 25 and the surgical support robot 20.
[0125] (Event Example 3) Detection of an abnormal value included in operation information.
[0126] As described above, typical examples of abnormal values included in the operation information include axis values and current values that indicate that the instrument arm 201A and the camera arm 201B interfere with each other and cannot move, and error codes indicating that the surgery support robot 20 needs to be restarted. In this case, the medical staff M may need the support of the service staff S. Therefore, the service staff S preferably can quickly share data with the medical staff M or remotely operate the surgery support robot 20.
[0127] In this case, the communication control unit 402 performs processing to establish (1) two-way communication between the server device 40, the terminal device 25 configured in the same medical-related facility 2 as the network camera 24, and at least one of the surgical support robot 20, and (2) between the server device 40 and the terminal device 42.
[0128] (Event Example 4) Detection of an abnormal situation based on image analysis of a video captured by the network camera 24 (specifically, detection of emergency action by the medical staff M).
[0129] Emergency actions refer to, for example, actions different from normal actions (such as suddenly moving around in a panic) and actions to operate emergency equipment (such as operating a specific phone). In this case, the medical staff M may need the support of the service staff S. Therefore, the service staff S is preferably able to quickly share data with the medical staff M or remotely operate the surgical support robot 20.
[0130] In this case, the communication control unit 402 performs processing to establish (1) two-way communication between the server device 40, the terminal device 25 configured in the same medical-related facility 2 as the network camera 24, and at least one of the surgical support robot 20, and (2) between the server device 40 and the terminal device 42.
[0131] (Event Example 5) Detection of an abnormal situation based on sound analysis of an image captured by the network camera 24 (specifically, detection of a specific phrase or a volume or frequency exceeding a threshold value).
[0132] When a phrase requesting help from the customer service center 4 is detected, or a volume or frequency exceeding a threshold is detected, it is highly likely that the person is speaking because of a problem, and it is highly likely that the medical staff M needs support from the service staff S. Therefore, it is preferred that the service staff S can quickly share data with the medical staff M or remotely operate the surgical support robot 20.
[0133] In this case, the communication control unit 402 performs processing to establish (1) two-way communication between the server device 40, the terminal device 25 configured in the same medical-related facility 2 as the network camera 24, and at least one of the surgical support robot 20, and (2) between the server device 40 and the terminal device 42.
[0134] (Data distribution processing)
[0135] The communication control unit 402 distributes various data transmitted from the device with established two-way communication to the server device 40 to the device at the destination of the established two-way communication. The following (Distribution Example 1) to (Distribution Example 7) show distribution examples.
[0136] (Distribution Example 1) The communication control unit 402 distributes data (at least one of sound, image, and text) input and sent to the server device 40 by either the surgical support robot 20 and at least one of the terminal devices 25 and the terminal device 42 established through two-way communication to the other party. As a result, suggestions and instructions can be quickly shared between the medical staff M and the service personnel S, so that the same effect as the case where the service personnel S goes to the site to provide support can be obtained. In addition, if the operation information of the surgical support robot 20 is always sent to the terminal device 42 through the data providing unit 403, the service personnel S can infer the cause and treatment method of the failure based on the provided operation information before the two-way communication is established, so that it is possible to provide support quickly and appropriately just after the two-way communication is established.
[0137] As an example, see Fig.11 , an example of the process of performing the processing of the distribution example 1 by establishing two-way communication through the above-mentioned event example 3 or event example 4 is described. The image photographed by the network camera 24 is sent to the server device 40 via the communication control device 261 (S41, S42). The image is received by the server device 40 (S43), and when the communication control unit 402 detects an abnormal situation ("Yes" in S44), the two-way communication establishment process is performed between the server device 40 and at least one of the surgical support robot 20 and the terminal device 25, and between the server device 40 and the terminal device 42 (S45). The two-way communication establishment process is as described above, so the details are omitted. After the two-way communication is established, the data input by the terminal device 42 and sent to the server device 40 is sent from the communication control unit 402 to at least one of the surgical support robot 20 and the terminal device 25 (S46-S47). At this time, the data storage unit 401 stores the sent data as a log of the two-way communication in the storage device 41 (S48). At least one of the surgical support robot 20 and the terminal device 25 that received the data via the communication control device 261 outputs the data (S49-S50). Similarly, the data input by the surgical support robot 20 or the terminal device 25 and sent to the server device 40 is sent from the communication control unit 402 to the terminal device 42 (S51-S53), and the terminal device 42 outputs the received data (S55). At this time, the data storage unit 401 stores the sent data as a log of the two-way communication in the storage device 41 (S54). In addition, when the two-way communication is full-duplex communication, S46-S50 and S51-S55 can be implemented in parallel.
[0138] (Distribution Example 2) When bidirectional communication is preferably established, the communication control unit 402 distributes the image captured by the network camera 24 and received by the server device 40 to at least the terminal device 42. The terminal device 42 displays the distributed image on the monitor. As a result, the current status of the operating room can be visually shared between the medical staff M and the service staff S to a degree equivalent to or greater than that of the service staff S going to the site, so the service staff S can provide appropriate support. In addition, the terminal device 25 can also be added as a distribution destination of the image.
[0139] (Distribution Example 3) When bidirectional communication is preferably established, the communication control unit 402 distributes the image captured by the endoscope 201b and received by the server device 40 to at least the terminal device 42. The terminal device 42 displays the distributed image on the monitor. As a result, the medical staff M and the service staff S can visually share the condition in the body cavity of the patient P to a degree equivalent to or greater than that of the service staff S going to the site, so the service staff S can provide appropriate support. In addition, the terminal device 25 can also be added as a distribution destination of the image.
[0140] (Distribution Example 4) It is preferable to perform the distribution shown in Distribution Example 2 and the distribution shown in Distribution Example 3 simultaneously. That is, it is preferable to distribute the image photographed by the network camera 24 and received by the server device 40, and the image photographed by the endoscope 201b and received by the server device 40 to at least the terminal device 42. The terminal device 42 displays both of the distributed images on the monitor at the same time. Thus, the service staff S can simultaneously confirm the current status of the operating room and the status in the body cavity of the patient P, so the service staff S can provide appropriate support. In addition, the terminal device 25 can also be added as a distribution destination of the image.
[0141] (Distribution Example 5) Assume that the communication control unit 402 distributes an image (at least one of an image photographed by the network camera 24 and an image photographed by the endoscope 201b) to the terminal device 42 with established two-way communication, and displays the image on the monitor of the terminal device 42. Furthermore, assume that the communication control unit 402 also distributes the image to at least one of the surgical support robot 20 and the terminal device 25 with established two-way communication, and displays the image on the monitor of the surgical support robot 20 (the touch panel 207 provided in the patient-side device 21, the monitor 231 connected to the image processing device 23) or the monitor of the terminal device 42. In this case, when data to be displayed superimposed on the image (for example, a graphic drawn by a drawing tool in the display area of the image) is input through the terminal device 42, the data is sent to the server device 40. Then, the communication control unit 402 distributes the data to at least one of the surgical support robot 20 and the terminal device 25 with established two-way communication. The surgical support robot 20 that has received the data displays the data in an overlapping manner on the image distributed from the communication control unit 402 and displayed on the monitor. Similarly, the terminal device 25 that has received the data displays the data in an overlapping manner on the image distributed from the communication control unit 402 and displayed on the monitor. Thus, both the medical staff M and the service staff S can share the image on which the data is displayed in an overlapping manner. Therefore, for example, by drawing a mark on a point that should be paid attention to in the image through the terminal device 42, the point that should be paid attention to by both the medical staff M and the service staff S can be easily shared.
[0142] As an example, see Fig.12, an example of the flow of processing of distribution examples 2 to 5 is described in a state where two-way communication is established between the server device 40 and the surgical support robot 20 and the terminal device 25, and between the server device 40 and the terminal device 42. First, an image captured by at least one of the network camera 24 and the endoscope 201b is sent to the server device 40 via the communication control device 261 (S61, S62). When the image is received by the server device 40 (S63), the communication control unit 402 distributes the image to the terminal device 42 where the two-way communication is established, and distributes the image to the surgical support robot 20 and the terminal device 25 (S64). The surgical support robot 20 and the terminal device 25 display the image sent via the communication control device 261 on the monitor (S65, S66). The terminal device 42 also displays the image on the monitor in the same manner (S67). As a result, the image is displayed on each monitor of the surgical support robot 20, the terminal device 25, and the terminal device 42. At this time, the data storage unit 401 stores the sent image as a log of two-way communication in the storage device 41 (S68). Next, when the service personnel S of the terminal device 42 inputs data to be displayed superimposed on the image (S69), the data is received by the server device 40 (S70), and the communication control unit 402 distributes the data to the surgical support robot 20 and the terminal device 25 via the communication control device 261 (S71, S73). The surgical support robot 20 and the terminal device 25 that receive the data display the image superimposed on the data on the monitor (S74). The data storage unit 401 stores the image superimposed on the data in the storage device 41 as a log of two-way communication (S72).
[0143] (Distribution Example 6) The communication control unit 402 distributes the operation command and alarm output instruction for the surgery support robot 20 inputted through the terminal device 42 and transmitted to the server device 40 to the surgery support robot 20. Thus, the service staff S can quickly remotely operate the surgery support robot 20.
[0144] (Distribution Example 7) When data such as sound, image, and text can be input through the surgery support robot 20, the communication control unit 402 can also distribute the data input through the surgery support robot 20 and sent to the server device 40 to the terminal device 42. Conversely, when data such as sound, image, and text can be output through the surgery support robot 20, the communication control unit 402 can also distribute the data input through the terminal device 42 and sent to the server device 40 to the surgery support robot 20. In this way, the medical staff M and the service staff S can communicate without using the terminal device 25.
[0145] [Data providing unit 403]
[0146] The data providing unit 403 transmits data corresponding to the request from the terminal device 25 to the terminal device 25 of the request source. Similarly, the data providing unit 403 provides data corresponding to the request from the terminal device 42 to the terminal device 42 of the request source. For this purpose, various APIs are prepared in the data providing unit 403. Typical examples of data provided to the terminal device 25 and the terminal device 42 are data stored in the storage device 41, in particular, the operation information of the surgical support robot 20, the image photographed by the network camera 24, and the image photographed by the endoscope 201b.
[0147] In addition, the data providing unit 403 preferably always sends the operating information received from the surgery support robot 20 to the terminal device 42. Thus, the service personnel S can always monitor the status of the surgery support robot 20 through the terminal device 42. Therefore, the service personnel S can confirm the cause and treatment method of the failure in advance based on the operating information of the surgery support robot 20 before the two-way communication is established, so that the service personnel S can quickly and appropriately provide support from the beginning of the two-way communication with the medical staff M.
[0148] When the provided data includes an error, the data providing unit 403 sends a notification about the error to the terminal device 25 or the terminal device 42 of the request source. The notification about the error may also include a method for handling the error. The method for handling the error refers to, for example, dynamically collecting a hyperlink with a corresponding place in the electronic operating manual as a link destination, data of only necessary information among the accumulated information about the method for handling, etc., but is not limited to these.
[0149] The data providing unit 403 may statistically analyze the provided data with the data analyzing unit 404 and then send the data, or may organize the provided data into a display format and then send the data.
[0150] Reference Fig.13 , which explains an overview of the processing performed by the data providing unit 403. When there is a request for data provision from the terminal device 25 or the terminal device 42 ("Yes" in S611), the data providing unit 403 obtains the requested object data from the storage device 41 (S612). Here, if the obtained data does not include an error ("No" in S613), the data is provided to the request source as is (S614). On the other hand, in the case where the obtained data includes an error ("Yes" in S613), a notification related to the error is sent to the request source (S615). Fig.13 After that, the data is continued to be provided to the request source (S614), but the provision of data to the request source may be omitted depending on the content of the error.
[0151] [Data Analysis Unit 404]
[0152] The data analysis unit 404 analyzes various data according to the request from the terminal device 25 or the terminal device 42, and sends the analysis results to the request source. For this purpose, various APIs for analysis are prepared in the data providing unit 403. A typical example of the data to be analyzed is the data stored in the storage device 41 (typically, the working information of the surgical support robot 20).
[0153] Reference Fig.14 , describing the outline of the processing performed by the data analysis unit 404. When there is a request for analyzing data from the terminal device 25 or the terminal device 42 ("Yes" in S711), the data analysis unit 404 selects an API for analyzing the target data (S712), and performs the analysis (S713). Then, the analysis result is sent to the request source (S714).
[0154] [Terminal device 42]
[0155] The terminal device 42 is a computer terminal used by the service personnel S of the customer service center 4 in the customer service center 4. Typical examples of the terminal device 42 are a personal computer, a tablet terminal, a smartphone, etc. The terminal device 42 is connected to the server device 40 via a LAN in the customer service center 4 so as to be communicable.
[0156] The terminal device 42 has functions commonly found in computer terminals, such as a communication function with other devices, a data input function such as a keyboard, a data display function such as a monitor, a sound input function such as a microphone, and a sound output function such as a speaker.
[0157] The terminal device 42 acquires data (typically, operation information of the surgery support robot 20 ) stored in the storage device 41 via the server device 40 , and displays the acquired data on a monitor.
[0158] The terminal device 42 can receive an operation requesting bidirectional communication with the surgery support robot 20 or the terminal device 25 from the service personnel S. The terminal device 42 transmits a request corresponding to the operation to the server device 40. An example of the operation is clicking a button displayed on a monitor or inputting a voice to a microphone, but the present invention is not limited to these.
[0159] The terminal device 42 can perform various functions for bidirectional communication and data sharing with the terminal device 25 in which bidirectional communication is established. That is, the terminal device 42 sends communication data to the server device 40 for sharing with the terminal device 25, and receives communication data input to the terminal device 25 from the server device 40.
[0160] The terminal device 42 can receive instructions to the surgery support robot 20 in which bidirectional communication is established. Examples of such instructions include an alarm output instruction from the surgery support robot 20 and an operation instruction for remotely controlling the surgery support robot 20 .
[0161] The terminal device 42 can acquire the screen displayed on the monitor 221 of the remote operation device 22 of the surgical support robot 20 in which the bidirectional communication is established, and display it on the monitor of the terminal device 42 .
[0162] When the surgery support robot 20 includes an output unit that outputs data such as sound, images, and text, the terminal device 42 may distribute the data to the surgery support robot 20 with which bidirectional communication has been established via the server device 40 .
[0163] [Storage device 41]
[0164] The storage device 41 is a device for storing various data, and is a magnetic disk such as a hard disk.
[0165] [Screen example]
[0166] Reference Figure 15 to Figure 17 , describing an example of a display screen of each monitor of the terminal device 42 and the terminal device 25. Fig.15 It is a schematic diagram showing an example of a display screen of the monitor of the terminal device 42. Fig.16 It is a schematic diagram showing an example of a display screen of the monitor of the terminal device 25.
[0167] Here, it is assumed that bidirectional communication is established between the server device 40 and the terminal device 42 and the terminal device 25. In addition, it is assumed that the image of the operating room taken by the network camera 24 and the image of the body cavity of the patient P taken by the endoscope 201b are distributed from the communication control unit 402 to the terminal device 42 and the terminal device 25.
[0168] The image of the delivered operating room is displayed in the display area A1 of the monitor of the terminal device 42 (see Fig.15 ), and the display area B1 of the monitor of the terminal device 25 (refer to Fig.16 ). The patient-side device 21 and the remote operation device 22 are captured in this image.
[0169] In addition, the image of the body cavity of the delivered patient P is displayed in the display area A2 of the monitor of the terminal device 42 (see Fig.15 ), and the display area B2 of the monitor of the terminal device 25 (refer to Fig.16 ). In this image, two instruments 201a introduced into the body of the patient P and the surgical site R are captured.
[0170] In this way, the same image is displayed simultaneously on both the monitor of the terminal device 42 and the monitor of the terminal device 25. As a result, the medical staff M and the service staff S can share the current situation of the operating room and the current situation in the body cavity of the patient P to the same or higher degree as when the service staff S goes to the site, so the service staff S can appropriately support the medical staff M.
[0171] In addition, Fig.15 In the display area A3 shown, the work information of each surgical support robot 20 obtained from the storage device 41 is displayed for each surgical support robot 20. Specifically, the status (Status), room name (LabName), model (Model), serial number (Serial No.), latest update date and time (Last update), latest error (Latest Error), surgical mode (OP Mode) are displayed, and buttons (Browse) for displaying detailed screens related to operation (Action), part information (Parts Info.), supply information (Supply Info.), self-check (Self Check), and remote view (Remote View) are displayed. In addition, in Fig.15 The display area A4 shows detailed information (Command Value, Actual Value, Current, Difference) of each part of the surgical support robot 20 installed in Room 1 of Hospital A. The service personnel S can provide more reliable support by viewing this operation information.
[0172] Fig.17 It is shown in Fig.15 as well as Fig.16 In the state of the display screen shown, a schematic diagram is depicted showing an example of a display screen on which data is displayed superimposed on an image in one of the terminal device 25 and the terminal device 42 . Fig.17 The figure shows a situation where a mark MK drawn by a drawing tool or the like in one of the terminal device 25 and the terminal device 42 is displayed overlapping with an image in the display screens of both. In this way, when data overlapping with an image is drawn in one of the terminal device 25 and the terminal device 42 in which two-way communication is established, an image overlapping with the data is shared on the monitors of both the terminal device 42 and the terminal device 25 through distribution by the communication control unit 402. Thus, for example, the medical staff M and the service staff S can easily share the focus points in the image.
[0173] [Modifications]
[0174] The patient-side device 21 may be provided with a sound input / output device and an image output device in addition to the touch panel 207. Thus, an operation requesting establishment of two-way communication can be performed through the patient-side device 21, and after the two-way communication is established, two-way communication and data sharing can be performed between the medical staff M using the patient-side device 21 and the service staff S using the terminal device 42.
[0175] Similarly, in addition to the monitor 221, an operation device for receiving input operations, a sound input / output device, and an image output device may be provided in the remote operation device 22. Thus, an operation requesting establishment of two-way communication can be performed through the remote operation device 22, and after the two-way communication is established, two-way communication and data sharing can be performed between the medical staff M using the remote operation device 22 and the service staff S using the terminal device 42.
[0176] Similarly, an operation device for receiving input operations, a sound input / output device, and an image output device may be connected to the image processing device 23 in addition to the monitor 231. Thus, an operation requesting establishment of two-way communication can be performed through the image processing device 23, and after the two-way communication is established, two-way communication and data sharing can be performed between the medical staff M using the image processing device 23 and the service staff S using the terminal device 42.
[0177] The terminal device 25 is usually connected to the in-facility LAN in a communicable manner, but may be connected to the external network 3 in a communicable manner without passing through the in-facility LAN.
[0178] [Implementation method 2]
[0179] In the above embodiment, the server device 40 is described as being disposed in the customer service center 4, but the server device 40 is not limited thereto. Fig.18 As shown, the medical-related facility 2b involved in this embodiment is different from the medical-related facility 2 in the first embodiment in that a server device 40 and a storage device 41 are installed. The server device 40 and the storage device 41 are connected to the LAN installed in the medical-related facility 2b in a communicable manner. In addition, the customer service center 4b involved in this embodiment is different from the customer service center 4 in the first embodiment in that the server device 40 and the storage device 41 are not installed.
[0180] In this embodiment, data is transmitted and received between each device in the medical-related facility 2b and the server device 40 via the LAN. Data is transmitted and received between the storage device 41 and the terminal device 42 of the customer service center 4b and the server device 40 via the external network 3.
[0181] [Implementation method 3]
[0182] In the above embodiment, the surgical support robot 20 is provided with a single patient-side device 21 having a plurality of surgical manipulators 201, but the structure of the surgical support robot 20 is not limited thereto. For example, a structure may be provided with a plurality of patient-side devices instead of the patient-side device 21, each of which has a manipulator to which surgical instruments such as an endoscope and forceps are attached.
[0183] Fig.19 The surgical support robot 20b involved in the present embodiment is shown. The surgical support robot 20b is different from the surgical support robot 20 in that it includes three patient-side devices installed on the top plate 214 of the operating table on which the patient P is placed, instead of the patient-side device 21. Specifically, the surgical support robot 20b includes a first patient-side device 211 having a surgical manipulator (first manipulator) installed with an endoscope, a second patient-side device 212 having a surgical manipulator (second manipulator) installed with surgical instruments such as forceps, and a third patient-side device 213. In the surgical support robot 20b, the remote operation device 22 remotely operates the surgical manipulator of the first patient-side device 211 by sending an action instruction to be executed by the surgical manipulator of the first patient-side device 211 to the controller of the first patient-side device 211. Similarly, the remote operation device 22 remotely operates the surgical manipulator of the second patient-side device 212 by sending an action instruction to be executed by the surgical instrument held by the surgical manipulator of the second patient-side device 212 to the controller of the second patient-side device 212. Similarly, the remote operation device 22 remotely operates the surgical manipulator of the third patient-side device 213 by sending an action instruction to be executed by the surgical instrument held by the surgical manipulator of the third patient-side device 213 to the controller of the third patient-side device 213.
[0184] In this embodiment, the communication control unit 402 performs processing to establish two-way communication between the server device 40, the surgical support robot 20b (the first patient side device 211, the second patient side device 212, the third patient side device 213, the remote operation device 22, the image processing device 23), and at least one of the terminal devices 25.
[0185] [Implementation method 4]
[0186] In the above embodiment, the surgical support robots 20 and 20 b are provided with the patient-side devices 21 , 211 , 212 , and 213 , the remote control device 22 , and the image processing device 23 . However, the structure of the surgical support robot 20 is not limited thereto.
[0187] Fig. 20The surgical support robot 20c involved in the present embodiment is shown. The surgical support robot 20c includes a robotic operating table 225 and an operating device 226. The robotic operating table 225 includes a top plate 223 on which a patient is placed, and a manipulator 224 for moving the top plate 223. One end of the manipulator 224 is fixed to the ground and the other end supports the top plate 223. The operating device 226 is a mobile terminal for operating the robotic operating table 225. The operating device 226 includes a button 227 for indicating the moving direction of the manipulator 224. The operating device 226 sends the instruction input through the button 227 to the controller of the robotic operating table 225. The manipulator 224 moves according to the instruction received by the controller to position the top plate 223 at the desired position.
[0188] The operating device 226 is provided with a communication button 228 for requesting two-way communication. When the medical staff M presses the communication button 228, a request for two-way communication is transmitted from the operating device 226 to the server device 40. In the present embodiment, the communication control unit 402 performs a process of establishing two-way communication between the server device 40, the surgical support robot 20c (the robotic operating table 225 and the operating device 226), and at least one of the terminal device 25.
[0189] The operating device 226 can input and output data such as images, sounds, and texts, and after two-way communication is established, medical staff M using the operating device 226 and service personnel S using the terminal device 42 can perform two-way communication and data sharing. Images and texts are output to the monitor 229.
[0190] [Additional Notes 1]
[0191] The timing when the medical staff M needs support from the service staff S may arise not only during the period when the surgical support robot 20 is operating (i.e., during surgery or training), but also during the period when the surgical support robot 20 is not operating. For example, when the patient-side device 21, the remote operation device 22, and the image processing device 23 are connected in a wired manner, it may be desirable to request support from the service staff S regarding their wiring during the preparation before surgery or the cleanup after surgery. Therefore, two-way communication may be established even during the period when the surgical support robot 20 is not operating, and two-way communication may be performed between the medical staff M and the service staff S.
[0192] [Additional Notes 2]
[0193] There are two types of two-way communication: one is performed between two devices, and the other is performed between multiple devices. A typical example of the former is suitable when a service staff S performs remote support for one medical-related facility 2 (i.e., a one-to-one relationship). In this case, two-way communication is performed between one terminal device 42 and a surgical support robot 20 or between the terminal device 42. A typical example of the latter is suitable when an instructor performs remote training for multiple training subjects during the training of the surgical support robot 20 (i.e., a one-to-many relationship). In this case, two-way communication is performed between one terminal device 42 and multiple surgical support robots 20.
[0194] [Additional Note 3]
[0195] The server device 40 does not necessarily need to be installed in the facility of the customer service center 4 as in the first embodiment, and may be installed outside the customer service center 4 and communicatively connected to the customer service center 4 via the external network 3 .
[0196] [Additional Note 4]
[0197] It is not necessary to provide all the functions of the communication control device 261 in the above-mentioned embodiment in the communication control device 261. For example, the surgery support robot 20 or the server device 40 may have a function of performing image processing to make identification information that can identify a person unclear visually for an image captured by at least one of the endoscope 201b and the network camera 24, or a function of adjusting the frame rate of an image captured by at least one of the endoscope 201b and the network camera 24.
[0198] [Additional Note 5]
[0199] In the above-mentioned embodiment, it is described that communication from the customer service center 4 to the medical-related facility 2 can be performed by receiving a request for two-way communication by the server device 40, but of course, in an emergency, communication from the customer service center 4 to the medical-related facility 2 can be performed even if a request for two-way communication is not sent.
[0200] 〔Through software〕
[0201] The control blocks of the surgical support robots 20, 20b, and 20c (especially the system collaboration device 27), the control block of the communication control device 261 (especially the control unit 262), and the control block of the server device 40 (especially the control unit 400) can be implemented either by a logic circuit (hardware) formed on an integrated circuit (IC chip), etc., or by software.
[0202] In the latter case, the surgical support robots 20, 20b, 20c, the communication control device 261, and the server device 40 are provided with a computer that executes commands of a program that is software for realizing each function. The computer, for example, has one or more processors and a computer-readable recording medium storing the program. Moreover, in the computer, the program is read and executed from the recording medium by the processor, thereby achieving the purpose of the present invention. As the processor, for example, a CPU and a GPU (Graphics Processing Unit) can be used. As the recording medium, in addition to "non-temporary tangible media", such as ROM (Read Only Memory), a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, etc. can also be used. In addition, a RAM (Random Access Memory) for expanding the program can also be provided. In addition, the program can also be supplied to the computer via any transmission medium (communication network, broadcast wave, etc.) that can transmit the program. In addition, in one embodiment of the present invention, the program can also be implemented in the form of a data signal that is embodied by electronic transmission and embedded in a carrier.
[0203] The present invention is not limited to the above-described embodiments, and various modifications can be made within the scope of the claims. Embodiments obtained by appropriately combining technical means disclosed in different embodiments are also included in the technical scope of the present invention.
Claims
1. A remote support method for a surgical support robot, wherein: A server device for performing remote support services for a surgical support robot receives a request for two-way communication between the server device and at least one of the surgical support robot and the first terminal device, and between the server device and a second terminal device for performing the remote support services, which is sent by a medical staff member's operation from the surgical support robot and at least one of the first terminal devices used at a facility where the surgical support robot is installed, thereby detecting a predetermined event. When the predetermined event is detected, the server device sends a query to the second terminal device regarding whether two-way communication is possible. When a response permitting the two-way communication is received from the second terminal device in response to an operation by the service personnel performing the remote support service, the server device performs a process of establishing the two-way communication. At least one of the sound, image and text inputted and transmitted to the server device by at least one of the surgery support robot and the first terminal device and the second terminal device for which the two-way communication is established is distributed from the server device to the other terminal.
2. The remote support method of a surgical support robot according to claim 1, wherein: Operation information related to the operation of the surgery support robot including the image captured by the surgery support robot is received by the server device.
3. The remote support method of a surgical support robot according to claim 1 or 2, wherein: The server device also receives images captured by a camera that captures an image of a room in which the surgery support robot is installed.
4. The remote support method of a surgical support robot according to claim 2, wherein: Data to be displayed superimposed on the received image is received by the server device.
5. The remote support method of a surgical support robot according to claim 4, wherein: After the predetermined event is detected, the server device distributes an image in which the data is superimposed and displayed to at least one of the surgery support robot and the first terminal device.
6. The remote support method of a surgical support robot according to claim 1 or 2, wherein: The server device detects the predetermined event based on operation information related to the operation of the surgery support robot.
7. The remote support method of a surgical support robot according to claim 2, wherein: The server device detects the predetermined event based on the received image.
8. A remote support system comprising: A server device for performing remote support operations of a surgical support robot; and a communication control device for communicating with the server device in order to receive information from the server device, The server device receives a request for two-way communication between the server device and at least one of the surgical support robot and the first terminal device used at the facility where the surgical support robot is installed, and between the server device and a second terminal device used to perform the remote support service, which is sent from the surgical support robot and at least one of the first terminal devices through the operation of medical staff, and when a predetermined event is detected, an inquiry as to whether two-way communication is possible is sent to the second terminal device, and upon receiving a response allowing the two-way communication sent from the second terminal device in response to the operation of the service personnel performing the remote support service, the server device performs a process to establish the two-way communication.
9. The remote support system according to claim 8, wherein: The server device is communicatively connected to the plurality of surgery support robots.
10. The remote support system according to claim 8 or 9, wherein: The surgical support robot comprises: A patient-side device having a first manipulator equipped with an endoscope and a second manipulator equipped with a surgical instrument; and A remote operation device is used to remotely operate the first robot and the second robot. At least one of the patient-side device and the remote control device is connected to the communication control device in a communicable manner.
11. The remote support system according to claim 8 or 9, wherein: The surgical support robot comprises: A first patient-side device having a first manipulator on which an endoscope is mounted; a second patient-side device having a second manipulator equipped with a surgical instrument; and A remote operation device is used to remotely operate the first robot and the second robot. At least one of the first patient-side device, the second patient-side device, and the remote operation device is connected to the communication control device in a communicable manner.
12. The remote support system according to claim 10, wherein: The surgical support robot further includes an image processing device that performs image processing on an image obtained by photographing the endoscope. The image processing device is connected to the server device in a communicable manner. The server device performs processing for establishing the bidirectional communication between the server device and the second terminal device and between the server device and the image processing device.
13. The remote support system according to claim 8 or 9, wherein: The surgical support robot comprises: A robotic operating table having a top plate on which a patient is placed and a manipulator for moving the top plate; and an operating device for operating the robotic operating table, At least one of the robot operating table, the operating device, and the first terminal device is connected to the server device in a communicable manner. The server device performs a process of establishing the bidirectional communication between the server device and the second terminal device, and between the server device and at least one of the robot operating table, the operating device, and the first terminal device.
14. The remote support system according to claim 8, wherein: The surgery support robot transmits the request to the server device in response to a predetermined operation on an operation unit connected to the surgery support robot in a communicable manner or provided in the surgery support robot.
15. The remote support system according to claim 8, wherein: The server device is configured to distribute at least one of sound, image, and text input and transmitted to the server device by at least one of the surgery support robot and the first terminal device and the second terminal device for which the two-way communication is established to the other party.
16. The remote support system according to claim 8 or 9, wherein: The server device is configured to receive images obtained by photographing a photographic device that photographs the condition of a room in which the surgical support robot is installed, and when two-way communication is established between the server device and at least one of the surgical support robot and the first terminal device, and between the server device and the second terminal device, the received image is distributed to at least the second terminal device.
17. The remote support system according to claim 16, wherein: The communication control device transmits the image captured by the imaging device to the server device, The communication control device is configured to check whether identifiable information capable of identifying a person is captured in the image, and if the identifiable information is captured, send the image to the server device after image processing has been performed on the image to make the identifiable information visually unclear.
18. The remote support system according to claim 16, wherein: The communication control device is connected to the communication network in a communicable manner, and transmits the image obtained by the photographing device to the server device. The communication control device is configured to monitor a communication load of the communication network and adjust a frame rate of the image according to the communication load.
19. The remote support system according to claim 8 or 9, wherein: The server device receives images obtained by endoscope photography installed on the surgical support robot, and when two-way communication is established between the server device and at least one of the surgical support robot and the first terminal device, and between the server device and the second terminal device, the received image is distributed to at least the second terminal device.
20. The remote support system according to claim 8 or 9, wherein: The server device is configured to deliver, when two-way communication is established between the server device and at least one of the surgical support robot and the first terminal device, and between the server device and the second terminal device, both an image obtained by photographing a camera that photographs the condition of a room in which the surgical support robot is installed and an image obtained by photographing an endoscope installed on the surgical support robot to at least the second terminal device.
21. The remote support system according to claim 20, wherein: The second terminal device displays the two images distributed from the server device on a monitor at the same time.
22. The remote support system according to claim 16, wherein: When the server device distributes the image to the second terminal device with which the two-way communication is established, and when data to be displayed superimposed on the image is input through the second terminal device, the server device distributes the data to at least one of the surgery support robot and the first terminal device.
23. The remote support system according to claim 8 or 9, wherein: The server device includes a data storage unit for storing operation information related to the operation of the surgery support robot in a storage device. A data providing unit is provided, wherein, in response to a request from the first terminal device or the second terminal device, the operation information read from the storage device in accordance with the request is transmitted to a request source.
24. The remote support system according to claim 23, wherein: When the read operation information includes an error, the data providing unit transmits a notification regarding the error to the request source.
25. The remote support system according to claim 8 or 9, wherein: The first terminal device and the second terminal device are any of a personal computer, a tablet terminal, and a smartphone.
