Remote operation assistance device, method, and program

Abstract

One aspect of the present invention is a remote operation system in which a remote control terminal that is on an operation side and a robot control device that is on a remote side are connected over a network, and the robot control device controls a robot catheter in accordance with various control signals transmitted from the remote control terminal in accordance with operations by an operator to perform a prescribed treatment at a treatment site on a patient. The remote operation system estimates the breathing state of the patient, generates breathing state information that represents the estimated breathing state, and presents the generated breathing state information to the operator.

Description

Remote operation support device, method, and program

[0001] One aspect of the present invention relates to a remote operation support device, method, and program used in a remote operation system in which, for example, a medical professional remotely controls a robot to perform medical treatment on a patient.

[0002] In medical settings, patients who are unable to cough up phlegm on their own require the periodic insertion of a suction catheter into their airways to aspirate the phlegm. When performing sputum suction, a catheter is inserted into the nasal or oral cavity, but the insertion of the catheter not only causes discomfort and pain to the patient, but also affects breathing and may lead to suffocation. For this reason, sputum suction procedures must be performed quickly and smoothly, and it is considered desirable for each suction to take no more than 10 to 15 seconds. Non-Patent Document 1, for example, is known as a guide for performing suction procedures.

[0003] In an ultra-aging society, the number of patients requiring sputum suction is increasing, and it is expected that there will be a shortage of qualified personnel, including medical professionals, who can perform this procedure. Therefore, in order to enable a limited number of qualified personnel to treat a large number of patients across regions, remote control of a robot to perform sputum suction is being considered. This type of system is realized, for example, by attaching a catheter to the tip of the arm of a robot device, and a qualified person operating the catheter with a controller while watching the video from a camera.

[0004] FY2012 Sputum Suction Instructor Training Program, "Sputum Suction Training Textbook (Third Training)," Ministry of Health, Labour and Welfare, Internet <URL: https: / / www.mhlw.go.jp / seisakunitsuite / bunya / hukushi_kaigo / shougaishahukushi / kaigosyokuin / >

[0005] However, when performing suction procedures using a remotely operated robot, medical professionals must operate the catheter while viewing video footage of the patient, inevitably encountering transmission delays due to the network and equipment. This makes it more difficult to grasp the sense of distance between the patient and the catheter, making it more likely that the position of the catheter will be misunderstood, compared to when performing procedures while directly holding the catheter in one's hand. Furthermore, while catheter operation must be performed at the appropriate timing while monitoring the patient's breathing, it is difficult to grasp the patient's breathing from the video, making it difficult to determine the appropriate timing for operation. This requires careful operation, which inevitably results in longer procedures.

[0006] This invention has been made in light of the above circumstances, and aims to provide a technology that enables an operator to properly grasp the patient's condition when performing treatment using a catheter by remote control, thereby enabling treatment to be performed accurately and in a short time.

[0007] In order to solve the above problems, one aspect of an operation assistance device or operation assistance method according to the present invention is a remote operation system in which a remote control terminal located on an operating side and a robot control device located on a remote side are connected via a network, and the robot control device controls a catheter of the robot in accordance with various control signals transmitted from the remote control terminal in response to operation by an operator to perform a predetermined treatment on a treatment target part of a patient, in which the respiratory state of the patient is estimated, respiratory state information representing the estimated respiratory state is generated, and the generated respiratory state information is presented to the operator.

[0008] According to one aspect of the present invention, for example, information indicating the patient's respiratory condition is presented to the operator. Therefore, the operator can grasp the patient's respiratory condition by looking at the information indicating the patient's respiratory condition, which allows the operator to operate the catheter and perform treatment on the treatment target site when the patient is in an optimal respiratory condition. Therefore, treatment can be performed smoothly and reliably without causing pain to the patient.

[0009] In other words, according to one aspect of the present invention, when performing treatment using a catheter by remote operation, a technology can be provided that allows the operator to properly grasp the patient's condition and perform treatment accurately in a short period of time.

[0010] FIG. 1 is a diagram illustrating an example of the configuration of a remote operation system according to an embodiment of the present invention. FIG. 2 is a block diagram illustrating an example of the hardware configuration of a remote control terminal provided on the operating side in the system illustrated in FIG. 1. FIG. 3 is a block diagram illustrating an example of the software configuration of the remote control terminal provided on the operating side in the system illustrated in FIG. 1. FIG. 4 is a block diagram illustrating an example of the hardware configuration of a robot control device with a remote operation support function provided on the remote side in the system illustrated in FIG. 1. FIG. 5 is a block diagram illustrating an example of the software configuration of a robot control device with a remote operation support function provided on the remote side in the system illustrated in FIG. 1. FIG. 6 is a flowchart illustrating an example of the processing procedure and processing content of remote operation support control executed by the remote control terminal illustrated in FIG. 3 and the robot control device illustrated in FIG. 5. FIG. 7 is a flowchart illustrating an example of the processing procedure and processing content of video switching and transmission control executed by the robot control device, part of the processing procedure for remote operation support control illustrated in FIG. 6. FIG. 8 is a diagram illustrating an example of a method for detecting a patient's respiratory state from a camera image. FIG. 9 is a diagram illustrating an example of the correspondence between a patient's respiratory state, a corresponding detected image, and an indicator image. FIG. 10 is a diagram illustrating an example in which an indicator image indicating the patient's respiratory state is displayed on a display image. Fig. 11 is a diagram for explaining a first example of a technique for detecting the positional relationship between the nasal cavity and the catheter. Fig. 12 is a diagram for explaining a second example of a technique for detecting the positional relationship between the nasal cavity and the catheter. Fig. 13 is a diagram showing an example of an image displayed when the position of the catheter with respect to the nasal cavity is in a first positional relationship. Fig. 14 is a diagram showing an example of an image displayed when the position of the catheter with respect to the nasal cavity is in a second positional relationship. Fig. 15 is a diagram showing an example of an image displayed when the position of the catheter with respect to the nasal cavity is in a third positional relationship.

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0012] [One Embodiment] (Configuration Example) (1) System FIG. 1 is a diagram showing an example of the configuration of a remote control system according to one embodiment of the present invention.

[0013] The remote control system of one embodiment enables data transmission via a network NW between an operating side A, such as a hospital where an operator US, such as a medical professional, is present, and a remote side B, such as a home or facility where a patient PT is present.

[0014] A remote control terminal UT is placed on the operating side A, and an output device OU including a display device and an input device IN functioning as a controller are connected to this remote control terminal UT. The operator US operates the input device IN while watching the image of the patient PT displayed on the display device to remotely control the robot on the remote side B.

[0015] The remote side B is equipped with a suction device AS that performs the suction operation for phlegm, a robot RB that performs the suction operation for the patient PT on behalf of the operator US, a robot control device RT that controls the robot RB, and a camera CM. A catheter CT is attached to the tip of the robot RB's arm, and the base end of the catheter CT is connected to the suction device AS via a suction tube AT. The camera CM is, for example, an RGB-D camera, and captures images of a predetermined area including the patient's face. A thermal camera (or thermo camera, thermography camera) is used to detect the temperature distribution of the inhaled and exhaled air associated with the patient PT's breathing. Instead of a camera, a sensor that detects the up and down movement of the chest or abdomen due to breathing may be used.

[0016] The network NW is composed of a wide area network such as the Internet and an access network for accessing the wide area network. The access network may be a wired or wireless local area network (LAN), an optical transmission network, or a mobile communication network that adopts the 5G standard.

[0017] (2) Remote Control Terminal UT FIGS. 2 and 3 are block diagrams showing an example of the hardware configuration and software configuration of the remote control terminal UT, respectively.

[0018] The remote control terminal UT is, for example, a personal computer, and has a control unit 1A that uses a hardware processor such as a central processing unit (CPU).To this control unit 1A, a memory unit having a program memory unit 2A and a data memory unit 3A, an input / output interface (hereinafter, the interface will be abbreviated as I / F) unit 4A, and a communication I / F unit 5A are connected via a bus 6A.

[0019] The input / output I / F unit 4A is connected to the output device OU and the input device IN. The input / output I / F unit 4A outputs video data transmitted from the remote side B to the output device OU, and receives an operation signal representing the operation content of the operator US from the input device IN.

[0020] The program storage unit 2A is, for example, a combination of a non-volatile memory such as a HDD (Hard Disk Drive) or SSD (Solid State Drive) as a storage medium that can be written to and read from at any time, and a non-volatile memory such as a ROM (Read Only Memory), and stores application programs necessary to execute various processes related to one embodiment of the present invention, in addition to middleware such as an OS (Operating System).

[0021] The data storage unit 3A is, for example, a combination of a non-volatile memory such as an HDD or SSD as a storage medium that can be written to and read at any time, and a volatile memory such as a RAM (Random Access Memory), and has a storage area used to temporarily store data when displaying video data or transmitting various control signals for remote operation.

[0022] The control unit 1A includes a video data receiving and displaying processing unit 11A and a catheter operation control signal transmitting processing unit 12A as control functions required to realize one embodiment of the present invention.

[0023] Each of the processing units 11A and 12A is realized by causing the processor of the control unit 1A to execute an application program stored in the program storage unit 2A. Note that a part or all of each of the processing units 11A and 12A may be realized using hardware such as an LSI (Large Scale Integration) or an ASIC (Application Specific Integrated Circuit).

[0024] The video data receiving and display processing unit 11A receives video data transmitted from the robot control device RT on the remote side B (described later) via the communication I / F unit 5A, and displays the received video data on the display device DP of the output device OU.

[0025] When the operator US performs remote operations on the input device IN to insert or remove the catheter CT or to suction phlegm, the catheter operation control signal transmission processing unit 12A generates an operation control signal corresponding to these operations and transmits the generated operation control signal from the communication I / F unit 5A to the robot control device RT.

[0026] (3) Robot Control Device RT FIGS. 4 and 5 are block diagrams showing an example of the hardware configuration and software configuration, respectively, of a robot control device RT having an operation support function according to an embodiment of the present invention.

[0027] The robot control device RT is composed of, for example, a personal computer attached to the robot RB, and is configured by connecting a processor 1B constituting a central control unit (CPU) with a memory unit having a program memory unit 2B and a data memory unit 3B, a sensor I / F unit 4B, a communication I / F unit 5B, and a robot I / F unit 7B via a bus 6B.

[0028] The sensor I / F unit 4B captures video data output from the camera CM. The communication I / F unit 5B transmits and receives video data and various control signals to and from the remote control terminal UT via the network NW. The robot I / F unit 7B outputs control signals to the robot RB and the suction device AS.

[0029] The control unit 1B has control functions for realizing one embodiment of the present invention, including a video data acquisition processing unit 11B, a respiratory state estimation processing unit 12B, a display video data transmission processing unit 13B, a catheter position detection processing unit 14B, and a catheter operation control unit 15B.

[0030] Each of the processing units 11B to 15B is realized by causing the processor of the control unit 1B to execute an application program stored in the program storage unit 2B. Note that some or all of the processing units 11B to 15B may be realized using hardware such as an LSI (Large Scale Integration) or an ASIC.

[0031] The video data acquisition processing unit 11B captures video data of a predetermined range including the face of the patient PT captured by the camera CM via the sensor I / F unit 4B, and temporarily stores the captured video data in the data storage unit 3B.

[0032] The respiratory condition estimation processing unit 12B reads the video data from the data storage unit 3B at regular time intervals (e.g., the frame period of the video data) and extracts from the video data an image representing the exhaled breath from the nasal cavity or oral cavity of the patient PT. The respiratory condition estimation processing unit 12B then estimates the respiratory condition of the patient PT from the extracted image representing the exhaled breath. An example of this respiratory condition estimation processing will be described in the operation example.

[0033] The catheter position detection processing unit 14B reads the video data from the data storage unit 3B at the above-mentioned fixed time intervals (for example, the frame period of the video data) and detects the positional relationship of the catheter CT with respect to the nasal cavity or oral cavity of the patient PT from the video data. An example of the detection process of the positional relationship will also be described below as an operation example.

[0034] The display video data transmission processing unit 13B generates display video data corresponding to the positional relationship of the catheter CT relative to the nasal cavity or oral cavity detected by the catheter position detection processing unit 14B based on the video data.

[0035] At the same time, the display video data transmission processing unit 13B superimposes an indicator image representing the respiratory state of the patient PT estimated by the respiratory state estimation processing unit 12B on the generated display video data, and transmits the display video data after the superimposition process from the communication I / F unit 5B to the remote control terminal UT.

[0036] The catheter operation control unit 15B receives various control signals transmitted from the remote control terminal UT via the communication I / F unit 5B. Then, in accordance with the received control signals, the catheter operation control unit 15B controls the operation of the robot RB or the suction device AS via the robot I / F unit 7B, thereby inserting and removing the catheter CT into and from the nasal cavity or oral cavity, and suctioning phlegm.

[0037] (Example of Operation) Next, an example of operation of the remote control system configured as above will be described.

[0038] FIG. 6 is a flowchart showing an example of the processing procedure and processing contents of the remote operation assistance control executed by the control unit 1A of the remote control terminal UT and the control unit 1B of the robot control device RT.

[0039] In the remote control terminal UT, for example, when an operator US performs a control start operation using the input device IN, the control unit 1A of the remote control terminal UT detects the control start operation in step S10, generates a control start signal, and transmits it from the communication I / F unit 5A to the robot control device RT.

[0040] In response to this, when the control unit 1B of the robot control device RT receives the control start signal from the remote control terminal UT in step S11, under the control of the video data acquisition processing unit 11B, in step S12, it captures video data including the patient's face captured by the camera CM via the sensor I / F unit 4B, and temporarily stores the captured video data in a buffer area in the data storage unit 3B.

[0041] (1) Estimation of Respiratory State of Patient PT First, in step S13, the control unit 1B of the robot control device RT estimates the respiratory state of the patient PT under the control of the respiratory state estimation processing unit 12B as follows.

[0042] That is, the respiratory condition estimation processor 12B first reads video data from the buffer area of ​​the data storage unit 3B at regular time intervals (e.g., the frame period of the video data). Then, each time video data is read, an image representing the exhaled breath from the nasal cavity or oral cavity of the patient PT is extracted from the video data. For example, the respiratory condition estimation processor 12B extracts an image representing a temperature distribution pattern specific to the exhaled breath that appears near the nasal cavity or oral cavity of the patient PT from the infrared image.

[0043] 8 shows an example of a temperature distribution pattern image VS of exhaled air appearing near the nasal cavity or oral cavity. That is, the difference between inhaled and exhaled air appears as a temperature difference in the temperature distribution pattern, and the difference between the beginning and end of exhaled air appears as a size of the temperature distribution pattern.

[0044] The respiratory condition estimation processing unit 12B then compares the extracted image VS representing the temperature distribution pattern of the exhaled breath with a preset basic image pattern of temperature distribution to estimate the respiratory condition of the patient PT, i.e., the inhalation and exhalation states. Based on the estimation result of the respiratory condition, the respiratory condition estimation processing unit 12B then generates an indicator image for displaying the respiratory condition.

[0045] 9 shows an example of a data table in which basic pattern images of temperature distribution and indicator images IS are associated with respiratory states, and the data table is stored in advance in the data storage unit 3 B. The respiratory state estimation processing unit 12 B reads out the corresponding indicator image from the data table based on information representing the respiratory state estimation result.

[0046] (2) Detection of the positional relationship of the catheter CT relative to the nasal cavity or oral cavity Next, in step S14, the control unit 1B of the robot control device RT, under the control of the catheter position detection processing unit 14B, executes a process to detect the positional relationship of the catheter CT relative to the nasal cavity or oral cavity of the patient PT as follows.

[0047] That is, the catheter position detection processing unit 14B reads the image data from the buffer area of ​​the data storage unit 3B at the same time intervals (e.g., frame period of the image data) as in the case of estimating the respiratory state described above. Then, based on the read image data, the positional relationship of the catheter CT with respect to the nasal cavity or oral cavity is detected. The following two methods can be considered for detecting the positional relationship.

[0048] (2-1) Method based on the distance between the nasal cavity or oral cavity and the catheter CT The catheter position detection processing unit 14B first extracts an image area including the nasal cavity or oral cavity and the catheter CT from the video data, and calculates the distance from the entrance of the nasal cavity or oral cavity to the tip of the catheter CT in the extracted image area. Figure 11 shows an example of the calculated distance D.

[0049] Then, the catheter position detection processing unit 14B compares the calculated distance with a preset threshold value and determines whether the distance is greater than the threshold value, greater than "0" but less than or equal to the threshold value, or less than or equal to "0." Here, assuming that an operator with average-sized hands directly operates the catheter by hand, the threshold value is set to a value corresponding to the length of the operator's hand, for example, 20 cm.

[0050] The catheter position detection processing unit 14B determines the working state based on the distance determination result. For example, if the distance is greater than a threshold value, it is recognized as a state before the start of catheter CT operation. If the distance is greater than "0" and equal to or less than the threshold value, it is recognized as a state in which the catheter CT is being inserted or removed from the nasal cavity or oral cavity. If the distance is equal to or less than "0", it is recognized as a state in which the tip of the catheter CT is inserted into the nasal cavity or oral cavity and treatment is being performed.

[0051] (2-2) Method based on the position of the catheter CT in the working space This method divides the working space into multiple spaces based on the working axis relative to the nasal cavity or oral cavity, and determines in which of the multiple spaces the position of the catheter CT is located.

[0052] 12 is a diagram for explaining the above-mentioned method (2-2). The catheter position detection processing unit 14B first recognizes the position and internal shape of the nasal cavity or oral cavity of the patient PT from the video data read from the buffer area of ​​the data storage unit 3B.

[0053] Specifically, an image of the nose or mouth is first recognized from the image of the face using, for example, a pattern recognition technique, and then an image of the nasal cavity or oral cavity is identified from the recognized image of the nose or mouth, and its position and internal shape are recognized. The catheter position detection processing unit 14B then estimates the axial direction AX based on the recognized internal shape of the nasal cavity or oral cavity.

[0054] The catheter position detection processor 14B then sets a cone-shaped virtual space QE in the axial direction AX, starting from the entrance of the nasal cavity or oral cavity. The height of the cone forming this virtual space is set to, for example, 20 cm, which corresponds to the length of a typical operator US's hand. Note that when the treatment target is the nasal cavity, the virtual space QE is set to have a shape that is a combination of two cones set respectively for the two nostrils of the nasal cavity.

[0055] The catheter position detection processing unit 14B detects the position of the tip of the catheter CT from the video data and compares the detected position of the tip of the catheter CT with the position in the virtual space QE to determine the operating state. For example, if the position of the tip of the catheter CT is farther from the virtual space QE, it is estimated to be in a first state, in which the catheter CT has not yet been operated. If the position of the tip of the catheter CT is within the virtual space QE, it is estimated to be in a second state, in which the catheter CT is being inserted or removed from the nasal cavity or oral cavity. If the position of the tip of the catheter CT is closer to the nasal cavity or oral cavity than the virtual space QE, it is estimated to be in a third state, in which the tip of the catheter CT is inserted into the nasal cavity or oral cavity and treatment is being performed.

[0056] (3) Generation and Transmission of Display Video Data In step S15, the control unit 1B of the robot control device RT, under the control of the display video data transmission processing unit 13B, executes the process of generating and transmitting display video data as follows.

[0057] FIG. 7 is a flowchart showing an example of the processing procedure and processing content of the display video data generation and transmission processing executed by the display video data transmission processing unit 13B.

[0058] In steps S31, S32, and S33, the display video data transmission processing unit 13B determines whether the result of the work state determination by the catheter position detection processing unit 14B is the first state, the second state, or the third state.

[0059] If the result of this determination is that the working state is the first state, then in step S34, the display video data transmission processing unit 13B generates a display video showing the entire face based on the video data stored in the buffer area of ​​the data storage unit 3B. Figure 13 shows an example of the display video generated at this time. This example shows a case where an image of the entire face of the patient PT is arranged over the entire one frame E0.

[0060] On the other hand, if the working state is determined to be the second state, the display video data transmission processing unit 13B generates a display video in which a video showing the entire face and a video of an enlarged portion of the nasal cavity or oral cavity are arranged side by side, based on the video data stored in the buffer area of ​​the data storage unit 3B, in step S35. Figure 14 shows an example of the display video generated at this time. In this example, one frame E0 is divided into two areas E1 and E2, and a video showing the entire face of the patient PT and a video of an enlarged portion of the nasal cavity or oral cavity are arranged in each of the divided areas E1 and E2, respectively.

[0061] Furthermore, if the working state is determined to be the third state, in step S36, the display video data transmission processing unit 13B generates a display video based on the video data stored in the buffer area of ​​the data storage unit 3B, in which an image showing the entire face, an enlarged image of the nasal cavity or oral cavity, and an enlarged image of the suction device AS (e.g., the tank part) are arranged side by side.

[0062] An example of the display image generated at this time is shown in Figure 15. In this example, one frame E0 is divided into three regions E1, E2, and E3, and an image showing the entire face of the patient PT, an enlarged image of the nasal cavity or oral cavity, and an enlarged image of the suction device AS (e.g., the tank portion) are arranged in each of the divided regions E1, E2, and E3.

[0063] At this time, the display video data transmission processing unit 13B collects suction sounds generated in the suction device AS or the suction tube AT during the suction treatment of phlegm, or generates synthetic sound data representing the suction sounds. The generated suction sounds or data representing the synthetic sounds may then be transmitted to the remote control terminal UT together with the display video data, and the suction sound data may be amplified and output from the speaker SP provided in the output device OU of the remote control terminal UT.

[0064] Next, in step S37, the display video data transmission processing unit 13B superimposes the indicator image IS representing the respiratory state generated by the respiratory state estimation processing unit 12B on the generated display video data.

[0065] Finally, in step S38, the display video data transmission processing section 13B transmits the display video data on which the indicator image IS is superimposed from the communication I / F section 5B to the remote control terminal UT.

[0066] The above-described generation and transmission process of the display video data is executed at regular time intervals (for example, a video frame period).

[0067] (4) Remote Control of Catheter CT When the display video data is transmitted from the robot control device RT, the control unit 1A of the remote control terminal UT receives the display video data via the communication I / F unit 5A under the control of the video data reception and display processing unit 11A in step S16, and outputs the received display video data from the input / output I / F unit 4A to the output device OU and displays it on the display device DP in step S17. Figure 10 shows an example of display video data displayed on the display device DP. This example shows a case where an image of the patient PT's face is displayed with an indicator image in the margin. The display of the display video data is updated at regular time intervals (e.g., the frame period of the video data).

[0068] Therefore, the operator US can continuously check the state of the patient PT's face as well as the state of his / her breathing from the display image data. The operator US operates the input device IN while checking the state of the patient PT's face and breathing.

[0069] When the control unit 1A of the remote control terminal UT detects operation of the input device IN in step S18, under the control of the catheter operation control signal transmission processing unit 12A, it generates an operation control signal corresponding to the operation content in step S19 and transmits the generated operation control signal from the communication I / F unit 5A to the robot control device RT.

[0070] In response to this, when the control unit 1B of the robot control device RT receives the catheter manipulation control signal in step S20, under the control of the catheter manipulation control unit 15B, it controls the robot RB and the suction device AS in accordance with the received manipulation control signal in step S21. As a result, a series of procedures including insertion of the catheter CT into the nasal cavity or oral cavity, suction of phlegm after insertion, and removal of the catheter CT after suction are performed by remote control.

[0071] (5) Change in display image during remote operation In one embodiment, as described above, the working status related to the treatment is determined based on the position of the catheter CT, and the display image is switched according to the determined working status.

[0072] For example, in the first state before the catheter CT operation is started, an image of the entire face of the patient PT is displayed in one frame E0 as shown in Fig. 13. This allows the operator US to check the facial expression of the patient PT before operating the catheter CT.

[0073] In the second state in which the catheter CT is being inserted into the nasal cavity or oral cavity of the patient PT, one frame E0 is divided into two areas E1 and E2, and an image of the entire face of the patient PT and an enlarged image of the nasal cavity or oral cavity are displayed in these areas, as shown in Fig. 14. This allows the operator US to clearly grasp the target area from the enlarged image of the nasal cavity or oral cavity while checking the appearance of the entire face of the patient PT, and to perform the catheter CT insertion operation.

[0074] Furthermore, in a third state in which treatment is being performed with the tip of the catheter CT inserted into the nasal cavity or oral cavity, an image of the entire face of the patient PT, an enlarged image of the nasal cavity or oral cavity region, and an enlarged image of the suction device AS are displayed in each of divided regions E1, E2, and E3 of one frame E0, as shown in Fig. 15. This allows the operator US to operate the suction treatment of phlegm while checking the appearance of the entire face of the patient PT and the state of the nasal cavity or oral cavity region, and further allows the operator US to check from the image of the suction device AS whether the phlegm has been reliably aspirated.

[0075] Then, when the sputum suction procedure is completed and the operator US removes the catheter CT from the nasal cavity or oral cavity, it is estimated that the working state has changed to the second state based on the position of the catheter CT, and the display image changes to one that shows an image of the patient PT's entire face and an enlarged image of the nasal cavity or oral cavity area, as shown in Figure 14.

[0076] Furthermore, when the operator US retracts the catheter CT to a position further away from the patient PT, it is estimated that the working state has returned to the first state, and the displayed image is switched to an image of the entire face of the patient PT.

[0077] On the other hand, in one embodiment, an indicator image IS indicating the respiratory state of the patient PT is displayed in the display image. For example, as shown in Fig. 9, when the patient PT is inhaling, the length of the indicator image IS becomes shorter, and when the patient PT is exhaling, the length of the indicator image IS becomes longer as the amount of exhaled air increases. Therefore, the operator US can insert and remove the catheter CT and suction phlegm at the optimal timing while checking the respiratory state of the patient PT in real time using the indicator image IS.

[0078] (Effects) As described above, in one embodiment, the robot control device RT detects the state of breathing from video data of the patient PT and estimates the respiratory state of the patient PT based on the result, and also detects the position of the tip of the catheter CT relative to the nasal cavity or oral cavity from the video data and determines the working state of the treatment based on the detection result. Then, the displayed image is switched based on the result of the determination of the working state, and an indicator image IS representing the estimated result of the respiratory state is superimposed on the switched image, and this display image is transmitted to the remote control terminal UT for display.

[0079] Therefore, by providing an automatic display image switching function, the operator US can view the display image required for the current work for each work state related to the treatment without having to perform any display image switching operations himself, thereby enabling smooth and reliable remote operation of the robot RB.

[0080] Furthermore, an indicator image IS showing the respiratory condition of the patient PT is superimposed on the display image. This allows the operator US to grasp the respiratory condition of the patient PT by looking at the indicator image IS, and allows the catheter CT and phlegm suction operation to be performed when the patient PT is in an optimal respiratory condition. This makes it possible to perform treatment smoothly and reliably without causing pain to the patient PT.

[0081] In other words, according to one embodiment, the operator US can properly grasp the condition of the patient PT, thereby enabling a series of remote operations related to suctioning phlegm from the patient PT to be carried out quickly, reliably, and without causing unnecessary pain to the patient PT.

[0082] [Other embodiments] (1) In one embodiment, the position of the catheter CT relative to the nasal cavity and Mataki oral cavity is detected based on video data, but this is not limited to this, and any device that can detect the position of the catheter may be used.

[0083] (2) In one embodiment, the respiratory condition of the patient PT is estimated based on the temperature distribution of the exhaled or exhaled breath contained in the infrared image. However, the respiratory condition may also be estimated based on other methods, such as the wind speed of the exhaled breath, or by detecting changes in the swelling of the patient PT's chest or abdomen based on the image data.

[0084] (2) In addition to displaying an indicator image IS, other methods for presenting the respiratory status of the patient PT to the operator US can be used, for example, by using a pen-shaped device capable of force control, which gradually increases the weight of the movement while the patient is exhaling and returns to the original weight when the patient finishes exhaling, or by using a device equipped with a vibrator, which gradually increases the vibration while the patient is exhaling and stops the vibration when the patient finishes exhaling.

[0085] (3) In one embodiment, the displayed image is automatically switched in accordance with the determination result of the working state of the treatment, but in addition, the automatic switching control may be interrupted and the image may be switched when the operator US performs a switching operation on the remote side A. Also, a learning model may be trained to perform a process for estimating the working state of the operator US based on the position of the catheter CT relative to the nasal cavity or oral cavity, and the working state may be estimated using the learned learning model.

[0086] (4) In one embodiment, when generating display video data, one frame is divided into multiple equal areas, and a different display video is displayed in each area. However, priorities may be set for multiple display videos for each work state, and the display size of videos with higher priorities may be increased, or the display position may be changed to a position that is easier for the operator to see.

[0087] (5) During each operation of inserting the catheter CT, suctioning phlegm, and removing the catheter CT, the condition of the patient PT and the condition of the surrounding area may be monitored, for example, based on video data, and if an abnormality in the patient PT or an abnormal object is detected in the surrounding area, the image containing the abnormal object may be displayed more emphasized than other images.

[0088] (6) The timing of presenting the display image may be appropriately controlled taking into consideration transmission delays between the remote control terminal UT and the robot control device RT and processing delays in the remote control terminal UT and the robot control device RT. For example, the breathing rhythm of the patient PT is estimated based on the video data or information acquired from a vital sensor, and the timing of presenting the display image data is advanced by the time corresponding to the transmission delay and processing delay based on the estimated breathing rhythm.

[0089] (7) In one embodiment, the main functions of the present invention, such as estimating the respiratory state, determining the working state, and generating video data for display in accordance with the results thereof, are described as being located in the robot control device RT. However, the present invention is not limited to the above example, and each of the main functions may be located in the remote control terminal UT.

[0090] Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. In other words, when implementing the present invention, specific configurations according to the embodiments may be appropriately adopted.

[0091] In short, this invention is not limited to the above-described embodiments, and in the implementation stage, the components can be modified and embodied without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

[0092] A...Operating side B...Remote side NW...Network UT...Remote control terminal IN...Input device OU...Output device US...Operator RT...Robot control device RB...Robot CT...Catheter AS...Suction device CM...Camera PT...Patient 1A, 1B...Controller 2A, 2B...Program memory 3A, 3B...Data memory 4A...Input / output I / F 4B...Sensor I / F 5A, 5B...Communication I / F 6A, 6B...Bus 7B...Robot I / F 11A...Video data reception and display processor 12A...Catheter operation control signal transmission processor 11B...Video data acquisition processor 12B...Respiratory state estimation processor 13B...Display video data transmission processor 14B...Catheter position detection processor 15B...Catheter operation controller

Claims

1. In a remote operation system in which a remote control terminal located on the operating side and a robot control device located on the remote side are connected via a network, and the robot control device controls the robot's catheter in accordance with various control signals sent from the remote control terminal in response to operation by an operator to perform a predetermined treatment on a patient's treatment target area, an operation assistance device provided in either the remote control terminal or the robot control device comprises: a first processing unit that estimates the respiratory state of the patient and generates respiratory state information representing the estimated respiratory state; and a second processing unit that presents the generated respiratory state information to the operator.

2. The operation assistance device according to claim 1, wherein the first processing unit estimates the patient's respiratory state as an inhalation state and an exhalation state, and generates the respiratory state information representing each state.

3. An operation assistance device as described in claim 1, further comprising: a third processing unit that detects the position of the catheter relative to the treatment target area and determines the working status of the treatment based on the detected position; a fourth processing unit that generates display information that shows the treatment target area of ​​the patient in different display forms depending on the working status; and a fifth processing unit that presents the display information to the operator.

4. The operation assistance device according to claim 3, wherein the third processing unit determines, based on the position of the catheter relative to the treatment area, whether the working state is a first state before the operator starts operating the catheter, or a second state in which the operator inserts and removes the catheter relative to the treatment area; and the fourth processing unit generates first display information including the treatment area and its surrounding area when the working state is determined to be the first state, and generates second display information including information on an enlarged view of the treatment area when the working state is determined to be the second state.

5. The operation assistance device according to claim 4, wherein the third processing unit further determines a third state in which the operator performs the treatment using the catheter on the inside of the treatment target area based on the position of the catheter, and the fourth processing unit generates third display information including at least one of the first display information and the second display information and information representing the state of the treatment when the working state is determined to be the third state.

6. The operation assistance device described in claim 3, wherein the third processing unit sets up a cone-shaped virtual space in which the tip contacts the treatment target area, and determines the working state related to the treatment by determining whether the detected position of the catheter is included in the virtual space.

7. In a remote operation system in which a remote control terminal located on an operating side and a robot control device located on a remote side are connected via a network, and the robot control device controls a catheter of a robot in accordance with various control signals transmitted from the remote control terminal in response to operations by an operator to perform a predetermined treatment on a treatment target part of a patient, an operation assistance method executed by either the remote control terminal or the robot control device, the operation assistance method comprising: a step of estimating the respiratory state of the patient and generating respiratory state information representing the estimated respiratory state; and a step of presenting the generated respiratory state information to the operator.

8. A program for causing a processor included in an operation assistance device according to any one of claims 1 to 6 to execute the processing executed by a processing unit included in said operation assistance device.

Images