Information presenting system, information presenting method, and program

Abstract

An information presenting system (1) according to one aspect comprises a first terminal (10), a second terminal (20) including a camera (25), and a palpation device (30). The first terminal includes an instruction acquiring unit (113) that acquires first input information and second input information input by a first user, and an instruction video creating unit (114) that creates a first instruction video and a second instruction video and transmits the first instruction video and the second instruction video to a second terminal. The second terminal includes: an intensity adjusting unit (215) that creates a third instruction video obtained by adjusting an operation intensity indicated by the second instruction video; a display control unit (212) that creates a superimposed video in which the first instruction video and the third instruction video are superimposed on a captured video, and that displays the superimposed video; a measurement data acquiring unit (216) that acquires, from the palpation device, measurement data relating to palpation performed by a second user using the palpation device on the basis of the superimposed video; and a palpation result creating unit (218) that creates a palpation result and transmits the palpation result to the first terminal.

Description

Information presentation system, information presentation method, and program

[0001] One aspect of the present invention relates to an information presentation system, an information presentation method, and a program.

[0002] Due to the extreme aging of society, there is an absolute shortage of medical personnel, making it impossible to maintain the traditional medical examination format that assumes close contact and face-to-face contact.For this reason, attempts are being made to realize telemedicine, which uses information and communication technology (ICT) to eliminate the constraints of time and space so that one medical personnel can examine more patients efficiently.

[0003] For example, there are attempts to use video calls to visually check the physical condition of patients from remote locations. In this attempt, nurses can visually check the physical condition of patients from remote locations via video calls. However, there are other ways to observe physical conditions besides visual inspection, such as auscultation and palpation, and this does not support these. Palpation is important for understanding physical conditions that cannot be judged visually. Therefore, there is great significance in making palpation remote.

[0004] Although not palpation, there is a technology in the teleoperation approach that transmits haptic sensations to a remote location during a surgical procedure (force haptic transmission in telemedicine). For example, this technology involves faithfully reproducing haptic sensations using a master-slave robot. Non-Patent Document 1 describes a robotic force sensation reproduction method for brain surgery. Non-Patent Document 2 describes a robotic haptic sensation reproduction method for endoscopic surgery. In these methods, the procedure is carried out while checking the on-site situation via video call from a remote location and receiving haptic feedback related to the robot operation.

[0005] Akio Morita et al., "Development of Robotic Surgery in the Field of Neurosurgery and Its Future Prospects," Neurosurgery, Vol. 31, No. 8, August 2022, pp. 500-508; Kohei Onishi et al., "Real Haptics Opening Up the Future Society," Journal of the Japanese Society of AEM, Vol. 25, No. 1, 2017, pp. 9-16

[0006] However, the techniques described in Non-Patent Documents 1 and 2 require large-scale devices, so the sites where they can be introduced are limited.The techniques described in Non-Patent Documents 1 and 2 are specialized for the treatment to be performed, so the placement of the device and the functions of the robot arm must be customized according to the treatment.

[0007] Another example of a case in which haptic transmission in a remote location can be applied is the common daily palpation of a specific part of a patient's body, but compared to surgery, palpation requires a wider variety of parts to be examined and ways of touching those parts, so it is necessary to acquire video call footage tailored to each individual patient and to respond flexibly to how to touch them.For this reason, it is difficult to directly apply the technologies described in Non-Patent Documents 1 and 2 to remote palpation.

[0008] This invention was made with the above-mentioned circumstances in mind, and provides a technology that enables a doctor in a remote location to appropriately issue instructions (palpation instructions) to a local nurse to touch a patient's body part, while also enabling the doctor to understand the results of the palpation of any part performed by the nurse.

[0009] In order to solve the above problem, an information presentation system according to one aspect of the present invention includes a first terminal, a second terminal including a camera and capable of wirelessly communicating with the first terminal, and a palpation device capable of communicating with the second terminal. The first terminal includes an instruction acquisition unit that acquires first input information input by a first user regarding a target area of ​​palpation on a subject and second input information regarding a type and intensity of palpation motion, and an instruction image creation unit that creates a first instruction image that visually indicates the target area based on the first input information, creates a second instruction image that visually indicates the type and intensity of motion based on the second input information, and transmits the first instruction image and the second instruction image to the second terminal. The second terminal includes an intensity correction unit that creates a third instruction image by correcting the action intensity instructed by the second instruction image, a display control unit that creates a superimposed image by superimposing the first instruction image and the third instruction image on an image captured by a camera and displays the superimposed image, a measurement data acquisition unit that acquires measurement data of the palpation performed by the second user using the palpation device based on the superimposed image from the palpation device, and a palpation result creation unit that creates a palpation result based on the measurement data and transmits the palpation result to the first terminal.

[0010] An information presentation method according to one aspect of the present invention is an information presentation method using an information presentation system including a first terminal, a second terminal including a camera and capable of wirelessly communicating with the first terminal, and a palpation device capable of communicating with the second terminal, the information presentation method including the steps of: the first terminal acquiring first input information input by a first user regarding a target range of palpation on a subject, and second input information regarding a type and intensity of a palpation action; the first terminal creating a first instruction video visually indicating the target range based on the first input information; and creating a second instruction video visually indicating the type and intensity of the action based on the second input information. and transmitting the first instruction video and the second instruction video to the second terminal; the second terminal creating a third instruction video by correcting the action intensity instructed by the second instruction video; the second terminal creating a superimposed video by superimposing the first instruction video and the third instruction video on an image captured by a camera and displaying the superimposed video; the second terminal acquiring, from the palpation device, measurement data of the palpation performed by the second user using the palpation device based on the superimposed video; and the second terminal creating a palpation result based on the measurement data and transmitting the palpation result to the first terminal.

[0011] According to one aspect of the present invention, a doctor in a remote location can appropriately instruct a local nurse to touch a patient's body part, while allowing the doctor to understand the results of the palpation of any part performed by the nurse.

[0012] FIG. 1 is a schematic diagram showing an example of the configuration of an information presentation system according to an embodiment. FIG. 2 is a block diagram showing an example of the hardware configuration of the information presentation system according to an embodiment. FIG. 3 is a block diagram showing an example of the functional configuration of the information presentation system according to an embodiment. FIG. 4 is a diagram showing an example of a captured image displayed by the information presentation system according to an embodiment. FIG. 5 is a diagram showing an example of an image capture instruction input screen displayed by the information presentation system according to an embodiment. FIG. 6 is a diagram showing an example of an image capture instruction superimposed image displayed by the information presentation system according to an embodiment. FIG. 7 is a diagram showing an example of a captured image after the position of a camera has been adjusted in the information presentation system according to an embodiment. FIG. 8 is a diagram showing an example of a first input screen displayed by the information presentation system according to an embodiment. FIG. 9 is a diagram showing an example of a first superimposed image displayed by the information presentation system according to an embodiment. FIG. 10 is a diagram showing an example of a first superimposed image after two markers have been placed in the information presentation system according to an embodiment. FIG. 11 is a diagram showing an example of a tracking superimposed image displayed by the information presentation system according to an embodiment. FIG. 12 is a diagram showing an example of a second input screen displayed by the information presentation system according to an embodiment. FIG. 13 is a diagram showing an example of a second superimposed image displayed by the information presentation system according to an embodiment. FIG. 14 is a diagram showing an example of a set of setting values ​​for the basic range of palpation used in the information presentation system according to an embodiment. FIG. 15 is a diagram showing another example of the second superimposed image displayed in the information presentation system according to the embodiment. FIG. 16 is a diagram showing yet another example of the second superimposed image displayed in the information presentation system according to the embodiment. FIG. 17 is a diagram showing yet another example of the second superimposed image displayed in the information presentation system according to the embodiment. FIG. 18 is a diagram showing an example of the second instruction image displayed in the information presentation system according to the embodiment. FIG. 19 is a diagram showing another example of the second instruction image displayed in the information presentation system according to the embodiment. FIG. 20 is a diagram showing an example of a set of force settings for the doctor and nurse used in the information presentation system according to the embodiment, and correction of the second instruction image based on this set of setting values. FIG. 21 is a diagram showing an example of the third superimposed image displayed in the information presentation system according to the embodiment.FIG. 22 is a diagram illustrating a state in which palpation is performed in the information presentation system according to the embodiment. FIG. 23 is a diagram illustrating the apparent hardness of a body part of a patient. FIG. 24 is a diagram illustrating an example of a palpation result created in the information presentation system according to the embodiment. FIG. 25 is a flowchart illustrating an example of an operation of the information presentation system according to the embodiment. FIG. 26 is a flowchart illustrating an example of an operation of the information presentation system according to the embodiment. FIG. 27 is a flowchart illustrating an example of an operation of the information presentation system according to the embodiment. FIG. 28 is a flowchart illustrating an example of an operation of the information presentation system according to the embodiment. FIG. 29 is a block diagram illustrating an example of a functional configuration of an information presentation system according to a first modified example of the embodiment. FIG. 30 is a diagram illustrating an example of AR tracking processing performed in the information presentation system according to the first modified example of the embodiment. FIG. 31 is a flowchart illustrating an example of an operation of the information presentation system according to the first modified example of the embodiment. FIG. 32 is a block diagram illustrating an example of a functional configuration of an information presentation system according to a second modified example of the embodiment. FIG. 33 is a diagram illustrating a state in which palpation is performed in the information presentation system according to the second modified example of the embodiment.

[0013] Hereinafter, embodiments will be described with reference to the drawings. In the following description, components having substantially the same functions and configurations will be designated by the same reference numerals. When particularly distinguishing between elements having similar configurations, different letters or numbers may be added to the end of the same reference numerals.

[0014] 1. Embodiment An information presentation system according to an embodiment will be described below. In the following, a remote palpation system will be described as an example of the information presentation system.

[0015] 1.1 Configuration 1.1.1 Configuration of the Information Presentation System The configuration of the information presentation system according to this embodiment will be described with reference to FIG. 1. FIG. 1 is a schematic diagram showing an example of the configuration of the information presentation system. FIG. 1 illustrates a situation in which, using the information presentation system 1, a video call is initiated between a remote location where a doctor is located and an on-site location where a patient and nurse are located, the nurse photographs the patient, the doctor issues visual instructions to the nurse, the nurse performs palpation of the patient based on the instructions, and the results of the palpation are communicated to the doctor. In this way, the information presentation system 1 is not a remote-controlled robot, but a system that supports palpation instructions from a doctor to a nurse in a situation where an on-site nurse touches the patient's body parts instead of a doctor in a remote location.

[0016] As shown in FIG. 1 , the information presentation system 1 includes a first terminal 10 , a second terminal 20 , and a palpation device 30 .

[0017] The first terminal 10 is, for example, a personal computer (PC), a tablet computer, or a smartphone. In the example of FIG. 1 , the first terminal 10 is a notebook PC. The first terminal 10 has, for example, a function to communicate with other devices and a function to create visual instructions. The first terminal 10 is configured to be able to communicate with the second terminal 20. For example, the first terminal 10 is connected to the second terminal 20 wirelessly. The first terminal 10 is used by a doctor (user).

[0018] The second terminal 20 is, for example, a PC, a tablet computer, a smartphone, or AR (Augmented Reality) glasses. In the example of FIG. 1 , the second terminal 20 is a smartphone. The second terminal 20 has, for example, a function for capturing video, a function for communicating with other devices, and a function for creating palpation results. The second terminal 20 is configured to be able to communicate with the first terminal 10 and the palpation device 30. For example, the second terminal 20 is connected to the palpation device 30 wirelessly or via a wire. The second terminal 20 is used by a nurse (user).

[0019] The palpation device 30 is, for example, a pressure sensor or a glove-type device. The palpation device 30 has functions such as measuring the force with which a nurse pinches or presses the body of a patient (subject) and the hardness of the palpation area, as well as the function of communicating with other devices. The palpation device 30 is configured to be able to communicate with the second terminal 20. The palpation device 30 is used by a nurse.

[0020] 1.1.2 Hardware Configuration of Information Presentation System The hardware configuration of the information presentation system 1 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the hardware configuration of the information presentation system 1.

[0021] 1.1.2.1 Hardware Configuration of First Terminal The following describes the hardware configuration of the first terminal 10. As shown in Fig. 2 , the first terminal 10 includes a processor 11, a memory 12, a communication interface (I / F) 13, a user interface (I / F) 14, a storage 15, and a display unit 16.

[0022] The processor 11 is, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), or an FPGA (Field Programmable Gate Array). The processor 11 executes, for example, the processing of the first terminal 10 described below. The processing of the first terminal 10 described below includes a captured image display process, a capturing instruction input screen display process, a capturing instruction display process, a first input screen display process, a first instruction display process, a tracking image display process, a second input screen display process, a second instruction display process, a palpation result display process, a capturing instruction input screen creation process, a first input screen creation process, a second input screen creation process, a capturing instruction acquisition process, a first instruction acquisition process, a second instruction acquisition process, a capturing instruction creation process, a capturing instruction transmission process, a first instruction creation process, a first instruction transmission process, a second instruction creation process, and a second instruction transmission process.

[0023] The memory 12 includes, for example, a read-only memory (ROM) and a random access memory (RAM). The ROM stores, for example, a program for causing the processor 11 to execute the processing of the first terminal 10 described below. The RAM is used as a working area for the processor 11. The RAM temporarily stores, for example, the above-mentioned program executed by the processor 11 and data used when the processing of the first terminal 10 described below is executed.

[0024] The communication interface 13 controls communication between the first terminal 10 and the second terminal 20. The communication interface 13 transmits audio, video, and data to the second terminal 20. The communication interface 13 receives audio, video, and data from the second terminal 20.

[0025] The user interface 14 is a means operated by the user to input various information, and is, for example, a mouse, a keyboard, a touch panel, or the like.

[0026] The storage 15 is a storage medium used to store data, programs, etc. The storage 15 stores, for example, data DTb (a set of setting values ​​for the basic range of palpation) described later.

[0027] The display unit 16 is, for example, a display, and displays the video and data received from the processor 11.

[0028] Although not shown in FIG. 2, the first terminal 10 includes a camera, a microphone, and a speaker.

[0029] 1.1.2.2 Hardware Configuration of Second Terminal The following describes the hardware configuration of the second terminal 20. As shown in Fig. 2 , the second terminal 20 includes a processor 21, a memory 22, a communication interface (communication I / F) 23, a storage 24, a camera 25, and a display unit 26.

[0030] The processor 21 is, for example, a CPU, an MPU, a GPU, an FPGA, etc. The processor 21 executes, for example, the processing of the second terminal 20, which will be described later. The processing of the second terminal 20, which will be described later, includes a captured image acquisition processing, a captured image display processing, a capturing instruction display processing, a first instruction display processing, a tracking image display processing, a second instruction display processing, a marker position acquisition processing, an AR tracking processing, an intensity correction processing, a measurement data acquisition processing, a hardness calculation processing, and a palpation result creation processing.

[0031] The memory 22 includes, for example, a ROM and a RAM. The ROM stores, for example, a program for causing the processor 21 to execute the processing of the second terminal 20 described below. The RAM is used as a work area for the processor 21. The RAM temporarily stores, for example, the above-mentioned program executed by the processor 21 and data used when the processing of the second terminal 20 described below is executed.

[0032] The communication interface 23 controls communication between the second terminal 20 and the first terminal 10 and the palpation device 30. The communication interface 23 transmits audio, video, and data to the first terminal 10. The communication interface 23 receives audio, video, and data from the first terminal 10. The communication interface 23 receives palpation measurement data DT from the palpation device 30.

[0033] The storage 24 is a storage medium used to store data, programs, etc. The storage 24 stores, for example, data DTp (a set of doctor and nurse force settings) described below.

[0034] The camera 25 is, for example, an RGB camera. Hereinafter, the video captured by the camera 25 will also be referred to as a “captured video VC.” The camera 25 transmits the captured video VC to the processor 21.

[0035] The display unit 26 is, for example, a display, and displays the video received from the processor 21.

[0036] Although not shown in FIG. 2, the second terminal 20 includes a microphone, a speaker, and a user interface.

[0037] 1.1.3 Functional Configuration of Information Presentation System The functional configuration of the information presentation system 1 will be described using Fig. 3. Fig. 3 is a block diagram showing an example of the functional configuration of the information presentation system 1. Fig. 3 also shows the communication interface 13, user interface 14, storage 15, and display unit 16 of the first terminal 10, the communication interface 23, storage 24, camera 25, and display unit 26 of the second terminal 20, and the palpation device 30.

[0038] 1.1.3.1 Functional Configuration of First Terminal The following describes the functional configuration of the first terminal 10. The first terminal 10 includes, as functional blocks, a display control unit 111, an input screen creation unit 112, an instruction acquisition unit 113, and an instruction video creation unit 114. The processor 11 of the first terminal 10 functions as the display control unit 111, the input screen creation unit 112, the instruction acquisition unit 113, and the instruction video creation unit 114.

[0039] (Display control unit 111) The display control unit 111 executes photographed image display processing, photographing instruction input screen display processing, photographing instruction display processing, first input screen display processing, first instruction display processing, tracking image display processing, second input screen display processing, second instruction display processing, and palpation result display processing.

[0040] The captured video display process includes a process in which the display control unit 111 receives the captured video VC captured by the camera 25 from the video acquisition unit 211 of the second terminal 20 via the communication interface 13, and a process in which the captured video VC is transmitted to the display unit 16. FIG. 4 is a diagram showing an example of the captured video VC displayed on the display unit 16. In the example of FIG. 4, the captured video VCx is an image in which the patient's entire body is small and visible in the upper left corner of the field of view. The captured video VCy is an image in which the patient is slightly visible in the lower right corner of the field of view. The captured video VCz is an image in which the patient's upper body is visible in the center of the field of view.

[0041] The photographing instruction input screen display process includes a process in which the display control unit 111 receives the photographing instruction input screen SIa from the input screen creation unit 112 and a process in which the display control unit 111 transmits the photographing instruction input screen SIa to the display unit 16. The photographing instruction input screen SIa is a screen for inputting instructions regarding the position of the camera 25. Fig. 5 is a diagram showing an example of the photographing instruction input screen SIa.

[0042] 5, the shooting instruction input screen SIa includes a pointing selection area R1, a movement selection area R2, and a rotation selection area R3. The pointing selection area R1 is an area for selecting a location to which the camera 25 should be moved. The movement selection area R2 is an area for selecting a direction in which the camera 25 should be moved. The rotation selection area R3 is an area for selecting a direction in which the camera 25 should be rotated.

[0043] The photographed video VC is displayed in the pointing selection area R1. The photographed video VC is, for example, the photographed video VCx, VCy, or VCz shown in FIG. 4. The doctor selects, via the user interface 14, a location on the photographed video VC in the pointing selection area R1 to which the doctor wants to move the camera 25 closer. The example in FIG. 5 shows a case where a position slightly to the upper left of the center on the photographed video VC is selected. This allows the doctor to instruct the nurse to move the camera 25 closer to the selected location.

[0044] Four triangular arrows indicating the movement direction (up, down, left, right) are displayed in the movement selection area R2. The doctor selects the triangular arrow corresponding to the direction in which he or she wants to move the camera 25 via the user interface 14. The example in FIG. 5 shows the case where "move right" has been selected. This allows the doctor to instruct the nurse to move the camera 25 in the selected direction.

[0045] Four arrows indicating the rotation direction (up, down, left, right) are displayed in the rotation selection area R3. The doctor selects the arrow corresponding to the direction in which he or she wants to rotate the camera 25 via the user interface 14. The example in FIG. 5 shows the case where "rotate right" has been selected. This allows the doctor to instruct the nurse to rotate the camera 25 around to the right side of the patient.

[0046] The shooting instruction display process includes a process in which the display control unit 111 receives a shooting instruction video VIa relating to the position of the camera 25 from the instruction video creation unit 114, a process in which the display control unit 111 creates a shooting instruction superimposed video VSa by superimposing the shooting instruction video VIa on the shot video VC, and a process in which the display control unit 111 transmits the shooting instruction superimposed video VSa to the display unit 16. FIG. 6 is a diagram showing an example of the shooting instruction superimposed video VSa. In the example of FIG. 6, the shooting instruction superimposed video VSax is a video in which a shooting instruction video VIa (an upward arrow) indicating pointing is superimposed on the shot video VCx shown in FIG. 4. The shooting instruction superimposed video VSay is a video in which a shooting instruction video VIa (two right-facing triangular arrows) indicating a movement to the right is superimposed on the shot video VCy shown in FIG. 4. The shooting instruction superimposed video VSaz is a video in which a shooting instruction video VIa (two clockwise arrows) indicating a right rotation is superimposed on the shot video VCz shown in FIG. 4.

[0047] The shooting instruction superimposed image VSa shown in Fig. 6 is also displayed on the display unit 26 of the second terminal 20. The nurse adjusts the position of the camera 25 while looking at the shooting instruction superimposed image VSa displayed on the display unit 26. Fig. 7 is a diagram showing an example of the shot image VC displayed on the display units 16 and 26 after the position of the camera 25 has been adjusted. In the example of Fig. 7, the shot image VC is an image in which the patient's left arm is captured near the center of the angle of view.

[0048] The first input screen display process includes a process in which the display control unit 111 receives the first input screen SI1 from the input screen creation unit 112 and a process in which the display control unit 111 transmits the first input screen SI1 to the display unit 16. The first input screen SI1 is a screen for inputting instructions regarding the target area for palpation. Fig. 8 is a diagram showing an example of the first input screen SI1.

[0049] As shown in FIG. 8 , the first input screen SI1 includes a target range selection area R4. The target range selection area R4 is an area for selecting the target range for palpation. The captured image VC is displayed in the target range selection area R4. The captured image VC is, for example, the captured image VC shown in FIG. 7 . The doctor selects two locations (start point and end point) on the captured image VC to specify the target range for palpation via the user interface 14. The example of FIG. 8 shows a case where a position near the patient's wrist on the captured image VC is selected as the start point, and a position near the patient's upper arm is selected as the end point. This allows the doctor to instruct the nurse on the target range for palpation.

[0050] The first instruction display process includes a process in which the display control unit 111 receives a first instruction video VI1 that visually indicates the target range for palpation from the instruction video creation unit 114, a process in which the display control unit 111 creates a first superimposed video VS1 by superimposing the first instruction video VI1 on the captured video VC, and a process in which the display control unit 111 transmits the first superimposed video VS1 to the display unit 16. Fig. 9 is a diagram showing an example of the first superimposed video VS1. In the example of Fig. 9, the first superimposed video VS1 is an image in which the first instruction video VI1 (an arrow pointing from the start point to the end point) indicating the target range for palpation is superimposed on the captured video VC.

[0051] The first superimposed image VS1 shown in Fig. 9 is also displayed on the display unit 26 of the second terminal 20. The nurse looks at the first superimposed image VS1 displayed on the display unit 26 and places markers MK on the patient's body parts that correspond to the starting point and ending point of the arrow, respectively. Fig. 10 is a diagram showing an example of the first superimposed image VS1 displayed on the display units 16 and 26 after the two markers MK have been placed. In the example of Fig. 10, the markers MK are placed near the patient's wrist and near the patient's upper arm.

[0052] The tracking video display process includes a process in which the display control unit 111 receives the tracking video VT from the tracking unit 214 of the second terminal 20 via the communication interface 13, a process in which the display control unit 111 creates a tracking superimposed video VSb by superimposing the tracking video VT on the first superimposed video VS1, and a process in which the display control unit 111 transmits the tracking superimposed video VSb to the display unit 16. The tracking video VT is, for example, an image that correctly tracks and highlights the position of the target area for palpation even when the camera 25 moves. FIG. 11 is a diagram showing an example of the tracking superimposed video VSb. In the example of FIG. 11, the tracking superimposed video VSb is an image in which the tracking video VT (bar-shaped highlights) is superimposed on the first superimposed video VS1 shown in FIG. 10.

[0053] The second input screen display process includes a process in which the display control unit 111 receives the second input screen SI2 from the input screen creation unit 112 and a process in which the display control unit 111 transmits the second input screen SI2 to the display unit 16. The second input screen SI2 is a screen for inputting instructions regarding the type and intensity of palpation movements. Fig. 12 is a diagram showing an example of the second input screen SI2.

[0054] 12, the second input screen SI2 includes an image display area R5, an action type selection area R6, and an action intensity selection area R7. The image display area R5 is an area for displaying an image. The action type selection area R6 is an area for selecting the action type of palpation. The action intensity selection area R7 is an area for selecting the action intensity of palpation.

[0055] The tracking superimposed image VSb is displayed in the image display region R5. The tracking superimposed image VSb is, for example, the tracking superimposed image VSb shown in FIG.

[0056] The action type selection area R6 displays two types of buttons and icons indicating the action types of palpation ("pinch" and "press"). The "pinch" icon is an icon that shows pinching with the index finger and thumb. The "press" icon is an icon that shows pressing with the index finger. The example in FIG. 12 shows the case where "press" is selected. The doctor selects "pinch" or "press" via the user interface 14. This allows the doctor to instruct the nurse on the action type of palpation.

[0057] The action strength selection area R7 displays three buttons indicating the action strength of the palpation ("weak," "medium," and "strong"). The example in FIG. 12 shows the case where "strong" is selected. The doctor selects "weak," "medium," or "strong" via the user interface 14. This allows the doctor to instruct the nurse on the action strength of the palpation.

[0058] The second instruction display process includes a process in which the display control unit 111 receives a second instruction video VI2 from the instruction video creation unit 114, which visually instructs the type and intensity of the palpation action, a process in which the display control unit 111 creates a second superimposed video VS2 by superimposing the second instruction video VI2 on the captured video VC, and a process in which the display control unit 111 transmits the second superimposed video VS2 to the display unit 16. FIG. 13 is a diagram showing an example of the second superimposed video VS2. In the example of FIG. 13, the second superimposed video VS2 is an image in which the second instruction video VI2 (two "press" icons and a string of characters "press x strong" indicating a combination of the action type and action intensity) is superimposed on the tracking superimposed video VSb shown in FIG. 11. The two press icons are superimposed on the target area of ​​palpation (highlight video VT). This allows the doctor to intuitively instruct and communicate to the nurse how to touch.

[0059] The palpation result display process includes a process in which the display control unit 111 receives the palpation result RES from the palpation result creation unit 218 of the second terminal 20 via the communication interface 13, and a process in which the display control unit 111 transmits the palpation result RES to the display unit 16.

[0060] (Input screen creation unit 112) The input screen creation unit 112 executes a photographing instruction input screen creation process, a first input screen creation process, and a second input screen creation process. The photographing instruction input screen creation process includes a process in which the input screen creation unit 112 creates a photographing instruction input screen SIa (e.g., FIG. 5) and transmits the photographing instruction input screen SIa to the display control unit 111. The first input screen creation process includes a process in which the input screen creation unit 112 creates a first input screen SI1 (e.g., FIG. 8) and transmits the first input screen SI1 to the display control unit 111. The second input screen creation process includes a process in which the input screen creation unit 112 creates a second input screen SI2 (e.g., FIG. 12) and transmits the second input screen SI2 to the display control unit 111.

[0061] (Instruction Acquisition Unit 113) The instruction acquisition unit 113 executes an imaging instruction acquisition process, a first instruction acquisition process, and a second instruction acquisition process. The imaging instruction acquisition process includes a process in which the instruction acquisition unit 113 acquires imaging instruction input information INa input by the doctor to the imaging instruction input screen SIa via the user interface 14, and a process in which the instruction acquisition unit 113 transmits the imaging instruction input information INa to the instruction video creation unit 114. The imaging instruction input information INa is a pointing position (coordinates), a movement direction, or a rotation direction. The first instruction acquisition process includes a process in which the instruction acquisition unit 113 acquires first input information IN1 input by the doctor to the first input screen SI1 via the user interface 14, and a process in which the instruction acquisition unit 113 transmits the first input information IN1 to the instruction video creation unit 114. The first input information IN1 is information regarding the target range of palpation of the patient (the start point (coordinates) and the end point (coordinates) of the target range of palpation). The second instruction acquisition process includes a process in which the instruction acquisition unit 113 acquires second input information IN2 input by the doctor to the second input screen SI2 via the user interface 14, and a process in which the instruction acquisition unit 113 transmits the second input information IN2 to the instruction video creation unit 114. The second input information IN2 is information regarding the type and intensity of the palpation movement.

[0062] (Instruction Video Creation Unit 114) The instruction video creation unit 114 executes a shooting instruction creation process, a shooting instruction transmission process, a first instruction creation process, a first instruction transmission process, a second instruction creation process, and a second instruction transmission process.

[0063] The shooting instruction creation process includes a process in which the instruction video creation unit 114 receives shooting instruction input information INa from the instruction acquisition unit 113, a process in which the instruction video creation unit 114 creates a shooting instruction video VIa (e.g., Figure 6) based on the shooting instruction input information INa, and a process in which the shooting instruction video VIa is transmitted to the display control unit 111.

[0064] The shooting instruction transmission process includes a process in which the instruction video creation unit 114 transmits the shooting instruction video VIa to the display control unit 212 of the second terminal 20 via the communication interface 13 .

[0065] The first instruction creation process includes a process in which the instruction video creation unit 114 receives first input information IN1 from the instruction acquisition unit 113, a process in which the instruction video creation unit 114 creates a first instruction video VI1 (e.g., Figure 9) based on the first input information IN1, and a process in which the instruction video VI1 is transmitted to the display control unit 111.

[0066] The first instruction transmission process includes a process in which the instruction video creation unit 114 transmits the first instruction video VI1 to the display control unit 212 of the second terminal 20 via the communication interface 13.

[0067] The second instruction creation process includes a process in which the instruction video creation unit 114 receives second input information IN2 from the instruction acquisition unit 113, a process in which the instruction video creation unit 114 creates a second instruction video VI2 (e.g., Figure 13) based on the second input information IN2 and data DTb, and a process in which the instruction video VI2 is transmitted to the display control unit 111.

[0068] The process of creating the second instruction video VI2 will be described below.

[0069] In this embodiment, the target range (distance) of palpation instructed by the doctor is defined as the distance from the start point to the end point (hereinafter also referred to as the "target range L1"). The range (distance) in which the nurse pinches or presses the patient's body part is defined as the "basic range L2" of palpation, and the setting value Vbr for the basic range L2 is stored in advance in the storage 15 as setting data DTb (data including a set of setting values ​​Vbrg for each combination of action type and action intensity). An example of the set of setting values ​​Vbrg for the basic range L2 is shown in FIG. 14. In the example of FIG. 14, when the action type is "pinching" and the action intensity is "weak," the setting value Vbr is 1.5 cm. When the action type is "pinching" and the action intensity is "medium," the setting value Vbr is 2.5 cm. When the action type is "pinching" and the action intensity is "strong," the setting value Vbr is 3.5 cm. When the action type is "pressing" and the action intensity is "weak," the setting value Vbr is 1 cm. When the action type is "press" and the action strength is "medium", the set value Vbr is 2 cm. When the action type is "press" and the action strength is "strong", the set value Vbr is 3 cm.

[0070] The instruction video creation unit 114 determines the number of palpation movements C by the nurse in the target range L1 depending on how many basic ranges L2 fit within the target range L1 (based on the number of basic ranges L2 included in the target range L1). For example, the number of movements C can be calculated using the target range L1 and the basic range L2 according to the following formula (1): C = floor(L1 / L2) (1)

[0071] When the number of actions C is 1, the instruction video creation unit 114 creates a second instruction video VI2 instructing the nurse to "pinch" or "press" once within the target range L1. An example of the second superimposed video VS2 in this case is shown in FIG. 15 . The example in FIG. 15 shows a case where the action type is "press" and the action intensity is "strong." This allows the doctor to instruct the nurse to "press" once within the target range L1 with "strong" intensity. When the number of actions C is 2, the instruction video creation unit 114 creates a second instruction video VI2 instructing the nurse to "pinch" or "press" twice within the target range L1. When the number of actions C is 3, the instruction video creation unit 114 creates a second instruction video VI2 instructing the nurse to "pinch" or "press" three times within the target range L1. An example of the second superimposed video VS2 in this case is shown in FIG. 16 . The example in FIG. 16 shows a case where the action type is "press" and the action intensity is "strong." This allows the doctor to instruct the nurse to "press" three times at "strong" intensity within the target range L1. The same applies when the number of actions C is four or more.

[0072] When the number of actions C is 2 or more, the instruction video creation unit 114 creates a second instruction video VI2 in which the position of the "pinch" icon or the "press" icon shifts in the order in which the "pinch" action or the "press" action is performed. An example of the second superimposed video VS2 in this case is shown in FIG. 17. The example in FIG. 17 shows a case in which the number of actions C is 3, the action type is "press," and the action intensity is "strong." This allows the doctor to instruct the nurse in a step-by-step order to perform three "presses" at the "strong" intensity within the target range L1. Furthermore, the nurse can visually understand the order in which the palpation will be performed within the target range L1.

[0073] The instruction video creation unit 114 can also create a second instruction video VI2 in which the motion speed of the fingers "pinching" is changed depending on the motion intensity. An example of the second instruction video VI2 in this case is shown in FIG. 18 . In the example of FIG. 18 , a "pinching" icon indicating the state of the fingers before pinching and a "pinching" icon indicating the state of the fingers after pinching are displayed, showing the state of the fingers "pinching." For example, these icons are displayed in the order of left icon and right icon, and by changing the time interval between the display of the left icon and the display of the right icon, a second instruction video VI2 can be created that expresses different motion speeds of the fingers "pinching." When the motion intensity is "strong," the motion speed of the fingers "pinching" can be made faster, and when the motion intensity is "weak," the motion speed of the fingers "pinching" can be made slower. This allows the nurse to visually understand the motion intensity of the "pinching" motion.

[0074] The instruction video creation unit 114 can also create a second instruction video VI2 in which the finger movement speed of "pressing" is changed depending on the motion strength. An example of the second instruction video VI2 in this case is shown in FIG. 19 . In the example of FIG. 19 , three "press" icons are displayed, showing the ripples spreading out in sequence when pressed. For example, these icons are displayed in the order of the left icon, the center icon, and the right icon. By changing the time interval between the display of the left icon and the center icon and the time interval between the display of the center icon and the right icon, a second instruction video VI2 can be created that expresses different finger movement speeds for "pressing." When the motion strength is "strong," the ripples can spread faster, and when the motion strength is "weak," the ripples can spread slower. This allows the nurse to visually understand the motion strength of the "pressing" motion.

[0075] The second instruction transmission process includes a process in which the instruction video creation unit 114 transmits the second instruction video VI2 to the display control unit 212 of the second terminal 20 via the communication interface 13.

[0076] 1.1.3.2 Functional Configuration of Second Terminal The functional configuration of the second terminal 20 will be described. As shown in Fig. 3 , the second terminal 20 includes, as functional blocks, an image acquisition unit 211, a display control unit 212, a marker position acquisition unit 213, a tracking unit 214, an intensity correction unit 215, a measurement data acquisition unit 216, a hardness calculation unit 217, and a palpation result creation unit 218. The processor 21 of the second terminal 20 functions as the image acquisition unit 211, the display control unit 212, the marker position acquisition unit 213, the tracking unit 214, the intensity correction unit 215, the measurement data acquisition unit 216, the hardness calculation unit 217, and the palpation result creation unit 218.

[0077] (Video Acquisition Unit 211) The video acquisition unit 211 executes a captured video acquisition process. The captured video acquisition process includes a process in which the video acquisition unit 211 acquires the captured video VC from the camera 25, a process in which the video acquisition unit 211 transmits the captured video VC to the display control unit 212, and a process in which the video acquisition unit 211 transmits the captured video VC to the display control unit 111 of the first terminal 10 via the communication interface 23.

[0078] (Display Control Unit 212) The display control unit 212 executes a captured image display process, a capturing instruction display process, a first instruction display process, a tracking image display process, and a second instruction display process.

[0079] The captured video display process includes a process in which the display control unit 212 receives the captured video VC from the video acquisition unit 211 and a process in which the display control unit 212 transmits the captured video VC to the display unit 26. The captured video VC displayed on the display unit 26 is, for example, the captured video VC shown in FIG.

[0080] The shooting instruction display process includes a process in which the display control unit 212 receives the shooting instruction video VIa from the instruction video creation unit 114 of the first terminal 10 via the communication interface 23, a process in which the display control unit 212 creates a shooting instruction superimposed video VSa by superimposing the shooting instruction video VIa on the shooting video VC, and a process in which the display control unit 212 transmits the shooting instruction superimposed video VSa to the display unit 26. The shooting instruction superimposed video VSa displayed on the display unit 26 is, for example, the shooting instruction superimposed video VSa shown in FIG.

[0081] The first instruction display process includes a process in which the display control unit 212 receives a first instruction video VI1 from the instruction video creation unit 114 of the first terminal 10 via the communication interface 23, a process in which the display control unit 212 creates a first superimposed video VS1 by superimposing the first instruction video VI1 on the captured video VC, and a process in which the display control unit 212 transmits the first superimposed video VS1 to the marker position acquisition unit 213 and the display unit 26. The first superimposed video VS1 displayed on the display unit 26 is, for example, the first superimposed video VS1 shown in FIG.

[0082] The tracking video display process includes a process in which the display control unit 212 receives the tracking video VT created by the tracking unit 214 from the tracking unit 214, a process in which the display control unit 212 creates a tracking superimposed video VSb by superimposing the tracking video VT on the first superimposed video VS1, and a process in which the display control unit 212 transmits the tracking superimposed video VSb to the display unit 26. The tracking video VT displayed on the display unit 26 is, for example, the tracking video VT shown in FIG.

[0083] The second instruction display process includes a process in which the display control unit 212 receives a second instruction video VI2 from the instruction video creation unit 114 of the first terminal 10 via the communication interface 23, and a process in which the display control unit 212 transmits the second instruction video VI2 to the intensity correction unit 115. The second instruction display process also includes a process in which the display control unit 212 receives a third instruction video VI3 created by the intensity correction unit 115 from the intensity correction unit 115, a process in which the display control unit 212 creates a third superimposed video VS3 by superimposing the third instruction video VI3 on ​​the tracking superimposed video VSb (by superimposing the first instruction video VI1, the tracking video VT, and the third instruction video VI3 on ​​the captured video VC), and a process in which the display control unit 212 transmits the third superimposed video VS3 to the display unit 26 (to display the third superimposed video VS3).

[0084] (Marker position acquisition unit 213) The marker position acquisition unit 213 executes a marker position acquisition process. The marker position acquisition process includes a process in which the marker position acquisition unit 213 receives the first superimposed image VS1 from the display control unit 212, a process in which the marker position acquisition unit 213 acquires position data DTm of the marker MK placed on the body part of the patient by the nurse based on the first superimposed image VS1, and a process in which the marker position acquisition unit 213 transmits the first superimposed image VS1 and the position data DTm to the tracking unit 214.

[0085] (Tracking Unit 214) The tracking unit 214 executes AR tracking processing. The AR tracking processing includes processing by the tracking unit 214 to create a tracking video VT by AR tracking based on the first superimposed video VS1 and the position data DTm of the marker MK, processing to transmit the tracking video VT to the display control unit 212, and processing to transmit the tracking video VT to the display control unit 111 of the first terminal 10 via the communication interface 23.

[0086] (Intensity Correction Unit 215) The intensity correction unit 215 executes an intensity correction process. The intensity correction process includes the following processes: receiving the second instruction video VI2 from the display control unit 212; acquiring the data DTp from the storage 24; creating a third instruction video VI3 by correcting the action intensity instructed by the second instruction video VI2 based on the data DTp; and transmitting the third instruction video VI3 to the display control unit 212.

[0087] The process of creating the third instruction video VI3 will be described below.

[0088] In the "picking" or "pressing" motion, doctors and nurses have individual differences in their perception of "weak, medium, or strong" pinching or pressing forces. Therefore, in this embodiment, the pinching and pressing forces of each doctor and nurse are measured in advance when "weak, medium, or strong" is intended, and the measured values ​​(setting values ​​Pdoc and Pnur) are stored in advance in storage 24 as setting data DTp (data including a set of doctor force setting values ​​Pdocg for each combination of motion type and motion intensity, and a set of nurse force setting values ​​Pnurg for each combination of motion type and motion intensity). An example of the setting value sets Pdocg and Pnurg, along with the correction of the second instruction image VI2 based on these setting value sets, is shown in FIG. 20 .

[0089] In the example of FIG. 20 , when the action type is “pinch” and the action intensity is “weak”, the set value Pdoc is 3N. When the action type is “pinch” and the action intensity is “medium”, the set value Pdoc is 5N. When the action type is “pinch” and the action intensity is “strong”, the set value Pdoc is 7N. When the action type is “press” and the action intensity is “weak”, the set value Pdoc is 2N. When the action type is “press” and the action intensity is “medium”, the set value Pdoc is 4N. When the action type is “press” and the action intensity is “strong”, the set value Pdoc is 6N.

[0090] Furthermore, when the action type is "pinch" and the action intensity is "weak", the set value Pnur is 4N. When the action type is "pinch" and the action intensity is "medium", the set value Pnur is 6N. When the action type is "pinch" and the action intensity is "strong", the set value Pnur is 8N. When the action type is "press" and the action intensity is "weak", the set value Pnur is 4N. When the action type is "press" and the action intensity is "medium", the set value Pnur is 6N. When the action type is "press" and the action intensity is "strong", the set value Pnur is 7N.

[0091] For example, as shown in FIG. 13 , if the doctor's instruction is "press hard," the doctor intends to instruct the nurse to press with a force of 6 N. As shown in FIG. 20 , the setting value Pdoc (=6 N) corresponding to the combination of "press" and "strong" in the setting value group Pdocg is equal to the setting value Pnur (=6 N) corresponding to the combination of "press" and "medium" in the setting value group Pnurg. Therefore, based on the data DTp shown in FIG. 20 , the intensity correction unit 215 creates a third instruction video VI3 by correcting the "press hard" in the second instruction video VI2 to "press medium." This allows for the nurse to receive instructions that accommodate individual differences in the sense of motion intensity between doctors and nurses.

[0092] The third instruction video VI3 created in this manner is transmitted from the intensity correction unit 215 to the display control unit 212, and the display control unit 212 creates the third superimposed video VS3 as described above and transmits the created third superimposed video VS3 to the display unit 26. The third superimposed video VS3 displayed on the display unit 26 is, for example, the third superimposed video VS3 shown in FIG.

[0093] The nurse views the third superimposed image VS3 displayed on the display unit 26, and wears the palpation device 30 to perform palpation, as shown in Fig. 22 . Fig. 22 is a diagram showing how palpation is performed. In the example of Fig. 22 , the palpation device 30 is a pressure sensor. For example, the pressure sensor measures the pressure of the fingertip based on the strain of the nail. The pressure sensor also measures the hardness of the patient's body part (the part to be palpated) based on changes in the pressure value applied to the fingertip.

[0094] (Measurement data acquisition unit 216) The measurement data acquisition unit 216 executes a measurement data acquisition process. The measurement data acquisition process includes a process in which the measurement data acquisition unit 216 acquires, from the palpation device 30, measurement data DT of the palpation performed by the nurse using the palpation device 30 based on the third superimposed image VS3, and a process in which the measurement data DT is transmitted to the hardness calculation unit 217.

[0095] (Hardness Calculation Unit 217) The hardness calculation unit 217 executes a hardness calculation process. The hardness calculation process includes a process in which the hardness calculation unit 217 receives the measurement data DT from the measurement data acquisition unit 216, a process in which the hardness calculation unit 217 calculates the hardness based on the measurement data DT, and a process in which the hardness calculation unit 217 transmits the measurement data DT and the hardness data DTd to the palpation result creation unit 218.

[0096] The process of calculating the hardness will be described below.

[0097] FIG. 23 is a diagram illustrating the apparent hardness (stiffness) of a patient's body part during a "pressing" action. In FIG. 23, P is the force (force applied by the nurse during palpation) pressing into the patient's body part. H is the actual hardness of the patient's body part (palpation area). D is the apparent hardness of the patient's body part. The measurement data DT acquired by the pressure sensor includes the pressing force P and the actual hardness H.

[0098] The hardness calculation unit 217 calculates the apparent hardness D based on the pressing force P and the actual hardness H. For example, the apparent hardness D can be calculated using the pressing force P and the actual hardness H according to the following formula (2): D = P - H (2)

[0099] Similarly, for the "pinching" action, the "pushing force P" can be read as the "pinching force P (the force exerted by the nurse during palpation)" of the patient's body part, and the apparent hardness D can be calculated using the above formula (2).

[0100] The apparent hardness D calculated in this manner is transmitted as hardness data DTd from the hardness calculation unit 217 to the palpation result creation unit 218.

[0101] (Palpation result creation unit 218) The palpation result creation unit 218 executes a palpation result creation process. The palpation result creation process includes a process in which the palpation result creation unit 218 receives the measurement data DT and hardness data DTd from the hardness calculation unit 217, a process in which the palpation result creation unit 218 creates a palpation result RES based on the measurement data DT and the hardness data DTd, and a process in which the palpation result RES is transmitted to the display control unit 111 of the first terminal 10 via the communication interface 23.

[0102] Fig. 24 is a diagram showing an example of the palpation result RES created by the palpation result creation unit 218. As shown in Fig. 24, the palpation result RES includes a palpation result for the first region and a palpation result for the second region. Each palpation result is, for example, data in which the apparent hardness D is visualized using a graph or the like.

[0103] The palpation result RES created in this manner is transmitted from the palpation result creation unit 218 to the display control unit 111 via the communication interface 23, and the display control unit 111 transmits the palpation result RES to the display unit 16. The palpation result RES displayed on the display unit 16 is, for example, the palpation result RES shown in Fig. 24. This allows the palpation result to be reproduced at a remote location.

[0104] The doctor makes a diagnosis by looking at the palpation results RES displayed on the display unit 16.

[0105] 1.2 Operation A description will be given of the operation of the information presentation system 1. Figures 25 to 28 are flowcharts showing an example of the operation of the information presentation system 1.

[0106] First, a doctor starts a video call using the first terminal 10, and a nurse starts a video call using the second terminal 20. After the video call starts, the nurse uses the second terminal 20 to photograph the patient.

[0107] 25, the video acquisition unit 211 of the second terminal 20 executes a captured video acquisition process (S201), whereby the captured video VC is transmitted to the display control unit 212 and the display control unit 111.

[0108] Next, the display control unit 212 of the second terminal 20 executes a captured image display process (S202). As a result, the captured image VC of the patient is displayed on the display unit 26. The display control unit 111 of the first terminal 10 executes a captured image display process (S101). As a result, the image of the patient is displayed on the display unit 16.

[0109] Next, the input screen creation unit 112 of the first terminal 10 executes a photographing instruction input screen creation process (S102), whereby a photographing instruction input screen SIa is sent to the display control unit 111.

[0110] Next, the display control unit 111 executes a photographing instruction input screen display process (S103), whereby the photographing instruction input screen SIa is displayed on the display unit 16.

[0111] When the imaging instruction input screen SIa is displayed, the doctor inputs instructions regarding the position of the camera 25 on the imaging instruction input screen SIa.

[0112] Next, the instruction acquisition unit 113 of the first terminal 10 executes a shooting instruction acquisition process (S104), whereby shooting instruction input information INa is transmitted to the instruction video creation unit 114.

[0113] Next, the instruction video creation unit 114 of the first terminal 10 executes a shooting instruction creation process (S105), whereby a shooting instruction video VIa is transmitted to the display control unit 111.

[0114] Next, the display control unit 111 executes a shooting instruction display process (S106), whereby a shooting instruction superimposed image VSa in which the shooting instruction image VIa is superimposed on the shot image VC is displayed on the display unit 16.

[0115] When the imaging instruction superimposed image VSa is displayed, the doctor looks at the imaging instruction superimposed image VSa and checks the content of the instructions regarding the position of the camera 25. If the doctor wants to change the instructions, he or she inputs them again. If the doctor does not need to change the instructions, he or she confirms the instructions.

[0116] When the instruction is confirmed, the instruction video creation unit 114 executes a shooting instruction transmission process (S107), whereby a shooting instruction video VIa is transmitted to the display control unit 212 of the second terminal 20 via the communication interface 13.

[0117] Next, the display control unit 212 executes a shooting instruction display process (S203), whereby a shooting superimposed video VSa in which the shooting instruction video VIa is superimposed on the shooting video VC is displayed on the display unit 26.

[0118] When the photographed superimposed image VSa is displayed, the nurse adjusts the position of the camera 25. As a result, the adjusted photographed image VC of the patient is displayed on the display unit 26 and the display unit 16.

[0119] 26, the input screen creating unit 112 executes a first input screen creating process (S108), whereby the first input screen SI1 is sent to the display control unit 111.

[0120] Next, the display control unit 111 executes a first input screen display process (S109), whereby the first input screen SI1 is displayed on the display unit 16.

[0121] When the first input screen SI1 is displayed, the doctor inputs instructions regarding the range to be palpated on the first input screen SI1.

[0122] Next, the instruction acquisition unit 113 executes a first instruction acquisition process (S110), which transmits first input information IN1 to the instruction video creation unit 114.

[0123] Next, the instruction video creation unit 114 executes a first instruction creation process (S111), whereby a first instruction video VI1 is sent to the display control unit 111.

[0124] Next, the display control unit 111 executes a first instruction display process (S112), whereby the display unit 16 displays a first superimposed video VS1 in which a first instruction video VI1 is superimposed on the captured video VC.

[0125] When the first superimposed image VS1 is displayed, the doctor looks at the first superimposed image VS1 and checks the content of the instructions regarding the range to be palpated. If the doctor wants to change the instructions, he or she inputs them again. If the doctor does not need to change the instructions, he or she confirms the instructions.

[0126] When the instruction is confirmed, the instruction video creation unit 114 executes a first instruction transmission process (S113), which transmits a first instruction video VI1 to the display control unit 212 of the second terminal 20 via the communication interface 13.

[0127] Next, the display control unit 212 executes a first instruction display process (S204), whereby a first superimposed video VS1 in which a first instruction video VI1 is superimposed on the captured video VC is displayed on the display unit 26.

[0128] When the first superimposed image VS1 is displayed, the nurse looks at the first superimposed image VS1 and places markers MK on the patient's body parts that correspond to the starting point and ending point of the arrow of the palpation target area.

[0129] Once the marker MK is placed, the marker position acquisition unit 213 of the second terminal 20 executes a marker position acquisition process (S205), which transmits the first superimposed image VS1 and the position data DTm to the tracking unit 214.

[0130] Next, the tracking unit 214 of the second terminal 20 executes the AR tracking process (S206), whereby the tracking video VT is transmitted to the display control unit 212 and the display control unit 111.

[0131] Next, the display control unit 212 executes a tracking video display process (S207). As a result, a tracking superimposed video VSb in which the tracking video VT is superimposed on the first superimposed video VS1 is displayed on the display unit 26. The display control unit 111 executes a tracking video display process (S114). As a result, a tracking superimposed video VSb in which the tracking video VT is superimposed on the first superimposed video VS1 is displayed on the display unit 16.

[0132] 27, the input screen creating unit 112 executes a second input screen creating process (S115), whereby the second input screen SI2 is sent to the display control unit 111.

[0133] Next, the display control unit 111 executes a second input screen display process (S116), whereby the second input screen SI2 is displayed on the display unit 16.

[0134] When the second input screen SI2 is displayed, the doctor inputs instructions regarding the type and strength of the palpation action on the second input screen SI2.

[0135] Next, the instruction acquisition unit 113 executes a second instruction acquisition process (S117), whereby second input information IN2 is sent to the instruction video creation unit 114.

[0136] Next, the instruction video creation unit 114 executes a second instruction creation process (S118), whereby a second instruction video VI2 is sent to the display control unit 111.

[0137] Next, the display control unit 111 executes a second instruction display process (S119), whereby the display unit 16 displays a second superimposed video VS2 in which the second instruction video VI2 is superimposed on the captured video VC.

[0138] When the second superimposed image VS2 is displayed, the doctor looks at the second superimposed image VS2 and checks the instructions regarding the type and intensity of the palpation motion. If the doctor wants to change the instructions, he or she inputs them again. If the doctor does not need to change the instructions, he or she confirms the instructions.

[0139] When the instruction is confirmed, the instruction video creation unit 114 executes a second instruction transmission process (S120), which transmits a second instruction video VI2 to the display control unit 212 of the second terminal 20 via the communication interface 13.

[0140] Next, the display control unit 212 executes a second instruction display process (S208). At this time, the intensity correction unit 215 of the second terminal 20 executes the intensity correction process to create a third instruction video VI3 in which the action intensity instructed by the second instruction video VI2 has been corrected. The display control unit 212 then creates a third superimposed video VS3 in which the third instruction video VI3 is superimposed on the captured video VC. As a result, the third superimposed video VS3 is displayed on the display unit 26.

[0141] When the third superimposed image VS3 is displayed, the nurse wears the palpation device 30, looks at the third superimposed image VS3, and performs palpation on the patient.

[0142] 28, the measurement data acquisition unit 216 of the second terminal 20 executes a measurement data acquisition process (S209), whereby the measurement data DT is transmitted to the hardness calculation unit 217.

[0143] Next, the hardness calculation unit 217 of the second terminal 20 executes the hardness calculation process (S210), which sends the measurement data DT and hardness data DTd to the palpation result creation unit 218.

[0144] Next, the palpation result creation unit 218 of the second terminal 20 executes the palpation result creation process (S211), whereby the palpation result RES is sent to the display control unit 111 of the first terminal 10 via the communication interface 23.

[0145] Next, the display control unit 111 executes a palpation result display process (S121), whereby the palpation result RES is displayed on the display unit 16.

[0146] When the palpation result is displayed, the doctor looks at the palpation result RES and makes a diagnosis.

[0147] 1.3 Effects of the Present Embodiment In the information presentation system 1 according to the present embodiment, the first terminal 10 acquires first input information IN1 regarding the target area of ​​palpation of the patient and second input information IN2 regarding the type and intensity of palpation motion, both of which are input by the doctor. Based on the first input information IN1, the first terminal 10 creates a first instruction video VI1 that visually indicates the target area of ​​palpation. Based on the second input information IN2, the first terminal 10 creates a second instruction video VI2 that visually indicates the type and intensity of palpation motion. The first terminal 10 transmits the first instruction video VI1 and the second instruction video VI2 to the second terminal 20.

[0148] The second terminal 20 creates a third instruction video VI3 by correcting the action intensity instructed by the second instruction video VI2. The second terminal 20 creates a third superimposed video VS3 by superimposing the first instruction video VI1 and the third instruction video VI3 on ​​the captured video VC captured by the camera 25. The second terminal 20 displays the third superimposed video VS3 on the display unit 26.

[0149] This allows the doctor to use visual representations to give instructions via a video call between the first terminal 10 and the second terminal 20 regarding the target part (target area) of the patient's body that the doctor wants the nurse to palpate and how to touch it (type of movement and movement intensity), intuitively and appropriately conveying nuances that are difficult to convey verbally.The nurse can also intuitively and appropriately understand the instructions from the doctor.

[0150] Based on instructions from the doctor, the nurse performs palpation using the palpation device 30 while viewing the third superimposed image VS3, and then the second terminal 20 acquires measurement data DT of the palpation from the palpation device 30. The second terminal 20 creates a palpation result RES based on the measurement data DT. The second terminal 20 transmits the palpation result RES to the first terminal 10. This allows the doctor to understand the palpation result RES.

[0151] Therefore, according to this embodiment, a doctor in a remote location can appropriately instruct a local nurse to touch a patient's body part, and the doctor can understand the results of the palpation of any part performed by the nurse.

[0152] The second terminal 20 calculates the apparent hardness D of the palpated area based on the force P applied by the nurse during palpation and the actual hardness H of the palpated area of ​​the patient, which are included in the measurement data DT. This makes it possible to obtain a more appropriate hardness D.

[0153] The palpation result RES created by the second terminal 20 is hardness data DTd that visualizes the apparent hardness D. This allows the doctor to visually understand the palpation result RES.

[0154] The second terminal 20 determines the number of palpation movements C by the nurse in the target range L1 based on the number of basic palpation ranges L2 included in the target range L1 of palpation. This allows the doctor to appropriately inform the nurse of the number of palpation movements C in the target range L1.

[0155] When the palpation device 30 is an AR glass, the nurse can receive instructions from the doctor hands-free.

[0156] 2. Modifications, etc. The information presentation system according to the embodiment can be modified as follows, for example.

[0157] 2.1 First Modification An information presentation system according to a first modification of the embodiment will be described. The information presentation system 1A according to the first modification of the embodiment differs from the embodiment in that AR tracking processing is performed without using a marker MK. The following description will focus on the differences from the embodiment.

[0158] 2.1.1 Hardware Configuration of Information Presentation System The hardware configuration of the information presentation system 1A is the same as the hardware configuration shown in FIG. 2 in the embodiment.

[0159] 2.1.2 Functional Configuration of Information Presentation System The functional configuration of the information presentation system 1A will be described with reference to Fig. 29. Fig. 29 is a block diagram showing an example of the functional configuration of the information presentation system 1A.

[0160] As shown in Fig. 29, in this modification, the functional configuration of the first terminal 10 is similar to the functional configuration of Fig. 3 shown in the embodiment. In the second terminal 20, the marker position acquisition unit 213 is eliminated. In addition, the functional configuration other than the display control unit 212A and the tracking unit 214A is similar to the functional configuration of Fig. 3 shown in the embodiment.

[0161] (Display Control Unit 212A) The display control unit 212A is the same as that in the embodiment except that it does not transmit the first superimposed video VS1 to the marker position acquisition unit 213.

[0162] (Tracking Unit 214A) The tracking unit 214A executes AR tracking processing. Fig. 30 is a diagram showing an example of the AR tracking processing. In the AR tracking processing by the tracking unit 214A, the processing for creating the tracking video VT differs from that in the embodiment.

[0163] As shown in Fig. 30 , in the process of creating the tracking video VT, first, feature points (feature amounts) of the patient's body parts are acquired by performing bone structure estimation and the like based on the first superimposed video VS1. In the example of Fig. 30 , the points indicated by circles are feature points. Next, a specified range is tracked based on the acquired feature points. Next, the tracked specified range is highlighted. In this way, the tracking video VT is created.

[0164] 2.1.3 Operation The operation of the information presentation system 1A will be described. A flowchart showing an example of the operation of the information presentation system 1A is obtained by replacing FIG. 26, one of FIGS. 25 to 28 shown in the embodiment, with FIG. 31. FIG. 31 is a flowchart showing an example of the operation of the information presentation system 1A according to this modification. In FIG. 31, step S205 of the flowchart in FIG. 26 shown in the embodiment is omitted, and step S206 is replaced with step S212. In step S212, AR tracking processing is performed by the tracking unit 214A.

[0165] 2.1.4 Effects of this Modification This modification provides the same effects as the embodiment. Furthermore, this modification allows AR tracking processing to be performed without using markers MK.

[0166] 2.2 Second Modification An information presentation system according to a second modification of the embodiment will be described. The information presentation system 1B according to the second modification of the embodiment differs from the embodiment in that palpation is performed using a glove-type device. The following description will focus on the differences from the embodiment.

[0167] 2.2.1 Hardware Configuration of Information Presentation System The hardware configuration of the information presentation system 1B is the same as the hardware configuration shown in FIG. 2 in the embodiment.

[0168] 2.2.2 Functional Configuration of Information Presentation System The functional configuration of the information presentation system 1B will be described with reference to Fig. 32. Fig. 32 is a block diagram showing an example of the functional configuration of the information presentation system 1B.

[0169] As shown in Fig. 32, in this modification, the functional configuration of the first terminal 10 is the same as the functional configuration of Fig. 3 shown in the embodiment. In the second terminal 20, the hardness calculation unit 217 is eliminated. In addition, the functional configuration other than the measurement data acquisition unit 216B and the palpation result creation unit 218B is the same as the functional configuration of Fig. 3 shown in the embodiment.

[0170] The nurse looks at the third superimposed image VS3 displayed on the display unit 26, and wears the palpation device 30 to perform palpation, as shown in Fig. 33. Fig. 33 is a diagram showing how palpation is performed. In the example of Fig. 33, the palpation device 30 is a glove-type device (tactile presentation device) that expresses hardness.

[0171] (Measurement Data Acquisition Unit 216B) The measurement data acquisition unit 216B is the same as the embodiment except that it does not transmit the measurement data DT to the hardness calculation unit 217, but transmits the measurement data DT to the palpation result creation unit 218. The measurement data DT acquired by the glove-type device includes data DTx that expresses the apparent hardness D.

[0172] (Palpation result creation unit 218B) The palpation result creation unit 218B executes a palpation result creation process. In the palpation result creation process by the palpation result creation unit 218B, the palpation result creation unit 218B receives the measurement data DT from the measurement data acquisition unit 216B, and transmits the data DTx included in the measurement data DT as a palpation result RES to the display control unit 111 of the first terminal 10 via the communication interface 23.

[0173] 2.2.3 Effects of this Modification This modification has the same effects as the embodiment. Furthermore, by using the glove-type device, this modification makes it possible to directly present the apparent hardness D to a doctor.

[0174] 2.3 Others The above-described embodiment and modified examples may be combined to the extent possible. For example, the second modified example of the embodiment may be combined with the first modified example of the embodiment as a modified example.

[0175] Furthermore, in the flowcharts described in the above embodiment and modified examples, the order of the processes can be changed as much as possible.

[0176] It should be noted that the present invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in appropriate combinations, in which case the combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by selecting and combining the multiple disclosed constituent elements.

[0177] DESCRIPTION OF SYMBOLS 1, 1A, 1B... Information presentation system 10... First terminal 11... Processor 12... Memory 13... Communication interface 14... User interface 15... Storage 16... Display unit 20, 20A, 20B... Second terminal 21... Processor 22... Memory 23... Communication interface 24... Storage 25... Camera 26... Display unit 30... Palpation device 111... Display control unit 112... Input screen creation unit 113... Instruction acquisition unit 114... Instruction image creation unit 211... Image acquisition unit 212, 212A... Display control unit 213... Marker position acquisition unit 214, 214A... Tracking unit 215... Intensity correction unit 216, 216B... Measurement data acquisition unit 217... Hardness calculation unit 218, 218B... Palpation result creation unit

Claims

1. A palpation device comprising: a first terminal; a second terminal including a camera and capable of wireless communication with the first terminal; and a palpation device capable of communication with the second terminal, wherein the first terminal comprises: an instruction acquisition unit that acquires first input information input by a first user regarding a target range of palpation on a subject and second input information regarding a type and intensity of the palpation action; an instruction image creation unit that creates a first instruction image that visually indicates the target range based on the first input information, creates a second instruction image that visually indicates the type and intensity of the action based on the second input information, and transmits the first instruction image and the second instruction image to the second terminal, wherein the second terminal comprises: an intensity correction unit that creates a third instruction image by correcting the intensity of the action indicated by the second instruction image; and a display control unit that creates a superimposed image by superimposing the first instruction image and the third instruction image on an image captured by the camera, and displays the superimposed image. An information presentation system comprising: a measurement data acquisition unit that acquires measurement data of the palpation performed by a second user using the palpation device based on the superimposed image from the palpation device; and a palpation result creation unit that creates a palpation result based on the measurement data and transmits the palpation result to the first terminal.

2. The information presentation system of claim 1, wherein the strength correction unit creates the third instruction image based on a first set of force values ​​of the first user for each combination of the action type and the action intensity, and a second set of force values ​​of the second user for each combination of the action type and the action intensity.

3. The information presentation system of claim 2, wherein the second instruction image includes a first action type and a first action intensity, the third instruction image includes the first action type and a second action intensity, and a first setting value corresponding to the combination of the first action type and the first action intensity in the first setting value group is equal to a second setting value corresponding to the combination of the first action type and the second action intensity in the second setting value group.

4. The information presentation system of claim 1, wherein the second terminal further includes a hardness calculation unit that calculates an apparent second hardness of the palpation area based on the force of the second user at the time of palpation contained in the measurement data and the actual first hardness of the palpation area of ​​the subject.

5. The information presentation system according to claim 4, wherein the palpation result is hardness data that visualizes the second hardness.

6. An information presentation system as described in claim 1, wherein the instruction image creation unit determines the number of times the second user performs the palpation action in the target range depending on how many basic palpation ranges fit within the target range.

7. An information presentation method using an information presentation system including a first terminal, a second terminal including a camera and capable of wirelessly communicating with the first terminal, and a palpation device capable of communicating with the second terminal, the method comprising: the first terminal acquiring first input information input by a first user regarding a target range of palpation on a subject, and second input information regarding a type and intensity of the palpation action; the first terminal creating a first instruction video that visually indicates the target range based on the first input information, creating a second instruction video that visually indicates the type and intensity of the action based on the second input information, and transmitting the first instruction video and the second instruction video to the second terminal; the second terminal creating a third instruction video in which the intensity of the action indicated by the second instruction video is corrected; the second terminal creating a superimposed video in which the first instruction video and the third instruction video are superimposed on an image captured by the camera, and displaying the superimposed video; The second terminal acquires, from the palpation device, measurement data of the palpation performed by the second user using the palpation device based on the superimposed image; and the second terminal creates a palpation result based on the measurement data and transmits the palpation result to the first terminal.

8. A program that causes a first processor provided in the first terminal and a second processor provided in the second terminal to respectively execute processing by the first terminal and the second terminal of the information presentation system described in any one of claims 1 to 6.

Images