Information presentation method, information presentation system, and program

Abstract

This information presentation system is provided with a first terminal and a second terminal. The first terminal is located on site and is provided with: a video information acquisition unit which acquires video information of the site; and a video information transmission unit which transmits the video information to the second terminal. The second terminal is located remotely from the site and is provided with: a video information reception unit that receives the transmitted video information; a video information display control unit that causes the received video information to be displayed on a display screen of the second terminal; an instruction input unit that receives an instruction for the displayed video information; an instruction information creation unit that creates instruction information by superimposing visual information based on the instruction on the displayed video information; and an instruction information transmission unit that transmits the created instruction information to the first terminal. The first terminal further comprises: an instruction information reception unit that receives the transmitted instruction information; and an instruction information display control unit that causes the received instruction information to be displayed on the display screen of the first terminal.

Description

Information presentation method, information presentation system, and program

[0001] The present invention relates to an information presentation method, an information presentation system, and a program, and in particular to an information presentation method, an information presentation system, and a program for efficiently conveying instructions, including ambiguous nuances of actions that take into account the implicit intentions of an instructor, such as a nurse, to on-site workers and patients in various remote nursing tasks.

[0002] In recent years, against the backdrop of the rapid aging of society as a whole, the medical and nursing fields are facing a shortage of nursing personnel due to an increase in the number of patients and a decrease in the working population.

[0003] Therefore, in a super-aging society, it is important to provide comprehensive care that allows patients to continue living in their familiar neighborhoods and homes as much as possible, rather than immediately being admitted to a hospital when they become dependent on care.

[0004] To provide this type of care, it is necessary to enable various nursing tasks that have previously been performed face-to-face in hospitals to be performed remotely and indirectly while the patient is at home.

[0005] This type of initiative, i.e., remote indirect nursing care, is currently being trialed in some cases. For example, one trial is being conducted in which a patient recuperating at home is loaned a tablet and measuring devices such as a blood pressure monitor, and the patient enters their physical condition and various measurements into the tablet. Based on this information, the nurse can communicate with the patient via a video conferencing system and understand their condition.

[0006] However, the above-mentioned attempt currently only covers "part of the observation of the patient's condition" and is mainly limited to the palm area, and has the following limitations:

[0007] The first limitation is that much of the information that can be obtained from patients is text-based information corresponding to predetermined items, and image- or video-based information is limited to facial images during conversations using a video conferencing system.

[0008] The second limitation is that if any treatment is required as a result of the observation, the patient will need to visit a hospital.

[0009] A third limitation is that one-to-one communication between patients and nurses is assumed.

[0010] For these reasons, currently, only observation of the patient's condition can be performed based on limited information. Therefore, in order to realize the above-mentioned care, the scope of telenursing must be further expanded.

[0011] However, when the scope of telenursing is expanded, the telenursing nurse's instruction resources become strained compared to face-to-face nursing, where the nurse takes the lead in following implicit work intentions.

[0012] In particular, in tasks that involve physical movements by both the worker and the patient, it is necessary to issue instructions to both the worker (the one doing the work) and the patient (the one being worked on) in parallel, including ambiguous nuances of movement that take into account implicit intentions. For example, ambiguous nuances of movement express subtle differences that are difficult to put into words, such as "I'd like you to move a little more like this."

[0013] Instead of traditional conversation, verbal instructions make it difficult to consistently convey instructions containing ambiguous nuances to both parties, and it is desirable to devise and introduce a more efficient method of conveying instructions.

[0014] For example, Non-Patent Document 1 proposes a method for supporting remote collaborative work while photographing the work object, in which a physical object is recognized from a camera image, and the remote worker uses AR to convey instructions to the object by pointing to or annotating the points that should be looked at.

[0015] This method helps workers know what to focus on and what kind of work to do. It is also an effective method because in machine repair and maintenance, etc., it is often sufficient to follow a fixed procedure to a certain extent, making it easy to eliminate ambiguity.

[0016] Gauglitz, S. et al., “World-stabilized annotations and virtual scene navigation for remote collaboration” (Link)

[0017] However, when considering application to tasks involving physical movements as mentioned above, even this method is insufficient to convey ambiguous nuances of movements.

[0018] Furthermore, if the work target is not an object such as a machine but a human being, and it is desired to control the actions of that human being, it is necessary to transmit separate instructions not only to the worker but also to the work target.

[0019] As such, all of the conventional techniques are difficult to apply to tasks that involve physical movements.

[0020] The present invention has been made in consideration of the above circumstances, and aims to provide an information presentation method, an information presentation system, and a program that can clearly communicate instructions that include ambiguous nuances of action to workers or patients who are located remotely.

[0021] To achieve the above object, a first aspect of the present invention is an information presentation system including a first terminal and a second terminal connected to each other so as to be able to communicate with each other. The first terminal is provided at a site and includes a video information acquisition unit that acquires video information of the site and a video information transmission unit that transmits the video information to the second terminal. The second terminal is provided remotely from the site and includes a video information receiving unit that receives the transmitted video information, a video information display control unit that displays the received video information on a display screen of the second terminal, an instruction input unit that accepts instructions input by an operator of the second terminal with respect to the displayed video information, an instruction information creation unit that creates instruction information by superimposing visual information based on the instructions on the displayed video information, and an instruction information transmission unit that transmits the created instruction information to the first terminal. The first terminal further includes an instruction information receiving unit that receives the transmitted instruction information and an instruction information display control unit that displays the received instruction information on a display screen of the first terminal.

[0022] According to the information presentation method, information presentation system, and program of the present invention, it is possible to clearly and efficiently convey instructions that include ambiguous nuances of action to workers or patients who are located remotely.

[0023] FIG. 1 is a conceptual diagram illustrating a remote nursing model assumed in each embodiment of the present invention. FIG. 2 is a diagram illustrating functions required to realize the remote nursing model assumed in each embodiment of the present invention. FIG. 3 is a block diagram illustrating an example of the configuration of an information presentation system according to each embodiment of the present invention. FIG. 4 is a diagram illustrating an example of a state in which an arrow image is superimposed by a remote nurse on a camera image from a worker. FIG. 5 is a diagram illustrating an example of a state in which an arrow image is superimposed by a remote nurse on a camera image from a patient. FIG. 6 is a diagram illustrating work process visualization and work process simplification. FIG. 7 is a diagram illustrating a specific example of marker-type visual information presentation. FIG. 8 is a diagram illustrating a specific example of frame-type visual information presentation. FIG. 9 is a diagram illustrating a specific example of sign-type visual information presentation. FIG. 10 is a diagram illustrating a specific example of wire-type visual information presentation. FIG. 11 is a process flow diagram illustrating an example of the basic operation of an information presentation system according to an embodiment of the present invention. FIG. 12 is a diagram illustrating a first application example in which vibration is used as a stimulus. FIG. 13 is a diagram illustrating a first application example in which sound is used as a stimulus. FIG. 14 is a diagram illustrating a second application example.

[0024] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The drawings are schematic or conceptual, and the relationship between the thickness and width of each part, the size ratio between parts, etc., are not necessarily the same as those in reality. Furthermore, even when the same part is shown, the dimensions and ratios may be different depending on the drawing. In this specification and each drawing, elements similar to those described above with reference to the previous drawings are designated by the same reference numerals, and detailed and redundant explanations will be omitted as appropriate.

[0025] FIG. 1 is a conceptual diagram for explaining a telenursing model assumed in each embodiment of the present invention.

[0026] The remote nursing model assumed in each embodiment is, as shown in Figure 1, a remotely located nurse (hereinafter also referred to as a "remote nurse") A issues instructions (d) to a patient B at the nursing site and surrounding collaborators C, such as the patient's family or visiting caregivers (hereinafter collectively referred to as "workers"), and has them perform tasks such as providing various information (e) and performing treatment (f) regarding patient B, thereby enabling a remote nursing model in which everything from detailed observation of patient B's condition to some simple treatments can be performed.

[0027] This telenursing model includes the following three functions:

[0028] The first function allows the patient or a worker to follow the instructions of a remote nurse and take pictures of any part of the patient's body, such as a wound, using a device such as a tablet or smartphone, thereby sharing flexible images and video-based information between the remote nurse and the on-site care center.

[0029] The second function is that even if treatment is required, for some simple treatments that do not require a hospital visit, the patient or a worker can follow the instructions of a remote nurse and carry out the tasks related to the treatment on their behalf.

[0030] The third function is a function that allows communication between a remote nurse and a worker in parallel with communication between a remote nurse and a patient.

[0031] FIG. 2 is a diagram illustrating the functions required to realize the remote nursing model envisioned in each embodiment of the present invention.

[0032] FIG. 3 is a block diagram showing an example of the configuration of an information presentation system according to each embodiment of the present invention.

[0033] The information presentation system 10 illustrated in Figure 3 is a system that realizes the remote nursing model shown in Figure 2, and connects an on-site terminal 20 operated by patient B or worker C and a remote terminal 30 operated by remote nurse A so that they can communicate with each other.

[0034] Both the on-site terminal 20 and the remote terminal 30 can be realized by, but are not limited to, a PC, tablet, or smartphone, and can be realized by any device that has a display and input interface and can communicate via a network such as the Internet.

[0035] Furthermore, the display screen of both the on-site terminal 20 and the remote terminal 30 can be realized as a touch screen, and can also be realized as a native application of each OS or a web application that runs on a browser.

[0036] 3 illustrates only one on-site terminal 20. However, there may be a plurality of on-site terminals 20, such as one operated by patient B, one operated by worker C, or one operated by multiple workers C. When the information presentation system 10 includes multiple on-site terminals 20, these multiple on-site terminals 20 and one remote terminal 30 can be connected to each other so that they can communicate with each other.

[0037] Furthermore, Figure 3 omits the display of parts that are common in systems that exchange information online, such as basic functions such as a login section and a session management database that links the display screens of the on-site terminal 20 and the remote terminal 30, i.e., parts that have functions that are not unique to this system.

[0038] In the information presentation system 10, the on-site terminal 20 includes a camera image information acquisition unit 21, a camera image drawing unit 22, a camera image information transmission unit 23, an instruction information reception unit 24, an instruction information display control unit 25, and an instruction type reception unit 26. On the other hand, the remote terminal 30 includes a camera image information reception unit 31, a camera image information display control unit 32, an instruction input unit 33, an instruction information creation unit 34, an instruction information transmission unit 35, an instruction type selection unit 36, and an instruction type transmission unit 37.

[0039] The camera video information acquisition unit 21 is a component that acquires on-site video information i through operation by an operator such as a worker C or a patient B, and is realized by, for example, a camera. When the on-site terminal 20 is realized by a smartphone or a tablet, it can be realized by a camera mounted on the smartphone or tablet. Therefore, hereinafter, the video information i will also be referred to as camera video i.

[0040] The camera image drawing unit 22 has a function for drawing an image or the like on the camera image i acquired by the camera image information acquisition unit 21. Therefore, an operator such as the worker C or the patient B can use the function of the camera image drawing unit 22 to draw on the camera image i acquired by the camera image information acquisition unit 21. The drawn camera image i can be viewed on the display screen of the on-site terminal 20.

[0041] The camera image information transmitting unit 23 transmits the camera image i acquired by the camera image information acquiring unit 21 to the remote terminal 30 .

[0042] In the remote terminal 30 , the camera image information receiving section 31 receives the camera image i transmitted by the camera image information transmitting section 23 .

[0043] The camera image information display control unit 32 displays the camera image i received by the camera image information receiving unit 31 on the display screen of the remote terminal 30 .

[0044] The remote nurse A can grasp the situation at the site by viewing the camera image i displayed on the display screen of the remote terminal 30. Then, when sending instructions to the patient B or the worker C at the site according to the situation at the site, the remote nurse A can operate the on-site terminal 20 and input instruction information j from the instruction input unit 33.

[0045] The instruction information creation unit 34 superimposes visual information based on the input instruction information j on the camera image i displayed on the display screen to create visual instruction information j. The visual information based on the instruction information j can be, for example, an arrow image k indicating an instruction from a remote nurse A, as shown in Figures 4 and 5 described below.

[0046] The instruction information j created by the instruction information creation unit 34 is displayed on the display screen of the remote terminal 30. The remote nurse A can check whether the instruction information j is what she intended by looking at the display screen of the remote terminal 30. If the instruction information j is different from what she intended and she wants to correct it, she can re-input the instruction information j from the instruction input unit 33.

[0047] If the instruction information j is as intended, the remote nurse A operates the remote terminal 30 to cause the instruction information transmitting unit 35 to transmit the instruction information j to the on-site terminal 20.

[0048] The instruction information j transmitted from the instruction information transmitting unit 35 to the field terminal 20 in this manner is received by the instruction information receiving unit 24 of the field terminal 20 .

[0049] The instruction information display control unit 25 causes the instruction information j received by the instruction information receiving unit 24 to be displayed on the display screen of the on-site terminal 20 .

[0050] FIG. 4 is a diagram illustrating an example in which an arrow image is superimposed by a remote nurse on the camera image from the worker.

[0051] FIG. 5 is a diagram illustrating an example in which an arrow image is superimposed by a remote nurse on a camera image from a patient.

[0052] Figures 4 (1), (2), and (3) show examples of a state in which a remote nurse A operates a remote terminal 30 to superimpose an arrow image k indicating "pointing," "movement," or "rotation" on a camera image i from a worker C as an image showing visual information.

[0053] Figures 5 (1), (2), and (3) show examples of a state in which a remote nurse A operates a remote terminal 30 to superimpose an arrow image k indicating "orientation," "position," and "posture" on a camera image i from a patient B as an image showing visual information.

[0054] Patient B, worker C, and remote nurse A can communicate with each other using their respective on-site terminals 20 and remote terminals 30, so patient B and worker C can use the on-site terminals 20 to ask questions to remote nurse A by voice, and conversely, remote nurse A can give instructions or supplements to patient B and worker C by voice.

[0055] Therefore, while remote nurse A shares the images shown in Figures 4 (1), (2), and (3) with worker C's on-site terminal 20, she can use the communication function of the remote terminal 30 to verbally provide additional instructions to worker C, such as "Get closer here," "Move a little further to the right," or "Go around to the right."

[0056] Similarly, while remote nurse A shares the images shown in Figures 5(1), (2), and (3) with patient B's on-site terminal 20, she can use the communication function of the remote terminal 30 to verbally provide supplementary instructions to patient B, such as "look down and to the right," "move your face closer," or "raise your left arm."

[0057] The visual information illustrated in Figures 4 and 5 is a simple example, and in each embodiment of the present invention, the visual information is prepared in the following two patterns and used appropriately from the perspective of how the work process is perceived depending on the level of knowledge, skill, etc. of the worker C receiving the instructions and the patient B.

[0058] The first pattern is work process visualization, and the second pattern is work process simplification.

[0059] FIG. 6 is a diagram for explaining work process visualization and work process simplification.

[0060] Hereinafter, work process visualization and work process simplification will be described with reference to FIG.

[0061] First, there are two types of work process visualization: marker type and sign type.

[0062] The marker type breaks down the work process and presents visual information for completing each process, and is suitable for instructing the camera movement (position, angle, etc.) to the worker C. For example, when giving instructions to the worker C for taking a picture of part of the face of the patient B from the lower left, the remote nurse A can transmit instructions by switching the marker used for each process, such as "move closer to the patient's face," "move the camera position a little lower," or "move the camera slightly to the left."

[0063] The sign type is used to break down and instruct the process of guiding the body to a desired state, and is suitable for instructing the body movements (posture, pose, etc.) of patient B. For example, it can convey movement instructions such as "turn your head to the upper right" or "lower your left shoulder."

[0064] There are two types of work process simplification: frame type and wire type.

[0065] The frame type conceals the work process and presents visual information only for achieving the final task (obtaining the expected value), and is suitable for instructing the worker C on camera movements (position, angle, etc.). With the frame type, the position and angle of the camera's outer frame are adjusted to fit the frame, without considering how the subject will be captured. For example, a four-sided frame can be displayed with the face of patient B as the starting point so that the image is captured from the bottom left, and instructions can be conveyed by adjusting the position and angle of this frame.

[0066] The wire type provides instructions while concealing the process of guiding the body to the desired state, and is suitable for instructing the body movements (posture, pose, etc.) of patient B. With the wire type, you do not need to think about the position to be taken, but simply adjust the wire to the position and angle of the face and body.

[0067] In this way, the types of instructions given by superimposing visual information on the camera image i can be broadly classified into either work process visualization or work process simplification, and further classified into marker type or sign type in work process visualization, and frame type or wire type in work process simplification, depending on how they are realized. Note that the expressions are not necessarily limited to those aimed at the worker or patient (for example, instructions may be given to the patient in marker type).

[0068] Therefore, the remote terminal 30 is equipped with an instruction type selection unit 36 ​​so that the remote nurse A can select the type of instruction to be given by superimposing visual information on the camera image i from the four instruction types mentioned above that have been prepared in advance.

[0069] The remote nurse A inputs a selection instruction to select one of the four instruction types described above into the instruction type selection unit 36 ​​.

[0070] The instruction type transmitting unit 37 transmits the instruction type m input to the instruction type selecting unit 36 ​​to the field terminal 20 .

[0071] In the field terminal 20 , the instruction type m transmitted by the instruction type transmitting unit 37 is received by the instruction type receiving unit 26 .

[0072] In this way, the information presentation system 10 can capture images of the condition of patient B at the scene and display the camera image i on the remote terminal 30 of remote nurse A. This allows remote nurse A to visually grasp the status of treatment f at the scene.

[0073] Remote nurse A can operate the remote terminal 30 to superimpose visual information, such as an arrow image k, on the displayed camera image i and present it to patient B and worker C. This allows remote nurse A to give accurate instructions to patient B and worker C on-site, ensuring that work is carried out appropriately at the site.

[0074] Next, specific presentation of visual information for each of the four instruction types mentioned above, namely, marker type, frame type, sign type, and wire type, will be described.

[0075] When visual information is to be presented using one of these four instruction formats, the remote nurse A operates the remote terminal 30 to select an instruction format from the instruction format selection unit 36 ​​.

[0076] The selected instruction type m is transmitted to the on-site terminal 20 by the instruction type transmitting unit 37, and is received by the instruction type receiving unit 26 in the on-site terminal 20. This allows the on-site terminal 20 to recognize in which instruction type the visual information will be presented from the remote terminal 30.

[0077] When the remote nurse A selects the instruction type m, he / she can further use the function of the instruction information creation unit 34 to give specific instructions according to the type, as will be described below.

[0078] (Marker Type) The marker type is suitable for a remote nurse A to visualize the work process and give instructions to a patient B or a worker C on-site.

[0079] FIG. 7 is a diagram for explaining a specific example of presentation of visual information in a marker type.

[0080] The visual information presented by the marker type includes, but is not limited to, a pointing instruction (1), a movement instruction (2), and a rotation instruction (3).

[0081] In the case of pointing instruction (1), remote nurse A designates an arbitrary location m1 in the camera image i displayed on the display screen 38 of the remote terminal 30. For example, if the display screen 38 is a touch panel type, this designation can be performed by directly pressing the display screen 38 with a touch pen or a finger. The following description will be given assuming that the display screen 38 is a touch panel type.

[0082] In response to a touch with a touch pen or a finger, the on-site terminal 20 displays a marker k1 such as an arrow in the camera image i displayed on the display screen 28 at a location corresponding to (the same location as) the location pressed by the remote nurse A. In this case, AR display may be used to make the marker k1 follow the object in the camera image pointed at. Note that the method for estimating and tracking the pointed object is not limited in this specification, but can be performed using an existing analysis engine such as ARCore or ARKit.

[0083] In this way, remote nurse A can instruct the site on the work target, and worker C can accurately understand the instructions from remote nurse A.

[0084] In the case of movement instruction (2), the remote nurse A presses one of the four movement buttons m2 in the left, right, up, and down directions on the display screen 38 of the remote terminal 30.

[0085] In response to this, the on-site terminal 20 displays a marker k2 such as a triangle in the camera image i displayed on the display screen 28. When the remote nurse A presses the movement button m2, the expected amount of movement can be gradually changed, for example, by the length of the press, and the size of the marker k2 displayed on the display screen 28 of the on-site terminal 20 and the animation when it is displayed can be adjusted based on the expected amount of movement.

[0086] In this way, remote nurse A can instruct worker C to move the imaging location, and worker C can accurately understand the instructions from remote nurse A.

[0087] In the case of rotation instruction (3), the remote nurse A presses one of the left / right, up / down rotation buttons m3 on the display screen 38 of the remote terminal 30.

[0088] In response to this, the on-site terminal 20 displays a marker k3 such as a wraparound arrow in the camera image i displayed on the display screen 28. In this case, as with the movement instruction (2), the amount of movement can be obtained according to the length of time the button is pressed and reflected in the marker display.

[0089] In this way, remote nurse A can instruct operator C to rotate the imaging location, and operator C can accurately understand the instructions from remote nurse A.

[0090] In addition, for the movement instruction (2) and the rotation instruction (3), in addition to the above-mentioned methods, if necessary, instruction buttons may be provided to convey movements such as moving backward or forward in the case of movement, or rotating the terminal itself clockwise or counterclockwise in the case of rotation.

[0091] (Frame Type) The frame type is suitable for a remote nurse A to simplify the work process and give instructions to a patient B and a worker C on-site.

[0092] FIG. 8 is a diagram for explaining a specific example of frame-type visual information presentation.

[0093] The visual information presented by the frame type includes, but is not limited to, the position of the frame in the camera image i seen on the display screen 28 of the on-site terminal 20 and adjustment of the angle.

[0094] In the frame-based approach, remote nurse A specifies the three-dimensional coordinates p1 and rotation angle p2 of the frame on the display screen 38 of the remote terminal 30. Alternatively, by providing a tracking function to the remote terminal 30, the remote terminal 30 can track the subject (e.g., patient B's face) from the camera image i and acquire its position and angle as needed, thereby displaying the frame r in AR so that it always follows the subject as the origin. While not limited to this specification, face tracking techniques can be performed using existing analysis engines such as ARCore and ARKit. The subject is not limited to patient B's face, but can also be, for example, other body parts estimated by skeletal structure or specific wounds determined by image analysis.

[0095] In this way, a frame of relative position and tilt with the subject being the origin is displayed on the display screen 28 of the on-site terminal 20. Therefore, when the on-site terminal 20 is moved, the worker C can recognize that the outer frame of the camera image i moves to exactly match the frame r.

[0096] (Sign Type) The sign type is suitable for a remote nurse A to visualize the work process and give instructions to guide the on-site patient B and worker C to the expected state.

[0097] FIG. 9 is a diagram for explaining a specific example of sign-type visual information presentation.

[0098] The visual information presented by the sign type includes, but is not limited to, movement instructions (1), rotation instructions (2), and poses (3).

[0099] In the case of movement instruction (1), remote nurse A pinches in / out at an arbitrary location s1 in the in-camera image i displayed on the display screen 38 of the remote terminal 30.

[0100] In response to this, on the field terminal 20, a sign k4 indicating moving away / approaching is displayed as an arrow in the camera image i displayed on the display screen 28.

[0101] In this way, remote nurse A can give instructions to the site to guide it to the desired state, and worker C can accurately understand the instructions from remote nurse A.

[0102] In this example, since the camera is an in-camera, it is assumed that the subject being photographed, for example, patient B, is captured near the center of the camera image i. However, if the camera is not an in-camera, a separate movement instruction function m2 for left / right and up / down directions may be provided in the remote terminal 30, similar to the marker-type movement instruction described using Figure 7.

[0103] The rotation instruction (2) is similar to the rotation instruction (3) explained using FIG. 7, so a duplicate explanation will be avoided.

[0104] In the case of pose instruction (3), the remote nurse A can select a desired pose from the pose list menu s2 displayed on the display screen 38 of the remote terminal 30. Poses include, but are not limited to, raising the left arm, raising the right arm, closing the eyes, opening the eyes, etc., as shown in FIG.

[0105] In response to a pose selected from the pose list menu s2, a sign k6 indicating the selected pose is displayed on the display screen 28 of the on-site terminal 20.

[0106] In this way, remote nurse A can give instructions to the site to guide it to the desired state, and worker C can accurately understand the instructions from remote nurse A.

[0107] (Wireless Type) The wireless type is suitable for a remote nurse A to simplify the work process and give instructions to guide a patient B or a worker C on-site to the expected state.

[0108] FIG. 10 is a diagram for explaining a specific example of wire-type visual information presentation.

[0109] By presenting visual information in the form of a wire, the remote nurse A can specify the value of the rotation angle v of the wire u in three-dimensional space from the display screen 38 of the remote terminal 30.

[0110] In response to this, the on-site terminal 20 draws a wire u on the display screen 28, with the wire u inclined relative to the subject being imaged as the origin. If the remote terminal 30 is provided with a tracking function, this tracking function may be used to display AR in the wire type so that the AR always follows an image such as the face of patient B as the origin, just like the frame type. Note that the subject being imaged is not limited to the face of patient B, and can also be, for example, another body part obtained by skeletal estimation or a specific wound determined by image analysis.

[0111] In this way, the remote nurse A can instruct the on-site, and in particular the patient B, on the angle of the face. In response, the patient B can recognize that he or she should adjust the angle of his or her face so that the contour line t of his or her face exactly matches the wire u.

[0112] Note that the method of individually manipulating each value of the rotation angle v as described above is merely one example of how to move the frame and wire of the frame-type and wire-type frames, and is not limited to this method, and other methods can also be used.

[0113] As an example, we will explain how to implement this by preparing a preset. For example, when pressing the preset button for the wire-type camera that instructs "view from the bottom left," the center of the frame moves to the top right of the face, resulting in the patient's face being viewed from the bottom left (the image captured by the camera is from the bottom left). In this way, by preparing various presets to which each value of the rotation angle v that realizes the content is linked, nurses can save time by manually adjusting the value.

[0114] As described above, according to the information presentation system of each embodiment of the present invention, by presenting visual information in the form of markers, frames, signs, and wires, it becomes possible for a remote nurse A to accurately convey instructions to a patient B or a worker C on-site, even if the instructions include ambiguous nuances of action.

[0115] Next, an example of the operation of the information presentation system according to the embodiment of the present invention will be described.

[0116] FIG. 11 is a process flow diagram showing an example of a basic operation of the information presentation system according to the embodiment of the present invention.

[0117] First, remote nurse A selects an instruction type m from the instruction type selection unit 36 ​​according to the person to whom the instruction is to be given (worker C, patient B, or both) and the skill and experience of that person (S1). When instructions are to be given to both worker C and patient B at the same time, an instruction type (marker type or frame type, sign type or wire type) is selected for each of them.

[0118] The selected instruction type m is sent from the instruction type selection unit 36 ​​to the instruction type transmission unit 37, and then transmitted from the instruction type transmission unit 37 to the field terminal 20. The instruction type m is received by the instruction type reception unit 26 in the field terminal 20. As a result, the instruction type m is notified to the field terminal 20 (S2).

[0119] In the on-site terminal 20, a function corresponding to the instruction type m is prepared based on the notified instruction type m (S3). For example, if the instruction type m is a wire type or a frame type, image processing for superimposing a wire or the like by face tracking is performed when the camera image i is acquired, and an AR library is read.

[0120] In the on-site terminal 20, when the camera image information acquisition unit 21 acquires the camera image i, the camera image drawing unit 22 displays the camera image i on the display screen 28, and the camera image information transmission unit 23 transmits the camera image i to the remote terminal 30 (S4). Thereafter, the camera image i is continuously transmitted in real time from the on-site terminal 20 to the remote terminal 30. At this time, as an example, the stream can be transmitted and received between the on-site terminal 20 and the remote terminal 30 using WebRTC.

[0121] The camera image i transmitted by the camera image information transmitting unit 23 is received by the camera image information receiving unit 31 in the remote terminal 30, and is displayed on the display screen 38 by the camera image information display control unit 32 (S5).

[0122] While viewing the camera image i displayed on the display screen 38 in this way, the remote nurse A can input and update visual information based on the instruction information j as needed using the operation UI of the instruction input unit 33 (S6). In this case, it is also possible to acquire events not only for the start of a new instruction, but also for the stop or change of an existing instruction (such as a change in the strength of a marker-type "move" or "rotate").

[0123] The instruction information j thus input and updated is transmitted from the instruction information transmitting unit 35 to the on-site terminal 20 and received by the instruction information receiving unit 24. In addition, visual information based on the instruction information j is superimposed on the camera image i by the instruction information creating unit 34 and displayed on the display screen 38 (S7).

[0124] In the field terminal 20, the instruction information display control unit 25 causes the display screen 28 to display visual information based on the instruction information j superimposed on the camera image i (S8).

[0125] Next, application examples that can be realized by the information presentation system according to each embodiment of the present invention will be described.

[0126] (First Application Example) In the first application example, instructions are presented in a different modal manner.

[0127] In the above embodiment, a method of transmitting instructions using visual information has been described, but in this application example, a stimulus in another modality is also used in combination.

[0128] FIG. 12 is a diagram illustrating a first application example in which vibration is used as a stimulus.

[0129] When vibration is used as a stimulus, the following stimuli can be presented, for example, using the vibration function of each on-site terminal 20 operated by the patient B or the worker C.

[0130] For example, with the marker type mentioned above, the strength and frequency of vibrations can be changed depending on the distance to the target of the "pointing instruction" or the strength of the "movement instruction" or "rotation instruction," making it possible to convey distance and strength in a physical sense.

[0131] In this case, the distance to the target can be acquired using, for example, a depth sensor of the on-site terminal 20 .

[0132] Furthermore, by using the frame type described above, the intensity and frequency of vibrations can be changed depending on the distance between the position of frame r and the current position of the on-site terminal 20, and the distance can be conveyed in a sensory manner.

[0133] FIG. 13 is a diagram illustrating a first application example in which sound is used as a stimulus.

[0134] When using sounds such as voice or acoustics as stimuli, the stimuli can be presented, for example, as follows using the stereophonic function of each on-site terminal 20 operated by patient B and worker C.

[0135] For example, by using the marker type mentioned above, audio can be played from the direction of the target depending on the distance and positional relationship to the target of the pointing instruction, making it possible to convey the sensory direction in which the user should approach.

[0136] In addition, by using the frame type described above, audio that can be heard from the direction of frame r can be played back depending on the distance and positional relationship between the position of frame r and the current position of the on-site terminal 20, so that the direction in which the user should approach can be conveyed intuitively.

[0137] (Second Application Example) In the second application example, variations in input of instructions in each of the above-mentioned types will be described.

[0138] FIG. 14 is a diagram illustrating a second application example.

[0139] In each of the above-described embodiments, the input of each instruction type m by remote nurse A is explained as being performed by manual input, but in the second application example, the input is performed by the physical movement of remote nurse A rather than by manual input.

[0140] To implement this, in this application example, a web camera is provided on the remote terminal 30 of remote nurse A. The position and angle of remote nurse A's face are acquired from this web camera, and instructions are automatically given based on the acquired information.

[0141] For example, if the instruction type m is a marker type, when a remote nurse A looks from the right while watching the camera image i from the on-site terminal 20, a "rotation instruction" requesting the nurse to turn to the right is displayed on the display screen 28 of the on-site terminal 20.

[0142] Furthermore, when the instruction type m is a wire type, if the remote nurse A lowers her face to the lower left while watching the camera image i from the on-site terminal 20, the center of the wire u on the display screen 28 of the on-site terminal 20 is also displayed to move to the lower left in accordance with the movement.

[0143] As described above, according to each embodiment of the present invention, it is possible to clearly and efficiently convey instructions that include ambiguous nuances of operation to a remote worker or patient.

[0144] The present invention is not limited to the above-described embodiments, and the components can be modified and embodied in practice without departing from the spirit of the invention. Furthermore, various inventions can be created by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

[0145] For example, in each of the above-described embodiments, a scene is envisioned in which, following instructions from a remote nurse A, patient B himself or worker C on-site acquires flexible video-based information, such as visually inspecting the affected area of ​​patient B (e.g., part of the face), and performs simple treatment based on that information.

[0146] However, the present invention is not limited to such scenes, but can also be applied to scenes involving remote collaborative work in general, such as work scenes in which instructions reflecting the intentions of an instructor in a remote location are transmitted to one or more workers.

[0147] 10 Information presentation system 20 On-site terminal 21 Camera image information acquisition unit 22 Camera image drawing unit 22 Camera image creation unit 23 Camera image information transmission unit 24 Instruction information reception unit 25 Instruction information display control unit 26 Instruction type reception unit 28 Display screen 30 Remote terminal 31 Camera image information reception unit 32 Camera image information display control unit 33 Instruction input unit 34 Instruction information creation unit 35 Instruction information transmission unit 36 ​​Instruction type selection unit 37 Instruction type transmission unit 38 Display screen A Remote nurse B Patient C Worker d Instruction e Information provision f Treatment i Image information, camera image j Instruction information k Arrow image k1 Marker such as arrow k2 Marker such as triangle k3 Marker such as a circling arrow k4 Sign indicating moving away / approaching k6 Sign indicating selected pose m Instruction type m1 Any location m2 Movement button, movement instruction function m3 Rotation button p1 Coordinates p2 Rotation angle r Frame s1 Any location s2 Pose list menu t Contour line u Wire v Rotation angle

Claims

1. An information presentation system comprising a first terminal and a second terminal connected to each other so as to be able to communicate with each other, wherein the first terminal is provided at a site and comprises a video information acquisition unit that acquires video information of the site, and a video information transmission unit that transmits the video information to the second terminal, the second terminal is provided remotely from the site and comprises a video information receiving unit that receives the transmitted video information, a video information display control unit that causes the received video information to be displayed on a display screen of the second terminal, an instruction input unit that accepts instructions input by an operator of the second terminal for the displayed video information, an instruction information creation unit that creates instruction information by superimposing visual information based on the instructions on the displayed video information, and an instruction information transmission unit that transmits the created instruction information to the first terminal, and the first terminal further comprises an instruction information receiving unit that receives the transmitted instruction information, and an instruction information display control unit that causes the received instruction information to be displayed on a display screen of the first terminal.

2. The information presentation system of claim 1, wherein the second terminal further comprises: an instruction type selection unit that accepts a selection instruction from an operator of the second terminal to select the instruction type from instruction types prepared in advance; and an instruction type transmission unit that transmits the instruction type selected in accordance with the selection instruction to the first terminal; and the first terminal further comprises an instruction type receiving unit that receives the transmitted instruction type.

3. The information presentation system of claim 1, wherein the site is a medical site, the operator of the first terminal is a patient or a worker performing medical procedures on the patient, and the operator of the second terminal is an instructor who instructs the patient or the worker to perform the medical procedures.

4. The information presentation system according to claim 1, wherein the visual information based on the instructions includes an arrow image representing an instruction from a person instructing the medical work.

5. The information presentation system of claim 1, wherein the visual information based on the instruction is a wire used to indicate the movement of a subject in the video information acquired by the video information acquisition unit.

6. An information presentation system as described in claim 1, wherein the visual information based on the instruction is a frame for adjusting the position and angle of the outer frame of the video information acquired by the video information acquisition unit.

7. An information presentation method implemented by an information presentation system comprising a first terminal and a second terminal connected to each other so as to be able to communicate with each other, wherein the first terminal provided at a site acquires video information of the site and transmits the video information to the second terminal, the second terminal provided remotely from the site receives the transmitted video information and displays the received video information on a display screen of the second terminal, accepts instructions input by an operator of the second terminal for the displayed video information, superimposes visual information based on the instructions on the displayed video information to create instruction information, and transmits the created instruction information to the first terminal, and the first terminal further receives the transmitted instruction information and displays the received instruction information on a display screen of the first terminal.

8. A program for causing a computer to function as each unit of the information presentation system according to any one of claims 1 to 6.

Images