Information presentation system, information presentation method, and program

Abstract

According to the present invention, an operator Hw views an observation object K (work object) through a glass type display terminal 10 (AR glass), and transfers and displays an object image PV including the observation object K imaged by an imaging communication terminal 20 (smartphone with a camera) attached to the chest to an instruction terminal 30 (PC) of a remote instructor Hi. When the instructor Hi operates so as to display a 3D marker M (instruction content) for instructing an action to the operator Hw on the object image PV displayed on the instruction terminal 30 and indicate a desired position and direction, the imaging communication terminal 20 converts, on the basis of a positional relationship between the imaging communication terminal 20 and the glass type display terminal 10 with respect to the observation object K, the 3D marker M received from the instruction terminal 30 into a correction 3D marker Mr (correction instruction content) in which a positional deviation between an imaging position at the imaging communication terminal 20 that corresponds to the viewpoint of the instructor Hi and a viewpoint of the operator Hw at the glass type display terminal 10 is corrected, and displays the correction 3D marker Mr by superimposition on a visual field image FV of the glass type display terminal 10.

Description

Information presentation system, information presentation method, and program

[0001] An embodiment of the present invention relates to an information presentation system, an information presentation method, and a program.

[0002] Due to factors such as a shortage of workers and the need to improve work efficiency, attention is being focused on remote work support that uses smart glasses to provide work instructions remotely.

[0003] In these work support cases, see-through smart glasses are often used, and images of the site, including the work object, are transmitted to a remote location through a camera attached to the glasses worn by the on-site worker, and work instructions are often given using AR (Augmented Reality) technology (see, for example, Non-Patent Document 1).

[0004] On the other hand, in contrast to expensive, highly functional smart glasses equipped with cameras, relatively inexpensive AR glasses that connect to host devices such as smartphones and are specialized for outputting images using AR technology are also in practical use.

[0005] If such AR glasses could be used to provide remote work support, there would be cost benefits.

[0006] "Development of a Remote Field Work System Using AR Display on Smart Glasses," Haruhisa Kato, Tatsuya Kobayashi, Tomohiro Tsuji, Masaru Kanno, Hiromasa Yanagihara, Journal of the Institute of Image Information and Television Engineers, Vol. 71, No. 1, pp. J35-J43 (2017)

[0007] The AR glasses currently in use do not have cameras, and many use the camera on a connected smartphone.

[0008] In this case, the position of the AR glasses and the smartphone on the worker is different, for example, the smartphone is attached to the worker's chest using a neck mount, so the image of the work target seen by the instructor on the remote side is from a different perspective than the worker's. The instructor gives work instructions based on an image from a different perspective than the worker's, which creates a sense of incongruity in the instructions displayed on the worker's AR glasses.

[0009] Therefore, conventional remote work support systems using AR glasses may cause workers to take incorrect actions or place a cognitive burden on the workers as they try to compensate for the difference in their viewpoint from that of the instructor.

[0010] The present invention has been made in consideration of such problems, and aims to provide an information presentation system, an information presentation method, and a program that enable the content of instructions given by an instructor to be displayed on AR glasses without creating a sense of incongruity, even when the position where the worker views the work object through AR glasses and the position where the instructor captures the image of the work object are separated.

[0011] An information presentation system according to an embodiment of the present invention is an information presentation system comprising: a glass-type display terminal; a first terminal connected to the glass-type display terminal and capturing an image of an observation target included in the field of view of the glass-type display terminal at a position away from the glass-type display terminal; and a second terminal having a communication function with the first terminal and displaying a target image including the observation target captured by the first terminal, wherein the first terminal comprises a control unit that executes the following processes: a process of receiving instructions of a 3D model displayed on the target image in response to a user operation of the second terminal; a process of converting the received instructions of the 3D model into instructions in a field of view image seen from the glass-type display terminal based on the positional relationship between the glass-type display terminal and the first terminal relative to the observation target; and a process of superimposing and displaying the converted instructions of the 3D model on the field of view image of the glass-type display terminal.

[0012] According to the information presentation system of an embodiment of the present invention, even if the viewpoint from which the instructor captures the image of the work object seen by the instructor is different from the viewpoint from which the worker views the work object through the AR glasses, it is possible to display the content of the instructor's instructions on the AR glasses without any sense of incongruity.

[0013] FIG. 1 is a diagram illustrating an overview (part 1) of a remote work support system 1 relating to an information presentation system, an information presentation method, and a program according to an embodiment. FIG. 2 is a diagram illustrating an overview (part 2) of the remote work support system 1 according to an embodiment. FIG. 3 is a functional block diagram illustrating the configuration of functions possessed by the remote work support system 1 according to an embodiment. FIG. 4 is a block diagram illustrating the hardware configuration of an imaging communication terminal 20 in the remote work support system 1. FIG. 5 is a flowchart illustrating an example of processing operations performed in cooperation between the imaging communication terminal 20 and the instruction terminal 30. FIG. 6 is a diagram illustrating the positional relationship between a camera-equipped smartphone 20 (imaging communication terminal) and AR glasses 10 (glasses-type display terminal) with respect to an observation target K (work target) calculated by a calculation unit 23 of the imaging communication terminal 20. Figure 7 is a diagram comparing the display state of a 3D marker M in response to an instruction from an instructor Hi on a target image (video) PV including an observation target K that is captured by an imaging communication terminal 20 and displayed on an instruction terminal 30, and the display state of a corrected 3D marker Mr that corrects the misalignment of the viewpoints of the instructor Hi and the worker Hw and is superimposed on the field of view image FV of the glass-type display terminal 10.

[0014] Hereinafter, an information presentation system, an information presentation method, and a program according to an embodiment will be described with reference to the drawings.

[0015] (Outline of the embodiment) FIG. 1 is a diagram illustrating an outline (part 1) of a remote operation support system 1 relating to an information presentation system, an information presentation method, and a program according to an embodiment.

[0016] FIG. 2 is a diagram illustrating an outline (part 2) of the remote operation support system 1 according to the embodiment.

[0017] In the embodiment, the remote work support system 1 will be described assuming that it is applied to telemedicine. It goes without saying that the remote work support system 1 is not limited to telemedicine, but can also be applied as a system for remotely supporting any on-site work, such as construction work or electrical work.

[0018] The remote work support system 1 of the embodiment includes AR glasses 10 (glasses-type display terminal) worn on the head of a nurse Hw (worker) in a local examination room R1, which is a work site as shown in FIG. 1(A), and a camera-equipped smartphone 20 (image capture communication terminal: first terminal) worn on the chest, and a PC (Personal Computer) 30 (instruction terminal: second terminal) used by a doctor Hi (instructor) in a remote examination room R2, which is an instruction site as shown in FIG. 1(B), and is configured by communicating between the smartphone 20 (image capture communication terminal) and the PC 30 (instruction terminal) via a communication line N.

[0019] The AR glasses 10 (glasses-type display terminal) are used by connecting to their host device, a smartphone 20 (imaging communication terminal), via a wired communication path L such as a USB (Universal Serial Bus) cable. The smartphone 20 (imaging communication terminal) is used by being worn on the chest of a nurse Hw (worker), for example, by a neck mount.

[0020] The glasses-type display terminal 10 is not limited to AR glasses, and may be other types of smart glasses that have a function of displaying an image (video) superimposed on the visual field image of the worker Hw.

[0021] The imaging communication terminal 20 is not limited to a smartphone, but may be other tablet terminals, PDAs (Personal Digital Assistants), portable game consoles, etc. that have the function of capturing images (video) and the function of communicating with other communication terminals.

[0022] The instruction terminal 30 is not limited to a PC, but may be a tablet terminal or the like that has the function of displaying images (video), the function of inputting data (signals) according to user operations, and the function of communicating with other communication terminals.

[0023] In the local examination room R1, a nurse Hw (worker) views an object of observation K (work object), such as a patient, in a visual field image FV through AR glasses 10 (glasses-type display terminal), and also fits the object of observation K within the angle of view captured by a smartphone 20 (imaging communication terminal).

[0024] At this time, the target image (video) PV including the observation object K (work object) captured by the smartphone 20 (imaging communication terminal) is transferred to the PC 30 (instruction terminal) used by the doctor Hi (instructor) in the remote examination room R2, as shown in Figure 1 (B), and displayed on the display screen of the PC 30 (instruction terminal).

[0025] A doctor Hi (instructor) can examine an observation object K (work object) in a local examination room R1 as an object image (video) PV while in a remote examination room R2.

[0026] The doctor Hi (instructor) operates input devices 31 such as a keyboard and mouse to display a 3D marker M (three-dimensional model) indicating instructions (here, the direction in which to move the observation target K) to the nurse Hw (worker) regarding the observation target K (work object) on the target image (video) PV displayed on the PC 30 (instruction terminal), and moves or rotates the 3D marker M in a desired direction to instruct the direction in which to move the observation target K. Note that the instructions to the nurse Hw (worker) by the 3D marker M may be the direction in which the nurse Hw moves based on the position of the observation target K, such as an instruction for the nurse Hw to move from the front of the observation target K to the side.

[0027] The 3D model (instruction content) that the instructor Hi displays on the target image PV to instruct the worker Hw on an action does not have to be limited to the 3D marker M.

[0028] Display (numerical) data including the type (shape), position, and inclination of the 3D marker M (instruction content) displayed on the target image (video) PV of the PC 30 (instruction terminal) is transmitted to the smartphone 20 (imaging communication terminal) of the nurse Hw (worker).

[0029] As shown in FIGS. 1A and 2 , the smartphone 20 (imaging communication terminal) converts the display (numerical value) data of the 3D marker M (instruction content) received from the PC 30 (instruction terminal) into a positional relationship (angle θ １ , θ ２, distances a, b, c), the positional deviation (the elevation / depression angle difference θ ２ ) is converted into display (numerical) data of the corrected corrected 3D marker Mr (correction instruction content), and the converted corrected 3D marker Mr is superimposed on the field of view image FV of the AR glasses 10 and displayed.

[0030] Even if the nurse Hw and the doctor Hi have different viewpoints of the observation object K due to a positional misalignment between the AR glasses 10 and the smartphone 20, the instructions for action from the doctor Hi to the nurse Hw based on the target image PV can be converted into instructions in the field of view image FV seen from the AR glasses 10 and displayed.

[0031] (Configuration of the embodiment) FIG. 3 is a functional block diagram showing the configuration of functions possessed by the remote operation support system 1 of the embodiment.

[0032] The glasses-type display terminal 10 (AR glasses) includes at least a display unit 11 and an inertial measurement unit (A: first inertial measurement unit) 12 (IMU: Inertial Measurement Unit).

[0033] The display unit 11 has, for example, an organic EL display, and displays an image (video) received from the imaging communication terminal (first terminal) 20 on the display, and also displays it superimposed on the visual field image FV via a mirror lens.

[0034] The inertial measurement unit (A) 12 detects and acquires the tilt (angle and angular velocity) of the glasses-type display terminal 10 in three orthogonal axial directions.

[0035] The imaging communication terminal 20 (smartphone) includes a control unit (not shown) that controls the operation of each unit according to an application program, a communication unit (not shown) for each of the glass-type display terminal 10 and the instruction terminal (second terminal) 30, and a touch panel display unit (not shown), as well as at least a camera unit 21, an inertial measurement unit (B: second inertial measurement unit) 22 (IMU), a calculation unit 23, and a rendering unit 24.

[0036] The camera unit 21 includes an imaging module built into and fixed to the imaging communication terminal 20, and captures an external image (video) corresponding to the angle of view of the imaging module as a color image. The captured image (video) of the imaging target (target image PV) is transmitted to the instruction terminal 30 via the communication unit.

[0037] The inertial measurement unit (B) 22 detects and acquires the tilt (angle and angular velocity) of the imaging communication terminal 20 in three orthogonal axis directions.

[0038] The calculation unit 23 acquires data on the tilt of the glass type display terminal 10 detected by the inertial measurement unit (A) 12 of the glass type display terminal 10 and data on the tilt of the imaging communication terminal 20 detected by the inertial measurement unit (B) 22, and calculates the difference between the data to calculate the elevation / depression angle difference θ between the glass type display terminal 10 and the imaging communication terminal 20. ２ In addition, a predetermined positional relationship between the glass type display terminal 10 and the imaging communication terminal 20 (a linear distance a (vertical distance b) and a direction θ １ ) and calculates and acquires the distance c from the glass type display terminal 10 to the observation target K.

[0039] The rendering unit 24 calculates the elevation / depression angle difference θ between the glass-type display terminal 10 and the imaging communication terminal 20 acquired by the calculation unit 23 for the 3D marker M (three-dimensional model) that instructs the worker Hw to take an action, received from the instruction terminal 30. ２ The imaging angle and imaging distance to be corrected by rendering are determined using the distance c from the glass-type display terminal 10 to the observation target K, and the 3D marker M (three-dimensional model) is rendered into a corrected 3D marker Mr in which the positional deviation (elevation / depression angle difference) between the imaging position on the smartphone 20 (corresponding to the viewpoint of the instructor Hi) and the viewpoint of the worker Hw on the AR glasses 10 is corrected (3D rendering). That is, the display (numerical value) data of the 3D marker M on the target image PV is corrected by the elevation / depression angle difference θ ２and the distance c, the positional deviation between the viewpoint of the instructor Hi and the viewpoint of the worker Hw is corrected, and the corrected 3D marker Mr is converted into a corrected 3D marker Mr. The corrected 3D marker Mr (three-dimensional model) obtained as a result of the rendering is transmitted to the display unit 11 of the glasses-type display terminal 10 via the communication unit, and is displayed superimposed on the visual field image FV.

[0040] The instruction terminal 30 (PC) has at least a control unit (not shown) that controls the operation of each unit in accordance with an application program, a communication unit (not shown) with the imaging communication terminal 20, a display unit (not shown) that displays images (video), and an input unit (31) such as a keyboard or mouse, and is a terminal that allows an instructor Hi (doctor) to instruct an operator Hw (nurse) on an action using a 3D marker M based on an image (video) of the observation object K (object image PV) that is captured by the imaging communication terminal 20 and displayed on the display unit.

[0041] FIG. 4 is a block diagram showing the hardware configuration of the imaging communication terminal 20 in the remote operation support system 1. As shown in FIG.

[0042] As shown in FIG. 4, the imaging communication terminal 20 includes a control circuit (processor) 41 which is a control unit.

[0043] The control circuit 41 is connected via a system and data bus to a memory 42 such as RAM, a storage device 43 including a storage medium 43a such as a magnetic disk or SSD (Solid State Drive), a user interface 44 including keys, switches, a touch panel, an external input terminal for the user to operate from the outside, and a display for outputting data, a speaker, a printer, and an external output terminal, an imaging module 44P including a CCD (Charge Coupled Device), a wired communication module 45 and / or a wireless communication module 46 for communicating with the outside, and the like.

[0044] The glasses-type display terminal 10 may be connected to the imaging communication terminal 20 via a wired communication module 45 via a wired communication path L (see FIG. 1).

[0045] The instruction terminal 30 may be connected to the imaging communication terminal 20 via a communication line N (see FIG. 1), for example, via a wired communication module 45 and / or a wireless communication module 46 .

[0046] The control circuit 41 controls the operation of each part of the imaging communication terminal 20 based on the control program that controls the processing of the imaging communication terminal 20 and that is stored in the control program area 42a of the memory 42, and on the management information that is stored in or read from the management information area 42b of the memory 42. The control circuit (processor) 41 is not limited to one processor, and may include multiple processors.

[0047] The control program stored in the control program area 42a of the imaging communication terminal 20 includes functions corresponding to the processes performed by the communication unit (not shown), touch panel display unit (not shown), camera unit 21, inertial measurement unit (B) 22, calculation unit 23, and rendering unit 24 described with reference to Figure 3.

[0048] In addition, the management information area 42b of the imaging communication terminal 20 stores or reads data that is input, acquired, generated, updated, output, etc. in accordance with each process performed by the communication unit (not shown), touch panel display unit (not shown), camera unit 21, inertial measurement unit (B) 22, calculation unit 23, and rendering unit 24 described with reference to Figure 3.

[0049] The instruction terminal 30 is configured with the same hardware as the imaging communication terminal 20. Functions corresponding to the processes performed by the camera unit 21 and the rendering unit 24 may be executed in cooperation with a control unit (processor) of the instruction terminal 30.

[0050] In the remote work support system 1 configured in this manner, the control circuits (control units) of the imaging communication terminal 20 and the instruction terminal 30 control the operation of each part in accordance with the commands written in the control program, and the software and hardware work together to realize various functions as described in the operation explanation below.

[0051] (Operation of the embodiment) Next, the operation of the remote operation support system 1 according to the information presentation system of the embodiment will be described.

[0052] FIG. 5 is a flowchart showing an example of a processing operation performed by the imaging communication terminal 20 and the instruction terminal 30 in cooperation with each other.

[0053] Figure 6 is a diagram explaining the positional relationship between the camera-equipped smartphone 20 (imaging communication terminal) and the AR glasses 10 (glasses-type display terminal) relative to the observation object K (work object) calculated by the calculation unit 23 of the imaging communication terminal 20.

[0054] Figure 7 is a diagram comparing the display state of a 3D marker M in response to an instruction from an instructor Hi on a target image (video) PV including an observation target K that is captured by an imaging communication terminal 20 and displayed on an instruction terminal 30, and the display state of a corrected 3D marker Mr that corrects the misalignment of the viewpoints of the instructor Hi and the worker Hw and is superimposed on the field of view image FV of the glass-type display terminal 10.

[0055] <Prerequisites> The worker Hw (nurse) adjusts the positional relationship between himself / herself and the object of observation K in advance so that the object of observation K is captured at the center of both the image (video) captured by the imaging communication terminal (first terminal) 20 (smartphone) and the field of view image FV through the glass-type display terminal 10 (AR glasses) (see Figure 1 (A)).

[0056] Distance a and direction θ of the glasses-type display terminal 10 (AR glasses) as seen from the imaging communication terminal 20 (smartphone) １ is determined in advance based on the physique of the worker Hw (nurse) (typically, the AR glasses 10 are located at a certain distance a above the smartphone 20 which is attached (fixed) to the chest of the worker Hw by a neck mount).

[0057] In step S1 of FIG. 5, the imaging communication terminal 20 receives the distance a and the direction (angle) θ to the glass-type display terminal 10 input in response to a user operation on the touch panel display unit, for example. １ is acquired (step S1).

[0058] In step S2, the imaging communication terminal 20 acquires inertial measurement data (tilts (angles and angular velocities) in three orthogonal axis directions of the glass-type display terminal 10) detected by the inertial measurement unit (A: first inertial measurement unit) 12 of the glass-type display terminal 10.

[0059] In step S3, the imaging communication terminal 20 acquires inertial measurement data (tilts (angles and angular velocities) of the imaging communication terminal 20 in three orthogonal axis directions) detected by the inertial measurement unit (B: second inertial measurement unit) 22.

[0060] In step S4, the imaging communication terminal 20 acquires a target image (video) PV captured by the camera unit 21 with the observation target K at the center (see FIG. 1A).

[0061] In step S5, the imaging communication terminal 20 transmits the target image PV acquired in step S4 to the instruction terminal (second terminal) 30.

[0062] In step P1 , the instruction terminal 30 receives the target image PV transmitted from the imaging communication terminal 20 .

[0063] In step P2, the instruction terminal 30 displays the target image PV transmitted from the imaging communication terminal 20 on the display unit (see FIG. 1B).

[0064] In step P3, the instruction terminal 30 synthesizes and displays a 3D marker M on the target image PV being displayed in response to operation of the input unit (31) by the instructor Hi (see Figures 1(B), 2, and 7(A)).

[0065] In step P4, in response to the operation of the input unit (31) by the instructor Hi, the instruction terminal 30 moves and rotates the 3D marker M displayed on the target image PV in step P3 so that the 3D marker M indicates the desired direction in order to instruct the worker Hw on how to act with respect to the observation target K (for example, how to change the posture of the observation target K).

[0066] In step P5, the instruction terminal 30 transmits to the imaging communication terminal 30 the display (numerical value) data of the 3D marker M (instruction content) that is moved and rotated and displayed on the target image PV in step P4.

[0067] In step S6 , the imaging communication terminal 20 receives the display (numerical value) data of the 3D marker M (instruction content) transmitted from the instruction terminal 30 .

[0068] In step S7, the imaging communication terminal 20 calculates the difference between the tilt data of the glass type display terminal 10 acquired in step S2 and the tilt data of the imaging communication terminal 20 acquired in step S3 by the calculation unit 23, and calculates the elevation / depression angle difference θ between the glass type display terminal 10 and the imaging communication terminal 20. ２ Calculate and obtain.

[0069] In step S8, the imaging communication terminal 20 calculates the elevation / depression angle difference θ between the glass type display terminal 10 and the imaging communication terminal 20 calculated in step S7 by the calculation unit 23. ２ and the distance a and direction θ of the glass type display terminal 10 as seen from the imaging communication terminal 20 acquired in step S1. １ Based on this, as shown in FIG. 6, the distance c from the glass type display terminal 10 to the observation target K is calculated and acquired according to the following formula. １ , θ ２ is 0<θ １ <180°, 0<θ ２ <180°.

[0070] b = a sin θ １ c = b / sinθ ２

[0071] In step S9, the imaging communication terminal 20 uses the rendering unit 24 to calculate the elevation / depression angle difference θ between the glass type display terminal 10 and the imaging communication terminal 20 acquired in step S7 in response to the 3D marker M (instruction content) that instructs the worker Hw to take an action and that was received in step S6. ２ The shooting angle and shooting distance are determined using the distance c from the glass-type display terminal 10 to the object of observation K obtained in step S8, and the 3D marker M is rendered into a corrected 3D marker Mr in which the positional deviation (difference in elevation and depression angles) between the imaging position (corresponding to the viewpoint of the instructor Hi) on the imaging communication terminal 20 and the viewpoint of the worker Hw on the glass-type display terminal 10 is corrected (3D rendering).

[0072] In step S10, the imaging communication terminal 20 transmits the corrected 3D marker Mr (instruction content) of the rendering result acquired in step S9 to the glass-type display terminal 10, displays it on the display unit 11, and superimposes it on the field of view image FV of the glass-type display terminal 10 (see Figures 1(A), 2, and 7(B)).

[0073] As a result, even if the imaging position of the imaging communication terminal 20 (smartphone) corresponding to the viewpoint of the instructor Hi (doctor) with respect to the observation object K is far from the viewpoint of the worker Hw (nurse) through the glass-type display terminal 10 (AR glasses), the instructions (3D marker M) for the action from the instructor Hi (doctor) to the worker Hw (nurse) based on the target image PV can be converted and displayed as instructions (corrected 3D marker Mr) that do not look strange in the field of view image FV seen from the glass-type display terminal 10 (AR glasses).

[0074] Therefore, it is possible to eliminate the risk of causing the worker Hw to take an erroneous action or imposing a cognitive load on the worker Hw to compensate for the difference in viewpoint between the worker Hw and the instructor Hi.

[0075] (Summary of the embodiment) According to the remote work support system 1 relating to the information presentation system of the embodiment, the worker Hw (nurse) views the observation target K (work target) through a glass-type display terminal 10 (AR glasses), and transfers a target image PV including the observation target K captured by an imaging communication terminal (first terminal) 20 (smartphone with a camera) worn on the chest to an instruction terminal (second terminal) 30 (PC) of a remote instructor Hi (doctor) for display.

[0076] When the instructor Hi displays a 3D marker M (instruction content) for instructing the worker Hw to take an action on the target image PV displayed on the instruction terminal 30 (PC) and operates it to instruct a desired position and direction, the imaging communication terminal 20 (smartphone) calculates the 3D marker M received from the instruction terminal 30 (PC) based on the positional relationship (angle θ １ , θ ２, distances a, b, c), the positional deviation between the imaging position on the imaging communication terminal 20 (smartphone) corresponding to the viewpoint of the instructor Hi and the viewpoint of the worker Hw on the glass-type display terminal 10 (AR glasses) is corrected and converted into a corrected 3D marker Mr (corrected instruction content), and the corrected 3D marker Mr is displayed superimposed on the field of view image FV of the glass-type display terminal 10 (AR glasses).

[0077] Therefore, even if the position where the worker Hw views the work object K through the AR glasses 10 and the position where the instructor Hi captures the image of the work object K are separated, it is possible to display the contents of the instructions given by the instructor Hi on the AR glasses 10 without any sense of incongruity.

[0078] The methods described in the embodiments may be stored as a program (software means) that can be executed by a computer on a recording medium such as a magnetic disk (e.g., a floppy disk, a hard disk, etc.), an optical disk (e.g., a CD-ROM, a DVD, an MO, etc.), or a semiconductor memory (e.g., a ROM, a RAM, a flash memory, etc.), or may be transmitted and distributed via a communication medium. The program stored on the medium also includes a configuration program that configures the software means (including not only executable programs but also tables and data structures) that the computer executes. The computer that implements this system reads the program stored on the recording medium and, in some cases, configures the software means using the configuration program, and executes the above-described processing by having the operation controlled by this software means. The recording medium may be a storage medium, such as a magnetic disk or semiconductor memory, installed inside the computer or in a device connected via a network, and may be distributed for distribution.

[0079] The present invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in appropriate combinations, in which case the combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by combining selected elements from the disclosed elements. For example, if the problem can be solved and the desired effect can be obtained even if some elements are deleted from all elements shown in the embodiments, the configuration from which these elements are deleted can be extracted as an invention.

[0080] DESCRIPTION OF SYMBOLS 1 ... Remote work support system (information presentation system) K ... Observation object (work object) Hw ... Worker (nurse) 10 ... Glasses-type display terminal (AR glasses) 11 ... Display unit 12 ... Inertial measurement unit (A: first inertial measurement unit) 20 ... Imaging communication terminal (first terminal) (smartphone with camera) 21 ... Camera unit (imaging unit) 22 ... Inertial measurement unit (B: second inertial measurement unit) 23 ... Calculation unit 24 ... Rendering unit FV ... Field of view image Mr ... Corrected 3D marker (correction instruction content) N ... Communication line Hi ... Instructor (doctor) 30 ... Instruction terminal (second terminal) (PC) 31 ... Input device (input unit) PV ... Target image (video) M ... 3D marker (instruction content) 41 ... Control circuit (control unit) 42 ... Memory 42a ... Control program area 42b ... Management information area 43 ... Storage device 44P ... Imaging module 44 ... User interface 45 ... Wired communication module 46 ... Wireless communication module

Claims

1. An information presentation system comprising: a glass-type display terminal; a first terminal connected to the glass-type display terminal and capturing an image of an observation target included in the field of view of the glass-type display terminal at a position away from the glass-type display terminal; and a second terminal having a communication function with the first terminal and displaying a target image including the observation target captured by the first terminal, wherein the first terminal comprises a control unit that performs the following processes: a process of receiving instructions for a 3D model displayed on the target image in response to a user operation of the second terminal; a process of converting the received instructions for the 3D model into instructions in a field of view image seen from the glass-type display terminal based on the positional relationship between the glass-type display terminal and the first terminal relative to the observation target; and a process of superimposing the converted instructions for the 3D model on the field of view image of the glass-type display terminal.

2. The information presentation system according to claim 1, wherein the positional relationship between the glass-type display terminal and the first terminal relative to the object of observation includes a difference in elevation / depression angle between the glass-type display terminal and the first terminal, and a distance from the glass-type display terminal.

3. The information presentation system according to claim 2, wherein the glass-type display terminal comprises a first inertial measurement unit that detects tilt in three orthogonal axis directions, the first terminal comprises a second inertial measurement unit that detects tilt in three orthogonal axis directions, and the difference in elevation / depression angle between the glass-type display terminal and the first terminal is calculated based on the difference between the tilt of the glass-type display terminal detected by the first inertial measurement unit and the tilt of the first terminal detected by the second inertial measurement unit.

4. The information presentation system according to claim 3, wherein the object to be observed is adjusted so as to be captured at the center of both the field of view image of the glass-type display terminal and the object image captured by the first terminal, and the distance between the object to be observed and the glass-type display terminal is calculated based on the difference in elevation / depression angle between the glass-type display terminal and the first terminal, and the distance and direction of the glass-type display terminal as seen from the first terminal.

5. The information presentation system according to claim 4, wherein the first terminal converts the instruction content of the 3D model received from the second terminal into instruction content in the field of view image seen from the glass-type display terminal by 3D rendering according to the difference in elevation and depression angles between the glass-type display terminal and the first terminal relative to the object of observation and the distance from the glass-type display terminal.

6. An information presentation system according to claim 1 or claim 2, wherein the instruction content of the 3D model is a 3D marker.

7. An information presentation method in which a control unit of an information presentation system comprising a glass-type display terminal, a first terminal connected to the glass-type display terminal and capturing an image of an observation target included in the field of view of the glass-type display terminal at a position away from the glass-type display terminal, and a second terminal having a communication function with the first terminal and displaying an object image including the observation target captured by the first terminal, receives instructions of a 3D model displayed on the object image in response to a user operation of the second terminal, converts the received instructions of the 3D model into instructions in a field of view seen from the glass-type display terminal based on the positional relationship between the glass-type display terminal and the first terminal in relation to the observation target, and displays the converted instructions of the 3D model superimposed on the field of view image of the glass-type display terminal.

8. A program that causes a control unit of an information presentation system comprising a glasses-type display terminal, a first terminal connected to the glasses-type display terminal and capturing an image of an observation target included in the field of view of the glasses-type display terminal at a position away from the glasses-type display terminal, and a second terminal having a communication function with the first terminal and displaying a target image including the observation target captured by the first terminal, to perform the following processes: receiving instructions for a 3D model displayed on the target image in response to a user operation of the second terminal; converting the received instructions for the 3D model into instructions in a field of view seen from the glasses-type display terminal based on the positional relationship between the glasses-type display terminal and the first terminal relative to the observation target; and displaying the converted instructions for the 3D model superimposed on the field of view image of the glasses-type display terminal.

Images