Projection device, projection method and program

Abstract

To provide a projection device, a projection method and a program capable of efficiently and effectively executing education about a structure.SOLUTION: A projection device executes training about an actual structure by projecting virtual information to a real space in which the actual structure exists. A three-dimensional model related to the actual structure is superimposed and projected to the actual structure, the avatar is projected to the real space, and the avatar has a content projection part for executing an explanation using attribute information of the three-dimensional model according to a scenario.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a projection device, a projection method, and a program, and more particularly to a technique for efficiently and effectively implementing education regarding structures. [Background technology]

[0002] Engineers are being trained to maintain and manage structures such as highways and viaducts as they age. In recent years, wearable devices that can project virtual information such as internal structures at life-size onto real structures (hereafter referred to as real structures) have been used in educational settings. Such devices can be used for a variety of purposes, including not only education but also the investigation, design, and maintenance of structures.

[0003] Patent documents 1 to 3 describe a system that uses a head-mounted see-through display to superimpose an image showing the internal structure of a real structure, such as a tunnel, a bridge, or a building, onto the real structure.

[0004] Furthermore, Patent Documents 4 and 5 describe systems that implement various types of education using avatars that are superimposed and displayed in real or virtual space. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent No. 6438995 [Patent Document 2] Patent No. 3653196 [Patent Document 3] Patent No. 6695529 [Patent Document 4] Patent No. 7367632 [Patent Document 5] JP 2002-062791 A Summary of the Invention [Problem to be solved by the invention]

[0006] In recent years, it has become necessary for educational institutions to respond to social issues such as infectious disease countermeasures, labor shortages, and the realization of work-life balance. One solution is to provide a system that allows students to study alone without having to come into direct contact with a teacher.

[0007] On the other hand, in the education of structures, emphasis has traditionally been placed on having instructors give instruction in front of actual structures. There is a demand for a system that allows for self-study without compromising the educational effect of such hands-on experience.

[0008] The present invention has been made to solve such problems, and has an object to provide a projection device, a projection method, and a program that can efficiently and effectively carry out education about structures. [Means for solving the problem]

[0009] In one form of the present invention, a projection device is a projection device that conducts training on a real structure by projecting virtual information into a real space in which the real structure exists, and has a content projection unit that superimposes and projects a three-dimensional model related to the real structure onto the real structure, projects an avatar into the real space, identifies a location to be explained by referring to attribute information of the three-dimensional model, and causes the avatar to explain the location to be explained in the vicinity of the location to be explained. In one form of the present invention, the content projection unit recognizes a question posed by a user, identifies the part to be explained that is related to the question by referring to attribute information of the three-dimensional model, and causes the avatar to provide an explanation regarding the part to be explained. In one embodiment of the present invention, the content projection unit recognizes the user's position or viewpoint, and when a relationship between the user's position or viewpoint and the explanation target location satisfies a predetermined condition, prompts the user to move or look. In one form of the present invention, the content projection unit recognizes the user's position or viewpoint, recognizes dangerous areas or dangerous objects in the real space, and warns the user when the relationship between the user's position and viewpoint and the dangerous areas or dangerous objects satisfies predetermined conditions. In one form of the present invention, the projection device further has an evaluation unit, wherein the content projection unit recognizes the user's position or viewpoint, and the evaluation unit generates an evaluation point when the relationship between the user's position or viewpoint and the portion to be explained satisfies a predetermined condition. In one form of the present invention, a projection device conducts training on a real structure by projecting virtual information into a real space in which a model of the real structure exists, and has a content projection unit that superimposes and projects a three-dimensional model related to the real structure onto the model, projects an avatar into the real space, identifies a location to be explained by referring to attribute information of the three-dimensional model, and causes the avatar to explain the location to be explained in the vicinity of the location to be explained. In one aspect of the present invention, the content projection unit changes a scale of the avatar in accordance with a scale of the model. In one form of the present invention, a projection device conducts training on a real structure by projecting a virtual structure created based on the real structure and virtual information into a real space other than the real space in which the real structure exists, and has a content projection unit that superimposes and projects a three-dimensional model related to the real structure onto the virtual structure, projects an avatar into the virtual space, identifies a location to be explained by referring to attribute information of the three-dimensional model, and has the avatar explain the location to be explained in the vicinity of the location to be explained. In one form of the present invention, a projection method is a projection method for conducting training on a real structure by projecting virtual information into a real space in which the real structure exists, and includes the steps of superimposing and projecting a three-dimensional model related to the real structure onto the real structure and projecting an avatar into the real space, identifying a location to be explained by referring to attribute information of the three-dimensional model, and having the avatar explain the location to be explained in the vicinity of the location to be explained. In one form of the present invention, a projection method is a projection method for conducting training on a real structure by projecting virtual information into a real space in which a model of the real structure exists, the projection method including the steps of superimposing and projecting a three-dimensional model related to the real structure onto the model and projecting an avatar into the real space, identifying a location to be explained by referring to attribute information of the three-dimensional model, and having the avatar explain the location to be explained in the vicinity of the location to be explained. In one form of the present invention, a projection method is a projection method for conducting training on a real structure by projecting a virtual structure created based on the real structure and virtual information into another real space different from the real space in which the real structure exists, the projection method including the steps of superimposing and projecting a three-dimensional model related to the real structure onto the virtual structure and projecting an avatar into the virtual space, identifying a location to be explained by referring to attribute information of the three-dimensional model, and having the avatar explain the location to be explained in the vicinity of the location to be explained. In one aspect of the present invention, a program causes a computer to execute the above method. Effect of the Invention

[0010] The present invention provides a projection device, a projection method, and a program that are capable of efficiently and effectively providing education about structures. [Brief description of the drawings]

[0011] [Figure 1] 1 is a diagram showing an example of the external appearance of a projection device 1. FIG. [Diagram 2] 1 is a diagram illustrating an example of a hardware configuration of a projection device 1. FIG. [Diagram 3] 2 is a diagram illustrating an example of a functional configuration of a projection device 1. FIG. [Figure 4A] FIG. 1 is a diagram illustrating an example of content projection. [Figure 4B] FIG. 1 is a diagram illustrating an example of content projection. [Diagram 5] 4 is a flowchart showing an example of an operation of the projection device 1. [Figure 6] 4 is a flowchart showing an example of an operation of the projection device 1. [Figure 7] 4 is a flowchart showing an example of an operation of the projection device 1. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] FIG. 1 shows an example of the appearance of an HMD type projection device 1. The projection device 1 is an information processing device having a three-dimensional space sensing device such as a camera or a depth sensor, and a display. The projection device 1 is typically an HMD (Head Mounted Display) device equipped with an optically transmissive or video transmissive display (hereinafter simply referred to as a transmissive display), such as Hololens (registered trademark) or Apple Vision Pro (registered trademark). Note that the projection device 1 is not limited to an HMD type, and may be, for example, a smartphone, a tablet terminal, a PC (Personal Computer), or the like.

[0013] 2 is a diagram showing an example of a hardware configuration of the projection device 1. The projection device 1 includes a camera 111, a depth sensor 112, a speaker 12, a display 13, a CPU 14, a volatile memory 15, a non-volatile memory 16, interfaces 171 to 175, a bus 18, and a communication device 19. If necessary, a microphone (not shown) may be included.

[0014] A CPU 14 (Central Processing Unit) reads out programs and data stored in the non-volatile memory 16 or the volatile memory 15 via a bus 18, and executes information processing according to the programs to realize specific functions.

[0015] The non-volatile memory 16 is a storage device that retains its stored state even when the projection device 1 is not operating, and is, for example, a hard disk or SSD. In general, the programs and data stored in the non-volatile memory 16 are expanded in the volatile memory 15 when the programs are executed.

[0016] The volatile memory 15 is a storage device that stores programs and data expanded from the non-volatile memory 16, temporary calculation data, data input / output between external devices via the communication device 19, and the like.

[0017] The camera 111 is a photographing device that captures image data of a real space including a real structure. The image data is passed to the CPU 14 via the interface 171.

[0018] The depth sensor 112 scans a real space including a real structure to obtain 3D point cloud data showing its shape. The depth sensor 112 typically irradiates a measurement electromagnetic wave such as infrared light to multiple measurement points on the structure and measures the distance to each measurement point by detecting the reflected wave. Alternatively, the depth sensor 112 may generate 3D information based on image data acquired by multiple cameras such as a stereo camera. The 3D point cloud data is passed to the CPU 14 via the interface 172.

[0019] The speaker 12 reproduces sound. The sound is assumed to be stored in the non-volatile memory 16 or the volatile memory 15. These data are passed from the CPU 14 to the speaker 12 via the interface 173.

[0020] The display 13 is a display device that allows the user to simultaneously view the real space including real structures and virtual information. This can be realized, for example, by displaying the virtual information on an optical transmission type display, or by superimposing an image of the real space captured by a camera on the virtual information on a video transmission type display. The virtual information includes, for example, objects such as models, images, drawings, figures, and texts showing the internal structure of the real structure, and an avatar acting as a lecturer. The virtual information is stored in the non-volatile memory 16 or the volatile memory 15. These data are passed from the CPU 14 to the display 13 via the interface 174.

[0021] The communication device 19 acquires the transmission data output by the CPU 14 via the interface 175 and transmits it to an external device. Also, the communication device 19 passes data received from an external device to the CPU 14 via the interface 175.

[0022] 3 is a diagram showing an example of the functional configuration of the projection device 1. The projection device 1 has a content storage unit 101, a content projection unit 103, and an evaluation unit 105 as functions for implementing an educational program related to structures.

[0023] The content storage unit 101 stores the content used in the educational program. An example of the content is shown below.

[0024] Lecturer Avatar. The lecturer avatar is a virtual character who plays the role of a lecturer. Typically, it is a three-dimensional model that imitates a human as shown in FIG. 4A. This three-dimensional model has movable joints and can perform a predetermined action according to the commands of a scenario described later. In addition, the avatar can have a sound source, and the position of the sound source generated from the avatar moves according to the position of the avatar, so that it is possible to have a mechanism that allows intuitive understanding of where the avatar is speaking. Note that the avatar does not need to be humanoid, and may be, for example, an animal, a robot, or any other shape. Note that even in the latter case, it is preferable that the avatar has a function for maintaining the educational effect, such as pointing to a specific place.

[0025] Models and objects. 3D models showing the internal structure of real structures, various 3D models related to structures, and 2D objects such as images, drawings, figures, and text. They are projected onto the display 13 according to the coordinates and timing specified by the scenario described below.

[0026] Scenario. Data that describes the procedure for proceeding with an educational program by combining one or more commands. The commands define, for example, the coordinates at which the instructor avatar should be displayed, the actions the instructor avatar should perform, the audio to be played (narration, etc.), the coordinates at which models and objects should be displayed, etc., in association with trigger events, etc. Typical examples of commands are shown below. When the user issues a command to start an educational program (trigger), the instructor avatar is projected at a specified distance in front of the user according to the user's position and orientation, and performs specified gestures and speaking actions. At the same time, a specified narration is played. Once the narration has ended and a specified amount of time has elapsed (trigger), the instructor avatar moves close to a specified component of the real structure and points out the specified component. If the user's position or orientation is in the opposite direction to the structure being pointed out at this time, the instructor avatar can be displayed according to the user's orientation and position, and can move towards the structure while pointing out. At the same time, the specified narration is played and a 3D model showing the internal structure of the component is superimposed on the real structure. After the narration has finished and a certain amount of time has passed (trigger), a question to check the user's comprehension and buttons to select the answer are displayed. When the user selects one of the buttons (trigger), text indicating whether it is correct or incorrect is displayed.

[0027] The content projection unit 103 reads out a scenario stored in the content storage unit 101. It monitors the occurrence of a trigger described in the scenario, and executes a command associated with the trigger that has occurred.

[0028] Furthermore, the content projection unit 103 can project virtual information on the display 13 according to the scenario, and can also reproduce a narration written in advance in the scenario as sound. The sound can be reproduced, for example, by providing text data of the narration to a known voice synthesis tool.

[0029] Furthermore, the content projection unit 103 can detect user actions during the execution of the educational program. User actions include, for example, designating a projected object with a fingertip, catching a specific object with the line of sight, recognizing words spoken by the user, etc. These actions can be detected by known functions that the projection device 1 is equipped with as standard.

[0030] The evaluation unit 105 monitors the behavior of the user while watching the educational program, and performs evaluation and advice, etc. The target user behavior includes, for example, whether or not a question presented in the educational program is answered, the content of the answer, the user's position during the lesson, the direction of movement, the direction of gaze, etc. When the evaluation unit 105 detects such behavior, it calculates an evaluation index according to a predetermined rule. Typically, it calculates a score. Alternatively, it gives advice or a warning at a predetermined timing.

[0031] [Example 1] 5 is a flowchart showing a first operational example of the projection device 1. The first operational example shows a flow in which a user takes an educational program in front of an actual structure such as a tunnel or a bridge.

[0032] S101: Start-up and calibration of projection device 1 (mapping with real structure) The user moves in advance close to a real structure (tunnel, bridge, etc.) that is the target of the educational program, and starts up the projection device 1. The projection device 1 performs mapping (association) between the real space and the MR space (a virtual space used by the projection device 1 to place objects, etc.) by, for example, a method described in Patent Document 3 (Patent No. 6695529). This makes it possible to display virtual information superimposed on the real structure.

[0033] S102: Start of Education Program The projection device 1 receives a request from a user to start an educational program. The content projection unit 103 reads a scenario stored in the content storage unit 101, and projects models and objects, plays audio, and so on, in accordance with the scenario.

[0034] The effectiveness of educational scenarios regarding structures can be increased by incorporating the following features, for example:

[0035] <Avatar instructors will provide on-site explanations in front of the actual structures> As shown in FIG. 4A, an avatar instructor is projected near a real structure, and plays back an explanatory voice (e.g., "This is the cylindrical frame") regarding a part to be explained in the real structure while performing actions such as pointing to the part to be explained in the real structure with hand gestures. At this time, it is preferable that a 3D model of the structure is superimposed and projected on the real structure. The content projection unit 103 specifies the part to be explained in the 3D model by referring to attribute information added to the components of the 3D model. The position where the specified part to be explained is located is given to the avatar instructor as the position to be pointed out, thereby realizing the above-mentioned action.

[0036] If the student user is looking in a direction different from the target location, the avatar teacher is first projected into the user's field of view and prompts the user to look at the target location (for example, saying "Look behind you. That's the cylindrical frame" and making a gesture to point behind), and then the avatar teacher moves around the user to an appropriate position, for example, near the target location. When it detects that the target location has come into the user's field of view, it continues the explanation.

[0037] If the distance from the user's current location to the location to be explained exceeds a certain threshold, the avatar instructor is first projected within the user's field of vision near the user and prompts the user to move to the location to be explained (for example, by pointing to the location to be explained with a hand gesture and saying, "That's the cylindrical frame. Let's move closer."), and then the avatar instructor moves close to the location to be explained. If it detects that the user has moved within a certain distance from the location to be explained, the instructor will continue the explanation.

[0038] If the location to be explained is in a position that the user cannot move to (for example, if it is in a high position, such as the underside of a bridge deck), the avatar instructor is first projected within the user's field of vision and then moves to the vicinity of the location to be explained by moving through the air. After moving, the instructor gives an explanation (for example, "There is a defect on the underside of the deck") while pointing to the location to be explained (the defect on the underside of the deck) with hand gestures.

[0039] In this way, users can receive instruction from an avatar instructor while visually viewing real structures, linking knowledge with reality and enabling effective learning.

[0040] <Overlaying a model related to the actual structure> When an avatar instructor gives a lecture on a real structure, a 3D model related to the structure and objects such as drawings, photographs, and text are superimposed on the real structure. For example, as shown in FIG. 4A, a 3D model showing the internal structure of the real structure (in this example, the arrangement of rebars) is superimposed. Note that a technical method for superimposing a model or object on a real structure is described in, for example, Patent Document 3 and is publicly known, so a description thereof will be omitted.

[0041] Preferably, information required for learning can be added in advance as attribute data to the model or object data to be projected. Typically, CIM (Construction Information Modeling / Management: a 3D model with attribute data such as names, costs, and other management information of components added) data can be used.

[0042] By using attribute data, it is possible to create various interactive educational programs. For example, when a question from a student user (e.g., "What is a cylindrical formwork?") is recognized, the avatar instructor searches for the existence of a model or object that contains information related to the question as attribute data (e.g., the component name "cylindrical formwork"). If such a model or object exists, the avatar instructor recognizes the model or object to which the attribute data is attached as the part to be explained and provides the necessary explanation. For example, the instructor may play an explanatory voice (e.g., "This is a cylindrical formwork") about the part to be explained, while using actions such as pointing to the location of the part to be explained (e.g., a 3D model of a cylindrical formwork) with hand gestures.

[0043] <An avatar answers the user's questions> The speech of the student user is picked up by a microphone. The speech is input to a speech recognition engine, which outputs the speech content as text. If the speech content is a question, the question is input to an answer generation engine, which generates an answer to the question and outputs it as text, audio, an image, or a video. The avatar instructor outputs the answer by synthetic voice or the like. As described above, an explanatory audio regarding the part to be explained may be played back, with actions such as pointing to the location of the part to be explained by hand gestures. Related images or videos may be projected.

[0044] The voice recognition engine and answer generation engine can use known technologies. For example, an inference algorithm constructed based on a rule base such as an expert system, or an inference algorithm constructed by machine learning such as deep learning can be adopted as the answer generation engine. This also includes generation AI that can generate new text, voice, images, etc. based on learned knowledge.

[0045] The voice recognition engine and the answer generation engine may be provided by a third party as a service available via a communication network. In this case, the content projection unit 103 communicates with the external service via the communication device 19 and uses the voice recognition engine and the answer generation engine.

[0046] <Warn the user of the danger> When an educational program is implemented at a site where a real structure exists, there is a possibility that a student user may enter a dangerous area or come into contact with a dangerous object. To prevent this, the content projection unit 103 can issue a warning to the user so that the user can avoid the danger.

[0047] When a danger area exists in the real space, for example, as shown in Fig. 4B, a color cone (registered trademark) and cone bar are projected around the danger area to visually warn the user not to enter the danger area. Alternatively, when the user's current location is acquired and the current location satisfies a predetermined condition, for example, when the current location approaches the danger area beyond a predetermined threshold, a warning is given by displaying text or images, playing audio, etc.

[0048] If there is a dangerous object in the real space (for example, a gutter, a protrusion on the ground, etc.), if the user approaches the dangerous object beyond a predetermined threshold, a warning is output in the same way. And / or, if the direction of the user's line of sight is determined, and the direction of the line of sight deviates from the position of the dangerous object by a certain amount or more, in other words, if it is considered that the user is not visually recognizing the dangerous object, the content projection unit 103 issues a warning by displaying text or an image, playing audio, etc.

[0049] The locations of dangerous areas and dangerous objects may be defined in advance, or may be automatically recognized from camera footage using techniques such as machine learning.

[0050] S103: Testing The content projection unit 103 presents the user with questions related to previous explanations (e.g., "Where is the cylindrical formwork?", "Where is the most stressed area on the underside of the deck?"). The questions may be played back as synthetic voice, or text may be projected.

[0051] The content projection unit 103 acquires an answer to the question from the user. For example, the user gives an answer by pointing to a specific location of a real structure. For example, the content projection unit 103 can recognize and acquire as an answer a location pointed to by the user's hand, a location gazed at by the user (where the user's gaze is fixed for a certain period of time or more), or a location pointed to by the user using a pointer controlled by eye tracking. Alternatively, the location serving as the answer may be specified using any other known technology. The content projection unit 103 identifies the location where the user has answered as coordinates on the real structure (real coordinates), converts the real coordinates into the coordinate system of a three-dimensional model of the structure (model coordinates), and identifies the component located at the model coordinates. In this way, the component pointed to by the user is recognized.

[0052] If the component name recognized as the answer matches a predefined correct answer, the content projection unit 103 presents the user with feedback indicating that the answer is correct. If they do not match, the content projection unit 103 presents feedback indicating that the answer is incorrect.

[0053] S104: Conducting evaluation The evaluation unit 105 acquires the test result (correct or incorrect) in step S103 from the content projection unit 103 and converts it into a score according to a predetermined rule. For example, a rule such as +n points for a correct answer and 0 points for an incorrect answer can be defined in advance. This makes it possible to quantify the level of understanding of the user, who is a student.

[0054] In addition, the evaluation unit 105 can evaluate whether the student user was concentrating on the study and actively participating by tracing the user's position and tracking the user's gaze while the avatar instructor is giving an explanation.

[0055] For example, if it is recognized that the user has been looking at the part to be explained for a certain period of time or more (the gaze has remained still) in a certain scene in the scenario, a certain number of points will be added. This is because it is assumed that the user has been concentrating on learning. On the other hand, if it is recognized that the user has been looking at a part other than the part to be explained for a certain period of time or more, a certain number of points will be subtracted. Also, if the user moves close to the part to be explained or a position specified by the avatar instructor, in other words, if the distance between the user's position and the part to be explained, etc., is within a certain distance, a certain number of points will be added. This is because it is assumed that the user has been actively participating in the learning.

[0056] The evaluation unit 105 tallys up and outputs the evaluation results of the level of understanding, the level of concentration, and the level of enthusiasm. For example, the evaluation results can be displayed in the form of numbers or graphs, and output as a report.

[0057] [Example 2] 6 is a flowchart showing an operation example 2 of the projection device 1. The operation example 2 shows a flow in which a user takes an educational program in front of a model of a real structure. For example, when a user cannot directly go to the site where the real structure is located, it is assumed that the user takes the educational program indoors. This is suitable for training on real structures that are difficult to visit, such as the inside of a tunnel or a slope.

[0058] In addition, in Operation Example 2, differences from Operation Example 1 will be mainly described. Descriptions of operations that overlap with Operation Example 1 will be omitted as appropriate.

[0059] S201: Start-up and calibration of projector 1 (mapping with the model) The user moves in advance close to a model of a real structure that is the target of the educational program, and starts up the projection device 1. The projection device 1 performs mapping (association) between the real space and the MR space (a virtual space used by the projection device 1 when placing objects, etc.) by, for example, a method described in Patent Document 3 (Patent No. 6695529). This makes it possible to display virtual information superimposed on the model of the real structure.

[0060] S202: Start of Education Program As in S102, the projection device 1 executes the education program by projecting models and objects, playing back audio, and so on, according to a scenario.

[0061] In the second operational example, the following operations are further carried out.

[0062] <Project the avatar instructor according to the scale of the model> In a scene where an avatar instructor points to a part of a model to be explained, if an avatar instructor of the same size as in Operation Example 1 is projected, there is a problem that it becomes unclear which part the instructor is pointing to. Therefore, in Operation Example 2, the avatar instructor is displayed reduced or enlarged according to the scale of the model. For example, if the model is a reduced model of a tunnel, the avatar instructor is displayed reduced according to the scale of the model. This makes it clear which part the avatar instructor is pointing to.

[0063] The scale of the avatar instructor may be changeable according to the scenario. For example, when the instructor is giving a general explanation to the user, the instructor is projected at life-size, and when the instructor is pointing to a part of a model to be explained, the scale of the instructor is changed according to the scale of the model.

[0064] S203: Testing As in S103, the content projection unit 103 presents a question to the user and acquires the answer.

[0065] S204: Conducting evaluation As in S104, the evaluation unit 105 evaluates the user's level of understanding, concentration, enthusiasm, and the like.

[0066] [Example 3] 7 is a flowchart showing an operation example 3 of the projection device 1. The operation example 3 shows a flow in which a user takes an educational program in front of a virtual structure projected in real space. For example, when it is desired to present to a user the occurrence of a deformation in a real structure or the process of collapse or destruction using animation or the like, changing the virtual structure using animation has an advantage of realism and enhancing the educational effect, rather than superimposing the animation on the real structure or its model.

[0067] The virtual structure is expected to be a so-called digital twin that is created in advance by scanning the real structure, etc. The educational program can be typically conducted in a real space other than the real space where the real structure exists, such as a conference room.

[0068] In addition, in the operation example 3, differences from the operation example 1 will be mainly described. Descriptions of operations that overlap with the operation example 1 will be omitted as appropriate.

[0069] S301: Start-up and calibration of projection device 1 (mapping with virtual structure) A user starts the projection device 1. The projection device 1 projects a three-dimensional model of a virtual structure created in advance onto the display 13. In the operation example 3, the display 13 does not need to be a transmissive type, and a non-transmissive type makes it easier to obtain a sense of immersion.

[0070] The 3D model of the virtual structure may be, for example, a digital twin created by scanning the real structure. For example, a 3D model of the virtual structure can be created based on shape data of the real structure acquired by various measuring devices or photogrammetry data (a method of taking pictures of a subject from various angles and analyzing and integrating the digital images to create a three-dimensional 3DCG model) using SFM (Structure from Motion: a technology that restores the shape of an object from multiple photographs of the object).

[0071] The projection device 1 performs mapping (association) between the VR space of the virtual structure and the MR space (a virtual space used by the projection device 1 when arranging educational avatars, models, objects, etc.). This makes it possible to superimpose and display educational virtual information on the virtual structure.

[0072] S302: Start of Education Program As in S102, the projection device 1 executes the education program by projecting models and objects, playing back audio, and so on, according to a scenario.

[0073] When a virtual structure is projected at approximately the same scale as the real structure, the behavior of the avatar instructor can be controlled in the same manner as in Operation Example 1. On the other hand, when a virtual structure is projected at a scale different from that of the real structure, the behavior of the avatar instructor can be controlled in the same manner as in Operation Example 2.

[0074] S303: Conducting tests As in S103, the content projection unit 103 presents a question to the user and acquires the answer.

[0075] S304: Conducting evaluations As in S104, the evaluation unit 105 evaluates the user's level of understanding, concentration, enthusiasm, and the like.

[0076] The present invention is not limited to the above-described embodiment, and can be modified as appropriate without departing from the spirit and scope of the present invention. For example, in the above-described embodiment, an example was shown in which an avatar instructor was used to teach, but it is not necessary for an avatar instructor to appear in the scenario. For example, an educational program may be progressed only by projecting objects such as a pointer, images, text, etc., or by playing back audio.

[0077] In the above embodiment, each processing unit is implemented in the projection device 1, but the present invention is not limited to this, and some processing may be performed in an external information processing device, such as a server or a cloud computing environment. Specifically, the projection device 1 can read and use content data (scenarios, models, objects, avatars, etc.) stored in the external information processing device as appropriate. Processing with a relatively large processing load, such as a voice recognition engine and an answer generation engine, can be executed by an external information processing device via an API or the like.

[0078] Furthermore, each processing means constituting the present invention may be configured by hardware, or any process may be realized by making a CPU execute a computer program. Furthermore, the computer program may be stored and supplied to a computer using various types of temporary or non-temporary computer-readable media. A temporary computer-readable medium includes, for example, an electromagnetic signal supplied to a computer by wire or wirelessly. [Explanation of symbols]

[0079] 1 Projection device 111 Camera 112 Depth Sensor 12 speakers 13. Display 14 CPU 15 Volatile Memory 16 Non-volatile memory 171-175 Interface 18 Bus 19. Communications Equipment 101 Content storage unit 103 Content Projection Unit 105 Evaluation Department

Claims

1. A projection device for conducting training related to a real structure by projecting virtual information into a real space in which the real structure exists, comprising: A three-dimensional model related to the real structure is superimposed and projected onto the real structure, and an avatar is projected into the real space; Identifying an explanation target portion by referring to attribute information of the three-dimensional model; A content projection unit that causes the avatar to provide an explanation about the explanation target location in the vicinity of the explanation target location. Projection device.

2. The content projection unit includes: Recognize the questions the user asks, Identifying the explanation target portion related to the question by referring to attribute information of the three-dimensional model; The avatar is caused to provide an explanation regarding the portion to be explained.

2. The projection device of claim 1.

3. The content projection unit includes: Recognizing a position or viewpoint of the user; When a relationship between the position or viewpoint of the user and the explanation target portion satisfies a predetermined condition, the user is prompted to move or look at the explanation target portion.

2. The projection device of claim 1.

4. The content projection unit includes: Recognizing a position or viewpoint of the user; Recognizing a dangerous area or a dangerous object in the real space; When the relationship between the user's position and viewpoint and the dangerous area or dangerous object satisfies a predetermined condition, a warning is given to the user.

2. The projection device of claim 1.

5. Further comprising an evaluation unit, The content projection unit includes: Recognizing a position or viewpoint of the user; The evaluation unit is When the relationship between the user's position or viewpoint and the explanation target portion satisfies a predetermined condition, an evaluation point is generated.

2. The projection device of claim 1.

6. A projection device for conducting training on a real structure by projecting virtual information into a real space in which a model of the real structure exists, comprising: A three-dimensional model related to the real structure is superimposed and projected onto the model, and an avatar is projected onto the real space; Identifying an explanation target portion by referring to attribute information of the three-dimensional model; A content projection unit that causes the avatar to provide an explanation about the explanation target location in the vicinity of the explanation target location. Projection device.

7. The content projection unit includes: The scale of the avatar is changed according to the scale of the model.

7. The projection device according to claim 6.

8. 1. A projection device for carrying out training related to a real structure by projecting a virtual structure and virtual information created based on the real structure into a real space different from a real space in which the real structure exists, comprising: A three-dimensional model related to the real structure is superimposed and projected onto the virtual structure, and an avatar is projected into the virtual space; Identifying an explanation target portion by referring to attribute information of the three-dimensional model; A content projection unit that causes the avatar to provide an explanation about the explanation target location in the vicinity of the explanation target location. Projection device.

9. 1. A projection method for conducting training related to a real structure by projecting virtual information into a real space in which the real structure exists, comprising the steps of: projecting a three-dimensional model related to the real structure onto the real structure in a superimposed manner, and projecting an avatar into the real space; Identifying an explanation target portion by referring to attribute information of the three-dimensional model; and causing the avatar to provide an explanation regarding the portion to be explained in the vicinity of the portion to be explained. Projection method.

10. 1. A projection method for carrying out training on a real structure by projecting virtual information into a real space in which a model of the real structure exists, comprising the steps of: a step of superimposing and projecting a three-dimensional model related to the real structure onto the model and projecting an avatar into the real space; Identifying an explanation target portion by referring to attribute information of the three-dimensional model; and causing the avatar to provide an explanation regarding the portion to be explained in the vicinity of the portion to be explained. Projection method.

11. 1. A projection method for carrying out training related to a real structure by projecting a virtual structure and virtual information created based on the real structure into a real space different from a real space in which the real structure exists, comprising: a step of superimposing and projecting a three-dimensional model related to the real structure onto the virtual structure and projecting an avatar into the virtual space; Identifying an explanation target portion by referring to attribute information of the three-dimensional model; and causing the avatar to provide an explanation regarding the portion to be explained in the vicinity of the portion to be explained. Projection method.

12. A program for causing a computer to execute the method according to any one of claims 9 to 11.

Images