Navigation program, device, and system for selectively using multiple types of virtual environments
The system enhances user interaction with historical sites by integrating real-world guidance and virtual environment images, addressing limitations of conventional technologies to provide immersive and feedback-driven learning experiences.
Patent Information
- Application Number
- JP2023076335
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-02
- Publication Date
- 2024-11-15
AI Technical Summary
Conventional technologies fail to consider the real-world context of historical sites, lack user interaction with the real world, and do not account for user feedback or immersion in virtual environments, limiting intellectual engagement and knowledge accumulation.
A system that uses a mobile client terminal to provide guidance information superimposed on the real world and creates a virtual environment image based on current and past map information, allowing users to interact with historical sites and receive feedback-integrated guidance.
Enables users to navigate historical sites while considering real-world accessibility and providing immersive virtual experiences, deepening knowledge and facilitating intellectual production activities.
Smart Images

Figure 2024160864000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a navigation program, device, and system for selectively using a plurality of types of virtual environments. [Background technology]
[0002] The applicant has previously disclosed a technology for playing back environmental sounds in a specific way in a navigation device (Patent Document 1).The applicant has also disclosed a technology for classifying and configuring a database of images and related audio information in a four-dimensional layer structure consisting of three-dimensional space and one-dimensional time for use in a car navigation device or a walking navigation device (Patent Document 2).
[0003] Meanwhile, attempts have been made to reproduce exhibits such as museums and historical archives in a virtual space and provide them for viewing. For example, Patent Document 3 discloses a visitor guidance support device that includes a means for detecting information on a visitor's interaction with each exhibit during a tour in the real space that is performed prior to a tour in the virtual space, a means for determining an activity level that indicates the degree of interest the visitor has in each exhibit based on the detected information, a means for determining a guidance mode in the virtual space based on the determined activity level, and a means for guiding the visitor in the virtual space according to the determined guidance mode.
[0004] Patent Document 4 discloses a system for developing a virtual museum for visitors to historical sites and historic sites where buildings and the like no longer exist and the appearance of the past is no longer preserved. The system has an object output terminal that outputs objects indicated by information including at least object positions and output information based on the output information, a positioning means that acquires the terminal position of the object output terminal, and an output means that outputs the object from the object output terminal when the relationship between the terminal position and the object position is a predetermined relationship, and is characterized in that the system has a virtual space management means that defines the correspondence between the virtual space in which the objects are placed and the real space in which the virtual museum is actually developed.
[0005] Non-Patent Document 1 discloses the SpaceTag system for superimposing a virtual world on the real world, and is characterized by superimposing virtual information on the real world using only location information as a clue.
[0006] Patent document 5 discloses a display control method in which filtering is performed in accordance with a scenario mode, such as a sightseeing mode or a historical site exploration mode, in the user's field of view within a virtual space, and objects are rendered based on location information via a small, see-through head-mounted display.
[0007] Furthermore, Patent Document 6 discloses a system for providing a virtual experience of traveling together with a local companion who has an imaging device, and claims that the system enables a user to have a virtual travel experience by boarding a moving object that moves based on local images. [Prior art documents] [Patent documents]
[0008] [Patent Document 1] Patent No. 4791064 [Patent Document 2] Patent No. 5865708 [Patent Document 3] Japanese Patent Application Publication No. 10-050351 [Patent Document 4] JP 2004-145657 A [Patent Document 5] International Publication No. 2016 / 031358 [Patent Document 6] JP 2022-142296 A [Non-patent literature]
[0009] [Non-Patent Document 1] Hiroyuki Tarumi. "SpaceTag: A fusion of reality and virtuality that can be commercialized immediately." IPSJ SIG Human-Computer Interaction (HCI) 2002.76 (2002-HI-099) (2002): 23-30. Summary of the Invention [Problem to be solved by the invention]
[0010] However, the conventional technology does not take into consideration the situation of the historical site in the real world. The system of Patent Document 3 is merely for displaying the site in a virtual world, and does not allow visitors to interact with the historical site in the real world.
[0011] Furthermore, the system of Patent Document 4 aims to virtually reproduce relics that do not exist in the real world, but does not take into consideration feedback from viewers regarding the virtual relics. Patent Document 4 also describes setting and warning the presence of areas that are off-limits in the real world, but does not consider how to prevent the warning from ruining the viewer's enjoyment. Furthermore, when showing a virtual image to a user, no consideration is given to the state of the user in the real world.
[0012] The SpaceTag system in Non-Patent Document 1 has the characteristic of tagging the real world based on location information, but users are forced to act within the same geographical structure as the real world, and there is no suggestion as to how to handle locations that users can no longer easily reach in modern times due to, for example, the loss of buildings or changes in the terrain.Non-Patent Document 1 also claims that there is interactivity by both content providers and individual users being able to place SpaceTags, but it only suggests that the information flowing there is organized by "tuning the channel," and does not suggest efficient access to more useful information.
[0013] The technique in Patent Document 5 is characterized by seamless transitions between objects included in multiple display modes that display map data, etc., but does not take into account the interest of the user in visiting the historical site, nor does it take into account immersion in the virtual environment image.
[0014] Furthermore, the technology of Patent Document 6 is merely intended to provide a user with a virtual experience using a mobile device, accompanied by a stranger who is present at the location, and does not take into consideration the user themselves having an actual experience at the location.
[0015] Therefore, the conventional technologies described above cannot integrate the user's activities in both the real world and the virtual world, and there is a problem that the contribution to the user's intellectual productivity is limited. Therefore, in order to solve these problems, new technologies that further develop the technologies described in Patent Documents 1 and 2 are eagerly awaited. [Means for solving the problem]
[0016] In view of the above-mentioned problems, the present invention provides the following aspects.
[0017] Aspect 1. A module for receiving user location information from a mobile client terminal having a function for acquiring user location information; a module for storing current map information identifying a plurality of coordinates related to the historic site and past map information that differs from the current map information and identifies a plurality of coordinates related to the historic site, and for associating one or more coordinates in the past map information with one or more coordinates in the current map information; a module for determining whether the user is in the vicinity of any of the coordinates associated with a historic site based on the received user location information and the current map information or the past map information or both; a module for providing guidance information for guiding the user in the real world to the mobile client terminal based on the received position information of the user and the current map information when it is determined that the user is in the vicinity of any of the coordinates related to the historical site, and displaying the guidance information via a display means included in the mobile client terminal so as to be superimposed on the field of view of the user viewing the real world; a module for presenting a route traveled by the user in the real world based on the current map information via a display means of the mobile client terminal; a module for virtually determining a movement route of the user in a past world based on the presented movement route of the user and / or a movement route obtained by tracking position information of the user corresponding to the presented movement route of the user and the past map information, creating a virtual environment image that virtually reproduces a field of view of the user in the past world related to at least one of the plurality of coordinates related to the historical site, and transmitting the virtual environment image to a display means of the wearable terminal; A computer-executable program comprising:
[0018] Aspect 2. the current map information further comprising information identifying coordinates that the user can and cannot access; The program further comprises: a module for determining whether the coordinates related to the one or more historical sites or the coordinates nearby the historical sites correspond to the coordinates that the user cannot enter as specified by the current map information, and if so, editing the user's travel route in the past world to include at least the one or more coordinates related to the historical sites or the coordinates nearby the historical sites, and reflecting the path in the virtual environment image; 2. The program according to claim 1, comprising:
[0019] Aspect 3. The past map information includes coordinates corresponding to information about past buildings in the world, a module for including visual and / or audio information about a view of the structure in the vicinity of the coordinates corresponding to the coordinates in the current map information in the virtual environment image displayed by the wearable terminal or the information displayed by the mobile client terminal; 3. The program according to embodiment 1 or 2, further comprising:
[0020] Aspect 4. The program described in aspect 3, characterized in that the sound information is calculated assuming that it is blocked and attenuated by part or all of the building and reaches the user's hearing.
[0021] Aspect 5. 5. The program according to any one of aspects 1 to 4, wherein the guidance information is edited to reflect user feedback on the virtual environment video.
[0022] Aspect 6. The program according to any one of aspects 1 to 5, wherein the mobile client terminal is physically the same terminal as the wearable terminal.
[0023] Aspect 7. the mobile client terminal is a terminal physically separate from the wearable terminal; The program according to any one of aspects 1 to 5, wherein the wearable terminal is placed at a different location that is physically separated from a movement path of the user in the real world.
[0024] Aspect 8. The program according to any one of aspects 1 to 7, wherein the virtual environment image includes a non-existent building, natural object, event, or person.
[0025] Aspect 9. A virtual environment image generation device, comprising: A receiving means for receiving user location information from a mobile client terminal having a function of acquiring the user location information; a storage means for storing current map information capable of identifying multiple coordinates related to historical sites, and past map information different from the current map information and capable of identifying multiple coordinates related to historical sites, wherein one or more coordinates in the past map information are stored so as to correspond to one or more coordinates in the current map information; A determination means for determining whether the user is in the vicinity of any of the coordinates related to a historic site based on the received user location information and the current map information or the past map information or both; a guidance means for providing guidance information for guiding the user in the real world via a display means of the mobile client terminal based on the received position information of the user and the current map information when it is determined that the user is in the vicinity of any of the coordinates related to the historical site, by superimposing the guidance information on the field of view of the user viewing the real world; a real world travel route presenting means for presenting a travel route of the user who has received the guidance information, based on the received position information of the user and the current map information; an image creation means for virtually determining the movement of the user in the past world based on the presented movement route of the user and / or a movement route obtained by tracking the position information of the user corresponding to the presented movement route of the user and the past map information, and creating a virtual environment image that virtually reproduces the field of view of the user in the past world related to at least any of the plurality of coordinates related to the historical site; a transmission means for transmitting the virtual environment image created by the image creation means to a display means included in the wearable device; 13. An apparatus comprising:
[0026] Aspect 10. An apparatus according to embodiment 9, a mobile client terminal operatively coupled to said device via a network; a wearable terminal operatively coupled to the device via a network; A system comprising: The wearable terminal is configured to block the user's field of view when providing the virtual environment image to the user. A system characterized in that Effect of the Invention
[0027] According to the present invention, it is possible to guide a user through guidance information superimposed on the user's field of view in the real world while taking into consideration the situation of the historical site in the real world, and to separately provide the user with a virtual environment image corresponding to the user's movement in the real world. This has the effect of enabling the user to move appropriately in both the real world and the virtual world, while also deepening their knowledge of the historical site and enabling them to systematically accumulate that knowledge. [Brief description of the drawings]
[0028] [Figure 1] FIG. 1 is a block diagram illustrating a basic configuration of the present invention. [Diagram 2]FIG. 1 is an explanatory diagram of the information flow of a 4D hybrid metaverse system according to one embodiment of the present invention. [Diagram 3] A contemporary example of a view and tourists from the Arch of Severus in the Roman Forum. [Figure 4] 1 shows an example of a metaverse image showing the entire Roman Forum that can be provided by an embodiment of the present invention. [Diagram 5] 1 shows an example of metaverse footage of a group of buildings that existed in ancient Rome, which can be provided by an embodiment of the present invention. [Figure 6] FIG. 1 is a conceptual diagram showing an example of an information layer configuration in Edo Castle that can be provided by an embodiment of the present invention. [Figure 7] FIG. 2 is a comprehensive explanatory diagram of the information system related to a 4D hybrid AI engine that can be provided by an embodiment of the present invention. [Figure 8] 1 shows an example of guide information including an example of AR and VR placement in Edo Castle that can be provided by an embodiment of the present invention. [Figure 9] 1 shows an example of a virtual environment image on a clear day that can be provided by an embodiment of the present invention. [Figure 10] 1 shows an example of a rainy virtual environment image that can be provided by an embodiment of the present invention. [Figure 11] The present-day ruins of Edo Castle are shown as an example of a historic site. [Figure 12] Showing high rise buildings adjacent to the modern-day site of Edo Castle. [Figure 13] The view from a high-rise building is shown in Figure 12. [Figure 14] Examples of modern Edo Castle scenery and tourists are shown. [Figure 15] Examples of modern Edo Castle scenery and tourists are shown. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0029] The present invention is characterized by utilizing a coordinate system grid commonly set for various parts of the globe, and using a coordinate system common to both the real world (including historical sites) and the virtual world (the past world), guiding the user in both worlds, thereby contributing to the user's intellectual productivity. In one embodiment, a program including instructions executable by a computer having a processor can be provided. The program includes at least: A first module for receiving user location information from a mobile client terminal having a function for acquiring user location information; a second module that stores current map information that identifies a plurality of coordinates associated with a historic site and past map information that differs from the current map information and identifies a plurality of coordinates associated with the historic site, and that associates one or more coordinates in the past map information with one or more coordinates in the current map information; a third module for determining whether the user is in the vicinity of any of the coordinates associated with a historic site based on the received user location information and the current map information or the past map information or both; a fourth module that, when it is determined that the user is in the vicinity of any of the coordinates related to the historical site, provides guidance information for guiding the user in the real world based on the received position information of the user and the current map information, by superimposing the guidance information on the user's field of view viewing the real world via a display means of the mobile client terminal; a fifth module for presenting a route in the real world of the user having received the guidance information via a display means of the mobile client terminal; and a sixth module that virtually determines the user's movement route in the past world based on the presented movement route of the user and / or the movement route obtained by tracking the user's location information corresponding to the presented movement route of the user and the past map information, creates a virtual environment image that virtually reproduces the user's field of view in the past world related to at least any of the multiple coordinates related to the historical site, and transmits the image to a display means possessed by the wearable terminal.
[0030] The following description will be based on the above program, but it should be noted that the program can be similarly operated in a system to be described later. This program can be executed on any computer, for example, on a server on a network.
[0031] Fig. 1 is a block diagram for explaining the basic configuration of the above-mentioned program. The program includes the above-mentioned first module 110, second module 120, third module 130, fourth module 140, fifth module 150, and sixth module 160, and can be communicatively coupled to a mobile client terminal 80 and a wearable terminal 90. It should be noted that the configuration of the device according to the present invention can also be conceived from Fig. 1 in a similar manner.
[0032] [Acquisition of user location information and map information] The first module 110 is for receiving user location information (e.g., location information obtainable by GPS or VPS (Visual Positioning System)). The location information can be obtained from a mobile client terminal 80 (e.g., a smartphone, a tablet, a laptop PC, a smartwatch, etc.) that the user has in the real world and that has a function of obtaining the user's location information. In other words, the module 110 can be communicatively coupled to the mobile client terminal 80 via some network (e.g., a mobile phone network, a WAN, a wireless LAN, or a wired LAN).
[0033] In one embodiment, the position information may be three-dimensional (a coordinate system including altitude) or two-dimensional (a coordinate system not considering altitude). This coordinate system may be common to current map information 122 (map information showing the current real world) stored in the second module 120 and past map information 124 (map information showing the past real world, i.e., the past world). In other words, these map information may be considered to be able to track changes in the real world over time (changes due to construction, reconstruction, or demolition of buildings, changes in topography, or changes in water levels, etc.). Note that the past map information 124 does not necessarily need to be completely accurate, and may partially correspond to the current map information 122, especially for information from periods such as ancient times or early modern times when sufficient information is no longer available. In other words, it is sufficient that one or more coordinates in the past map information 124 correspond to one or more coordinates in the current map information 122, respectively.
[0034] In addition, these map information include the coordinates of multiple "sights" related to the historical sites (including the contents corresponding to the coordinates in this specification). This is because the effects of the present invention are not applicable to areas not related to historical sites. In this specification, a historical site is a city, castle site, or remains that existed in the past and includes multiple "sights" academically. Examples of such historical sites include Edo Castle, the Roman Forum, Pompeii, Chichen Itza, and the Yin Ruins. In this specification, such historical sites are sometimes referred to as "urban historical sites" to distinguish them from other natural heritage sites. In addition, "sights" refer to places that are academically significant, such as the Great Hall, the Pine Corridor, the Nakaoku Garden and group of buildings, the complex functions of the Ooku, the government space, the castle tower, and the combined functions of these in Edo Castle. Historic sites in the current map information 122 may refer to those objects that remain, or may be partially preserved, or may only have a monument or sign remaining, or may only have information indicating that something was once there.
[0035] The plurality of "sights" related to the historical sites may be preset in advance in the second module 120 in association with map information. Alternatively, the user may input the "sights" he or she wishes to see, and the second module 120 may specify the coordinates of the "sights" based on the input in association with the map information.
[0036] In one embodiment, the current map information 122 may further include information that identifies coordinates that are accessible to users and those that are not. The terms "accessible" and "impossible" here are not necessarily absolute, but may mean that tourists usually do not access these locations. For example, in Edo Castle, there are points where buildings once stood, but they are now lost and turned into garden paths to view beautiful plants. Such points can be classified as coordinates that are not accessible to users. Another point may be one where it is difficult to access due to rising water levels. Even in this case, it is not absolutely impossible to access the location by wearing diving equipment, but if a normal tourist would not go that far, this may be classified in the data as "impossible for users to access."
[0037] In one embodiment, the virtual environment may further include a module for determining whether the coordinates related to the historical site or the coordinates nearby correspond to coordinates that are inaccessible to the user, as identified by the current map information 122. If the coordinates correspond to inaccessible coordinates, the user's travel route in the past world, which will be described later, may be edited to include at least one or more coordinates related to the historical site or the coordinates nearby the historical site, and the edited route may be reflected in the virtual environment image.
[0038] [Determining when a user is near a historic site] The third module 130 can determine whether the user is in the vicinity of such a historical site based on either or both of the map information described above. The vicinity here refers to a distance that is generally considered to be close for tourism and learning purposes, and is generally within a distance of 10 m. However, if one is daring to grasp the entire city, it may be considered to be within a radius of 5 km, 3 km, or 1 km from any of the coordinates included in a certain historical site in the map information. Any distance other than these can be arbitrarily set as the "nearby" if it is appropriate in light of technical common sense. In any case, the threshold value of the vicinity can be set as a software configuration.
[0039] [Guiding the user in the real world and presenting or determining the user's travel route] The fourth module 140 is for providing guidance information to the user, and the fifth module 150 is for presenting a route for the user to travel in the real world based on the current map information 122. In one embodiment, based on location information (e.g., location information obtained by GPS) that can be obtained by the mobile client terminal 80 owned by the user, the user can receive guidance information (information that guides the user around the site) in the vicinity of a historical site in the real world, and the guidance information can suggest (suggest) a route for the user to travel in the real world.
[0040] The guidance information is provided (projected, etc.) via a display means of a mobile client terminal so as to be superimposed on the field of view of a user viewing the real world. This provision may be considered to be an AR (Augmented Reality)-like technique. This allows the user to perceive the guidance information superimposed on the real world while viewing the real world. This provision may be, for example, superimposed on the field of view of a user viewing the real world via a screen of a glasses-type device, projected onto the user's retina via a projection means, projected into the atmosphere as a holographic image, or projected as an image onto a wall or the like in the real world.
[0041] Examples of such guidance information include visual information such as a guide route (such as a route to visit each of the highlights in the historic site), guide points (such as a specific highlight in the historic site), and guide images (such as an image of the whole or part of the historic site or an image of a painting that imitates it). The guidance information may also include other information accompanying the visual information, such as audio information such as a narration (such as an audio commentary on the historic site). The guidance information is provided to the user via a mobile client terminal. The method of providing the information may be unidirectional or bidirectional. For example, multiple types of guidance information may be presented via a display means (such as a display or audio output) of the mobile client terminal, and the user may be able to select one of them. In one embodiment, the PLATEAU database created under the initiative of the Ministry of Land, Infrastructure, Transport and Tourism may be utilized to create guide data from three-dimensional coordinate data. Alternatively, the guidance information may include information searched from a predefined database using a query based on the user's location information (such as keywords including latitude and longitude, altitude, or place names, country names, former country names, road names, etc. related to either of them). For example, based on map information, a keyword can be associated with each coordinate, and when location information is received from a mobile client terminal indicating that a user is near a certain historical site (or a historical attraction), a predefined database can be searched using the keyword associated with the coordinate.
[0042] The user's movement route in the real world may be determined by the guidance information, or may be forced to follow a certain movement route, for example. In another embodiment, the user may be passively prompted (nudged) by the guidance information and independently determine the movement route. In either case, the location information from the user's mobile client device can be tracked to obtain (determine) the actual movement route of the user. In other words, even if the guidance information forces the user to follow a certain movement route, the user, who is actually a human, may deviate from the route, and therefore such a configuration can be adopted.
[0043] By receiving guide information in this way, users can easily recognize the highlights of the historical site and can have a bodily experience associated with visiting and appreciating the site. The guide information may be updated by feeding back the user's experience in some way (for example, by storing the user's experience as data on a server through text input, voice input, or recording). For example, if a certain historical site is damaged for some reason and it becomes difficult to enter, the user can provide feedback on this, and the guide information can be updated so that the user will not approach the historical site in the future. In addition, for example, statistics on multiple types of guide information used by users can be collected, and the guide information that was used most frequently can be updated to be provided with higher priority. From that feedback, keywords related to the historical site, the distance and speed traveled by the user, etc. can be extracted as features, and machine learning can be performed using these as input values to update the guide information.
[0044] The guide information may also be created by inputting data on works related to the historical site. For example, a book database or a history database may be searched for a phrase related to a certain historical site, and the obtained information may be used as input values for machine learning to automatically generate the guide information.
[0045] In some embodiments, a preset guide route through the historical site (e.g., a route around a specific number of points of interest) may be predefined as a default. In some embodiments, feedback from a previous experience of the historical site may be utilized when the user revisits the same historical site. For example, a default or different route may be presented based on the actual route the user previously took or feedback data entered by the user. In another embodiment, feedback about other users may be used to present a route to another user. In any case, the overall goal of the present invention is to go beyond individual knowledge and experience and to remember the reality of history, the hardships of the people who were active, and the accomplishments they made.
[0046] [Creating virtual environment footage] The sixth module 160 plays a role of virtually determining the user's movement route in the past world based on the user's movement route in the real world presented as described above and the past map information 124, and creating and transmitting a virtual environment image (VR) that virtually reproduces the user's field of view in the past world. As described above, such an operation is possible because at least some of the coordinates of the past map information 124 correspond to the coordinates of the current map information 122 (a common coordinate system is used). The above-mentioned guidance information may also be applied to the creation of the virtual environment image. In another embodiment, the sixth module 160 may create the virtual environment image based on a movement route (actual movement route) obtained by tracking the user's position information corresponding to the presented user's movement route (for example, by a GPS possessed by the mobile client terminal). This means that the virtual environment image can be created even if the user does not necessarily follow the presented movement route exactly. In yet another embodiment, the sixth module 160 may create the virtual environment image based on both the presented user's movement route and the actual movement route. The virtual environment imagery may include, for example, the architectural concepts behind some of the great Roman buildings, the designs of temples, and the stories of the Romans carved into their triumphal columns.
[0047] The virtual environment image virtually reproduces the field of view of the user in the past world that is related to at least one of a number of points of interest (coordinates) related to the historical site. Note that the "field of view" here refers to the depiction of the world that the user will experience (view) along the user's travel route in the virtually determined past world. Note that even if the virtual environment image does not include visual information, as described below, the "field of view" should be understood as a concept that encompasses the information experienced by the user. In other words, the "field of view" can be said to be a reproduction of the information perceived by the user in the virtually reproduced past world.
[0048] The virtual environment image may be in the so-called metaverse, and may reproduce things from the past world. In some embodiments, the virtual environment image may include buildings, natural objects, events, or people that no longer exist. The virtual environment image may reproduce, for example, the exterior or interior, or both, of a structure that has now disappeared (such as Edo Castle). The virtual environment image may also be a clever integration of interesting functions as the core of politics and the splendor of a worship hall, in addition to the defensive features of the castle. After forming consciousness in the brain while actually walking around the site, the user can acquire deeper knowledge of the past world by viewing the virtual environment image via the wearable terminal 90. This will be specifically explained below. Note that the term "virtual environment image" in this specification is an expression used for ease of understanding, and is not limited to visual information, but may also include information based on signals that give the user audio information, text information, tactile information, olfactory information, warmth, coldness, air blowing, or other stimuli. In some embodiments, the virtual environment image may intentionally not include visual information (e.g., for users with impaired vision). In this specification, the term "virtual environment image" may be rephrased as "a representation (of some kind) based on signals for providing a virtual experience to a user."
[0049] The created virtual environment image is transmitted to a wearable terminal (e.g., a head-mounted display, smart glasses, etc.) that can be used by the user, and can be provided to the user via a display means there. In one embodiment, the wearable terminal may exist and function at a location physically separated from the user's travel route in the real world described above. In other words, the user can receive the virtual environment image at a safe location (preferably in a private room, inside a vehicle, inside a building, etc.) away from the historical site, and enjoy traveling in the recreated past world. In a preferred embodiment, the wearable terminal may have a function of blocking the user's field of view (a function of allowing the user to be immersed in the virtual environment image as VR).
[0050] In one embodiment, as shown in the lower part of Fig. 1, the simple VR may be displayed on another mobile client terminal 85. Switching between the mobile client terminal 85 and the mobile client terminal 80 may be achieved, for example, by selecting a function by switching a switch on the AR / VR shared device. The mobile client terminal 85 and the mobile client terminal 80 may be the same physical hardware, or may be different hardware.
[0051] In one embodiment, when a user is detected (e.g., based on location information from a mobile client device) to be in the vicinity of a historical site (e.g., passing by the historical site), the sixth module 160 may create and provide to the wearable device an image of a virtual environment related to the historical site.
[0052] In some embodiments, there may be a means for providing a user with feedback about their experience as they navigate (along a path) through the virtual environment or afterward, and such feedback may be used to update navigation information as described above.
[0053] In another embodiment, the sixth module 160 may use the guidance information generated by the fourth module 140 to reflect the guidance information in the virtual environment image. In this case, the guidance information provided to the mobile client terminal 80 in the form of AR may be light (for example, only text data), while the guidance information provided to the wearable terminal 90 in the form of VR may be heavy (for example, data including video). For example, the guidance information received by the mobile client terminal 80 may not include video, still images, or both. Also, for example, the guidance information received by the wearable terminal 90 may include video, still images, or both. In this way, the mobile client terminal 80 can enjoy a smooth experience while strolling through the historical site, while the wearable terminal 90 can enjoy an immersive experience.
[0054] In one embodiment, if it is determined (by location information from the mobile client terminal) that the user is in the vicinity of a particular point (coordinates) in the historical site, the fourth module 140 may select and generate and provide guidance information related to the point. In another embodiment, even if it is detected that the user is not in the vicinity of the historical site, the fourth module 140 may still generate and provide guidance information (e.g., information about an overall map of the particular historical site), thereby attracting the user to the particular historical site.
[0055] In a preferred embodiment, the mobile client terminal and the wearable terminal capable of providing a virtual environment image may be physically separate devices. In another embodiment, the same device may function as both the mobile client terminal and the wearable terminal.
[0056] In cases where the mobile client terminal and the wearable terminal are physically the same device, the device may be capable of switching between AR and VR display, such as a wearable glasses terminal capable of communicating via a mobile phone network, or a smartwatch terminal equipped with some sort of image projection function.
[0057] In the case where the mobile client terminal and the wearable terminal are physically separate devices, the user may, for example, use the wearable terminal at a location away from the historical site to receive images of the virtual environment, or may use it at a location close to the historical site.
[0058] In one embodiment, as an optional additional function, the user's level of interest in a certain historical site near a certain coordinate may be quantitatively evaluated according to some threshold value based on the distance traveled from one coordinate to another, or the length of time spent near a certain coordinate, or a combination of these. Then, based on the quantified level of interest of the user, the type of content related to the historical site to be provided in the virtual environment image may be selected. For example, if it is determined that a certain user is located at coordinates near the Imperial Palace, and it is determined that the user has a high interest in the Imperial Palace, information about historical facts related to the Imperial Palace may be provided in the virtual environment image. Also, if it is determined that the user has a low interest in the Imperial Palace, only the name of the place may be displayed.
[0059] In one embodiment, it may be possible to detect whether the user moves to a certain extent while experiencing the virtual environment video (e.g., based on location information from a mobile client terminal). If such movement is detected, some kind of notification may be given for the user's safety (e.g., stopping the virtual environment video, outputting a warning sound, outputting a warning screen, etc.).
[0060] In one embodiment, the program may further include a module that generates visual information and / or environmental sound information about a landscape related to a building in the past world in the vicinity of coordinates in the current map information that correspond to information about the building in the past map information. The visual information and / or environmental sound information may be included in the virtual environment image displayed by the wearable terminal or the information displayed by the mobile client terminal, or both. With such a configuration, for example, a user can experience a landscape viewed from the inside of a virtual building to the outside at coordinates corresponding to the inside of the non-existent past building. Also, for example, it is possible to make a sound when the user virtually passes through the wall of such a virtual building, thereby attractively presenting the change of landscape (i.e., the change of image and sound). In one example, the wall of the virtual building may be partially or completely blocked from sound (such as environmental sound) from a virtual sound source, thereby increasing the realism of the virtual building. For example, when a user is (virtually) near a virtual room separated by a virtual wall, for sound information (e.g., natural sounds, the sound of musical instruments (such as koto and drums), human conversations and songs, etc.) originating from within the room, an attenuation curve of a sound signal relating to the distance or positional relationship between the user and the sound source and the obstruction conditions may be calculated according to preset data such as the material, thickness, temperature and humidity conditions, and weather conditions of the wall, so that the volume that actually reaches the user's hearing can be adjusted. In one embodiment, sound information (such as environmental sound information) can be calculated assuming that it is obstructed and attenuated by a part or the whole of the building, and can be made to reach the user's hearing.
[0061] In some embodiments, the height of the user's viewpoint in the virtual environment image may be changed according to certain conditions. For example, it is possible to adjust the viewpoint so that a child's avatar can view the scenery from a low viewpoint, and a tall avatar can view the scenery from a high viewpoint. In some embodiments, when a user is viewing an AR image based on position information including height information, if the user moves based on the original height information of the historical site, the user's line of sight may be close to the floor, creating a sense of incongruity. This is because, although people are currently walking on the ground, they used to walk on the floor above the foundations or bases. In such cases, it is also possible to display the scenery seen from the perspective of people at the time in AR and / or VR by correcting the user's line of sight inside the historical site to that of people at the time.
[0062] [Equipment and Systems] In some embodiments, a device including hardware and software may be provided, either in the form of a program or a device, capable of enabling communication between a mobile client terminal held by a user (in the real world) and a wearable terminal (e.g., a head mounted display, smart glasses, etc.) capable of providing an image of a virtual environment to the user.
[0063] The device comprises at least A receiving means (such as a communication device or an I / O device) for receiving user location information from a mobile client terminal having a function of acquiring the user location information; A storage means (such as a local or network storage or memory) that stores current map information that can identify multiple coordinates related to historical sites, and past map information that is different from the current map information and can identify multiple coordinates related to historical sites, where one or more coordinates in the past map information are stored so as to correspond to one or more coordinates in the current map information, respectively; A determination means (software, circuitry, etc.) for determining whether the user is in the vicinity of any of the coordinates related to a historic site based on the received user location information and the current map information or the past map information or both; a guidance means (software, circuit, etc.) for providing guidance information for guiding the user, based on the received position information of the user and the current map information, via a display means of the mobile client terminal, so as to be superimposed on the user's field of view in the real world, when it is determined that the user is in the vicinity of any of the coordinates related to the historical site; A real world travel route determination means (software, circuit, etc.) that presents a travel route for the user who has received the guidance information based on the received position information of the user and the current map information; an image creation means (software, circuit, etc.) for virtually determining the movement of the user in the past world based on the presented movement route of the user and / or a movement route obtained by tracking the position information of the user corresponding to the presented movement route of the user and the past map information, and creating a virtual environment image that virtually reproduces the field of view of the user in the past world related to at least any of the plurality of coordinates related to the historical site; a transmission means (such as a communication device or an I / O device) for transmitting the virtual environment image created by the image creation means to a display means included in the wearable terminal; It can be made to include.
[0064] In some embodiments, a system may be provided that includes the above-described apparatus and a mobile client terminal and / or a wearable terminal. EXAMPLES
[0065] The following is a further explanation of an embodiment of the present invention. Figure 2 shows an explanatory diagram of the information flow of a 4D hybrid metaverse system as an embodiment of the present invention. Here, an example is shown in which the Roman Forum is assumed to be the target historical site.
[0066] The subject of the program according to the embodiment of the present invention is shown as a 4D hybrid AI engine in the middle right of Figure 2. The user's actions in the real world near the historical site are shown in the lower left of Figure 2, and the virtual environment image is shown in the upper left of Figure 2. Roughly speaking, the concept of adding a virtual environment image recreating the past world and a one-dimensional time (period scale) connecting the real world to the three-dimensional space related to the historical site is called 4D hybrid. In other words, the main feature is the combination of the user's (human) experience based on the archaeology of real space in the real world and the user's experience in the past world.
[0067] The 4D Hybrid AI Engine (hereinafter also referred to as the "Main Engine") first obtains location information from the mobile client device carried by the user (human system). Based on that location information, it determines whether the user is near a historical site (or a point of interest). In this example, the Main Engine can determine whether the user is near the Roman Forum (or a point of interest) based on the received location information and current map information (which can be obtained, for example, by accessing a map database not shown).
[0068] If it is determined that the user is in the vicinity of a historical site (in this example, a specific location that constitutes the Roman Forum), the main engine can provide guidance information to the user on-site via the output system (which may include video signal output, audio signal output, etc.) through the local guide system creation engine. Information that may be included in such guidance information includes guide routes, guide points, guide images, narration, and past images related to the historical site (such as images of buildings). Such information may be provided by any appropriate expression method, such as video, audio, or text, and may preferably be provided as a combination of video and dynamic head tracking audio, for example. With such a configuration, it is possible to reproduce and provide more realistic sounds (such as the direction from which the sound is coming, reverberation, etc.) to the user.
[0069] A local user who receives guidance information through a mobile client terminal (which may be one terminal or a combination of multiple terminals, the same applies below) can use the guidance information to move around the historical site in the real world and deepen their knowledge. This can be considered as a real-space archaeological activity, so to speak, and the user can recognize the historical landscape surrounding the historical site, become aware of the real landscape including the surrounding area of the historical site, and memorize the whole-body sensation in his / her brain. It is also possible to adopt a bidirectional interface that allows the user to further extract guidance information that interests them by operating the guide system from the mobile client terminal. In providing (guidance information) to users in such a virtual environment image (for example, AR-related information that can be provided to a mobile client terminal, or VR-related information that can be provided to a wearable terminal, or both), the technology described in Patent No. 5865708 by the present inventor may be adopted.
[0070] The user's travel route in the real world may be determined by the user himself / herself with reference to the guide information as described above, or the main engine may force the user to follow a certain travel route via the local guide system creation engine (for example, via guidance using images or dynamic tracking audio provided via the display means of the mobile client terminal). In either case, the user's travel route in the vicinity of the historical site in the real world will be stored by the user himself / herself, but it can also be acquired by the main engine via the mobile client terminal. In addition, the main engine may also acquire, as feedback, via the mobile client terminal, a record of how the user used the guide system at which point (coordinate) on-site.
[0071] Additionally, a user may be able to use a mobile client terminal or other terminal to send feedback to the main engine via some input means (text input, voice input, etc.) while traveling in the real world or after traveling in the real world. For example, a user visiting a historical site in the real world may provide input about new issues (e.g., archaeological discoveries) that he or she discovers while on-site, which the main engine receives.
[0072] Using the feedback data obtained in this way, the main engine may further edit the guidance information (4D guide data) to provide more personalized content for each user, enabling more advanced guidance. The 4D guide data referred to here should be understood to mean "data based on a database that is classified into a four-dimensional layer structure consisting of three-dimensional space and one-dimensional time" as defined in the above-mentioned Patent Document 2. For user navigation within such a virtual environment image (for example, AR-related information that can be provided to a mobile client terminal, or VR-related information that can be provided to a wearable terminal, or both), the technology described in Patent No. 4050983 by the present inventor may be adopted.
[0073] The guidance information may also include digital data extracted from works related to the historical site. This makes it easier to utilize a body of knowledge selected from a vast number of past works in the guidance information, which has the effect of improving the quality of the user's intellectual production activities. In one embodiment, the main engine may generate information recommended to the user (e.g., based on extraction or editing of the digital data) by statistically analyzing the user's events and fields of interest based on the feedback data from the user as described above. Such information recommended to the user may also be included in the guidance information.
[0074] When the user has completed viewing the historical site in the real world, the main engine can adjust the virtual space system based on that. In other words, the main engine can provide the user with a virtual environment image using a coordinate system common to the above-mentioned real-world travel route via the output system and the metaverse input / output engine. This virtual environment image is created based on past map information (and guide information, if necessary) as the past appearance of historical sites such as the Roman Forum, and can also reflect feedback based on the user's experience of the "sights" in the historical site. The user can virtually move through the virtually recreated historical site in its past appearance, and can enjoy intellectual exploration of the "sights" of the past, making use of the frontal association area. This virtual environment image may be enhanced in reality by utilizing precise 3D graphics and 3D head-tracking audio.
[0075] In this example, the user can wear a wearable device at a location physically separated from the user's route in the real world, i.e., the actual Roman Forum (for example, at home in Japan), and virtually experience the past Roman Forum. In another example, the user can wear a wearable device and experience a virtual environment image while taking in a bird's-eye view of the entire view at a resting place (for example, a tranquil hilltop) in or near the actual Roman Forum. This allows the user to safely enjoy the virtual environment image, and the virtual environment image reflects the user's actual route of travel in the current Roman Forum. This function builds an organic relationship between the whole and the individual in the user's mind, structuring their understanding.
[0076] Figure 3 shows a view from the Arch of Severus in the modern Roman Forum. Figure 4 shows an example of an overall virtual environment image (represented in the metaverse) of the Roman Forum, a historical site that includes the attractions corresponding to those in Figure 3. As mentioned above, users can view the overall image in Figure 4 from a distance. Users can also experience the city of the past from the perspective of the residents at the time, as shown in Figure 5. The ability to experience VR (metaverse) by switching perspectives in this way is also one of the features of the present invention. Specifically, this place is where Caesar, the hero of the Roman Republic, gave a passionate speech and led the people. It is also the place where a temple was built for Caesar, who became a god after his death.
[0077] The main engine can also provide users with guidance information (such as historical background information) in the virtual environment, allowing users to continue their intellectual learning even while experiencing a re-creation of a past world.
[0078] Additionally, while experiencing the virtual environment image, the user can send feedback to the main engine in the same manner as described above, and the main engine can use that feedback to further edit the virtual environment image and guidance information.
[0079] Figure 6 is a conceptual diagram showing an example of information layer configuration when the target historical site is Edo Castle. In this example, the main engine forms multiple virtual information layers on a common coordinate system to create virtual environment images (which may include audio), and these layers can be used together to output dramatic VR information.
[0080] The D-1 level in Figure 6 is an information layer that recreates the scenery, including the buildings, of Edo Castle from the past, based on historical map information. This virtually recreated Edo Castle is constructed in a coordinate system that is common to the current map information. In order to provide the user with virtual movement within the virtual environment image, this can be achieved by recreating the actual historical space that corresponds to the space in which Edo Castle actually existed.
[0081] Level E is a layer that consolidates information on architectural elements and provides concrete examples of Shoin-zukuri architecture, which was an architectural style at the time.
[0082] The D-2 level is a layer that aggregates information about the movements of people in the past world. This allows the behavior of a virtually reproduced person to change according to time, so it is possible to configure a virtual environment video to match the real time zone of the user receiving the virtual environment video. For example, if the real time of the user is 10:00 a.m., it is possible to reproduce a scene in which a senior councilor as a character in the virtual environment video enters the palace accompanied by a drum to ascend the castle, holds a meeting with the shogun, and then returns to the government space to carry out politics. In another embodiment, a time that is shifted from the real time zone of the user may be reproduced in the virtual environment video.
[0083] In some embodiments, a user may be able to interact with characters within a virtual environment. Such interaction with virtual historical characters may employ techniques described in my patent application Ser. No. 6,910,629.
[0084] The D-3 level is a layer that aggregates information about the potential of sound. By specifying the position coordinates of a sound source such as a climbing drum, the direction of the sound propagating from the sound source can be reproduced to the user's hearing (for example, in a dynamic head tracking manner or in a surround manner). In addition, the technology described in Patent No. 4791064 by the present inventor may be adopted as a method for generating such an acoustic signal.
[0085] FIG. 7 is a comprehensive explanatory diagram of the information system related to the 4D hybrid AI engine, and explains the details of the main engine described in FIG. 2. The core system can generate and edit guide information (guide data) according to the type of historical site set. For example, by inputting data related to the historical site into an LLM (Large Language Model) type AI engine, it is possible to output guide information. Such input data may be data obtained by digitally converting existing works related to the historical site, or data obtained from an existing central spatiotemporal database.
[0086] For example, as explained in Fig. 7, a scenario may be created according to the user's movement route within the virtual environment image. For example, keywords related to Edo Castle may be extracted from the central time-space database, and a scenario related to Edo Castle corresponding to the past world may be created. As an example, the core system can reproduce the people, light, and weather related to the Pine Corridor of Edo Castle in the virtual environment image, and can infer the actual state of Edo society at that time while reproducing even the physical and psychological background of the stabbing incident that occurred on the morning of March 14, 1701 (Genroku 14). It is then possible to have the user experience this, obtain feedback from the user, and update the scenario.
[0087] FIG. 8 shows an example of providing guide information (local guide route) in Edo Castle. Edo Castle has the castle gate, the great hall, the pine corridor, the inner palace and the government office, the inner palace, the castle tower, the observation deck, and so on. The user can receive the guide information as AR via the display means of a mobile client terminal such as a smartphone, and is guided along the route illustrated in the figure. The user can also experience the Edo Castle of the past in VR via a wearable terminal such as smart glasses in a safe space such as a chair or lawn installed in the historic site. Alternatively, it is possible to obtain a lot of information while getting a bird's-eye view of the entire Edo Castle of the past using VR from the observation deck of a high-rise building located in a position overlooking Edo Castle. In some embodiments, the height may be included in the user's position information by using an altimeter (which may be built into the mobile client terminal or may be a separate hardware). The former VR may be viewed simply through a lightweight device, for example. The latter VR may be experienced using a device such as a full-scale head-mounted display.
[0088] Figures 9 and 10 show examples of VR that can be used in the Pine Corridor in Figure 7. By recreating the shoji screens in VR with a multi-layered structure in this way, it is possible to change the display realistically according to weather changes in VR. These weather changes can be determined independently by the VR, or they can be changed according to the weather, date, season, time, etc. in the real world. With this configuration, for example, it is possible to obtain the effect that the user can realistically experience the way that officials in the (virtual) castle at the time would choose one of three states: "let in outside air," "let in light," or "protect against wind and rain." Figure 9 shows a rendering that corresponds to a sunny day, and Figure 10 shows a rendering that corresponds to a rainy day. This corresponds to the description in Figure 7 above.
[0089] Figure 11 shows the modern Edo Castle ruins as an example of a historical site. There are many beautiful flower beds, garden paths and lawns here, regardless of the layout of the historical site. This system aims to prevent actual access to these areas while providing a deeper understanding of the precious world of the past. Figures 12 and 13 show examples of high-rise buildings around Figure 11 and the view from them.
[0090] Figures 14 and 15 show photographs of people strolling around the present-day ruins of Edo Castle (historic site). These people, who had no way of fully understanding what Edo Castle was actually like with conventional technology, can obtain an excellent intellectual learning experience with the present invention. For example, even if they could somewhat recognize the atmosphere of the castle at a point where the castle gate remains, it was difficult to recognize the atmosphere of the past when they came to an open area as in Figure 14. With the present invention, the lost link between the real world and the former Edo Castle can be reproduced and provided to users by switching between AR and VR. In addition, the present invention makes it possible to visually experience and understand the entire extremely complex space and function while overlooking Edo Castle as in Figure 13 from a modern Tokyo high-rise building as shown in Figures 12 and 15. The present invention is also considered to be suitable for academic use, and its contribution is expected to be significant. [Explanation of symbols]
[0091] 80 Mobile client terminals 85 (another) mobile client terminals 90 Wearable Devices 110 First Module 120 Second Module 122 Current Map Information 124 Past Map Information 130 Third Module 140 Fourth Module 150 Fifth Module 160 Sixth Module
Claims
1. A module for receiving user location information from a mobile client terminal having a function for acquiring user location information; a module for storing current map information identifying a plurality of coordinates related to a historic site and past map information that differs from the current map information and identifies a plurality of coordinates related to the historic site, and for associating one or more coordinates in the past map information with one or more coordinates in the current map information; a module for determining whether the user is in the vicinity of any of the coordinates associated with a historic site based on the received user location information and the current map information or the past map information or both; a module for providing guidance information for guiding the user in the real world based on the received position information of the user and the current map information when it is determined that the user is in the vicinity of any of the coordinates related to the historical site, via a display means of the mobile client terminal, by superimposing the guidance information on the user's field of view viewing the real world; a module for presenting a route in the real world of the user having received the guidance information, based on the current map information, via a display means of the mobile client terminal; a module for virtually determining a movement route of the user in a past world based on the presented movement route of the user and / or a movement route obtained by tracking position information of the user corresponding to the presented movement route of the user and the past map information, creating a virtual environment image that virtually reproduces a field of view of the user in the past world related to at least any one of the plurality of coordinates related to the historical site, and transmitting the virtual environment image to a display means of the wearable terminal; A computer-executable program comprising:
2. the current map information further comprising information identifying coordinates that the user can and cannot access; The program further comprises: a module for determining whether the coordinates related to the one or more historical sites or the coordinates nearby the historical sites correspond to the coordinates that the user cannot enter as specified by the current map information, and if so, editing the user's travel route in the past world to include at least the one or more coordinates related to the historical sites or the coordinates nearby the historical sites, and reflecting the path in the virtual environment image; The program of claim 1 .
3. The past map information includes coordinates corresponding to information about past buildings in the world, a module for including visual and / or audio information about a view of the building in the vicinity of the coordinates corresponding to the coordinates in the current map information in the virtual environment image displayed by the wearable terminal or the information displayed by the mobile client terminal; The program according to claim 1 or 2, further comprising:
4. The program according to claim 3, wherein the audio information is calculated assuming that it is blocked and attenuated by a part or all of the building and reaches the user's hearing.
5. The program according to claim 1 , wherein the guidance information is edited to reflect user feedback regarding the virtual environment image.
6. The program according to claim 1 , wherein the mobile client terminal is physically the same terminal as the wearable terminal.
7. the mobile client terminal is a terminal physically separate from the wearable terminal; The program according to claim 1 , wherein the wearable device is placed at a different location physically separated from the user's movement path in the real world.
8. The program of claim 1 , wherein the virtual environment image includes a non-existent building, natural object, event, or person.
9. A virtual environment image generation device, comprising: A receiving means for receiving user location information from a mobile client terminal having a function of acquiring the user location information; a storage means for storing current map information capable of identifying multiple coordinates related to historical sites, and past map information different from the current map information and capable of identifying multiple coordinates related to historical sites, wherein one or more coordinates in the past map information are stored so as to correspond to one or more coordinates in the current map information; A determination means for determining whether the user is in the vicinity of any of the coordinates related to a historic site based on the received user location information and the current map information or the past map information or both; a guidance means for providing guidance information for guiding the user in the real world via a display means of the mobile client terminal based on the received position information of the user and the current map information when it is determined that the user is in the vicinity of any of the coordinates related to the historical site, by superimposing the guidance information on the field of view of the user viewing the real world; a real world travel route presenting means for presenting a travel route of the user who has received the guidance information, based on the received position information of the user and the current map information; an image creation means for virtually determining the movement of the user in the past world based on the presented movement route of the user and / or a movement route obtained by tracking the user's position information corresponding to the presented movement route of the user and the past map information, and creating a virtual environment image that virtually reproduces the field of view of the user in the past world related to at least any of the plurality of coordinates related to the historical site; a transmission means for transmitting the virtual environment image created by the image creation means to a display means included in the wearable device; 13. An apparatus comprising:
10. An apparatus according to claim 9, a mobile client terminal operatively coupled to said device via a network; a wearable terminal operatively coupled to the device via a network; A system comprising: The wearable terminal is configured to block the user's field of view when providing the virtual environment image to the user. A system characterized in that
Citation Information
Patent Citations
Catalyst for olefin polymerization, manufacture and olefin polymerization
JP1983065708A
Battery over-discharge prevention method and pack battery used thereof
JP1998050351A
Virtual museum system
JP2004145657A
Virtual experience providing device, virtual experience providing system, virtual experience providing method, and program
JP2022142296A
Navigation device with ambient sound playback
JP4791064B2
Cited By
A data processing method and system based on multi-modal life narrative
CN122795231A
