Mobile information terminal and method for controlling display of virtual reality object
The mobile information terminal with an HMD system addresses the challenge of displaying hidden features by generating virtual reality objects, enhancing user convenience and spatial awareness by providing a comprehensive view of features within and beyond the user's direct line of sight.
Patent Information
- Application Number
- PCT/JP2023/018942
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-22
- Publication Date
- 2024-11-28
AI Technical Summary
Existing mobile information terminals struggle to accurately display the positional relationship of surrounding features, including those not directly visible from the user's viewpoint, limiting user convenience and spatial awareness.
A mobile information terminal with a Head-Mounted Display (HMD) system that generates and displays virtual reality objects representing feature information, allowing users to visualize features within their field of view, including hidden or obstructed ones, by using a combination of cameras, sensors, and network communication to create a comprehensive spatial representation.
Enhances user convenience by providing a clear and wide understanding of spatial relationships around the user, including features not directly visible, through the generation and display of virtual reality objects that overlay or position information in an intuitive manner on the display screen.
Smart Images

Figure JP2023018942_28112024_PF_FP_ABST
Abstract
Description
Mobile information terminal and virtual reality object display control method
[0001] The present invention relates to a mobile information terminal that displays a virtual reality object and a method for controlling the display of the virtual reality object.
[0002] In recent years, Augmented Reality (AR) technology has become widely used, which adds digital information to the real world and reflects and extends virtual objects (virtual reality objects) in a virtual space created using CG (Computer Graphics) into the real space. Mobile information devices that allow users to easily handle virtual reality objects while recognizing the real space in three dimensions have become widespread.
[0003] As an example of displaying the positional relationships of surrounding features from a user's viewpoint, Patent Document 1 describes the following: "A portable terminal with a navigation function equipped with a camera and a display for displaying an image captured by the camera, comprising: first detection means for detecting the current position of the portable terminal; second detection means for detecting the orientation of the camera; setting means for setting a destination; acquisition means for acquiring information relating to the route from the current position to the destination from a server based on the destination set by the setting means and the current position detected by the first detection means; calculation means for calculating the direction to travel on the image captured by the camera based on the information relating to the route acquired by the acquisition means and the orientation of the camera detected by the second detection means; and means for adding an indicator indicating the direction to travel calculated by the calculation means to the image captured by the camera and displaying the same on the display."
[0004] Japanese Patent Application Laid-Open No. 2006-194665
[0005] In the above-mentioned Patent Document 1, images captured in real time by a camera mounted on a portable terminal are displayed on a display of the portable terminal, and the direction to travel can be displayed on the captured image displayed on the display. However, the Patent Document 1 only handles the display of information on features visible from the user's viewpoint. Therefore, there is a problem in that information on features not visible from the user's viewpoint is not displayed, making it difficult to accurately grasp the positional relationship of surrounding features from the user's viewpoint.
[0006] Taking the above-mentioned problems into consideration, the present invention provides a mobile information terminal and a display control method for virtual reality objects that make it easier for a user to understand the spatial positional relationships of features around the user, including features that cannot be directly seen from the user's viewpoint, thereby improving convenience.
[0007] Among the inventions disclosed in this application, the outline of representative inventions will be briefly explained as follows.
[0008] (1) A portable information terminal that allows a user to view the field of view ahead and has the function of generating and displaying feature information regarding features within the user's field of view as virtual reality objects, wherein the portable information terminal generates feature information representing the shapes of features within the user's field of view, including features within the user's field of view that cannot be directly seen, as virtual reality objects, and displays the generated virtual reality objects on a display screen in a manner that corresponds to the position of the features within the user's field of view.
[0009] (2) A mobile information terminal according to (1), characterized in that the mobile information terminal displays a virtual reality object indicating feature information of a feature within the user's field of view so as to be superimposed on the viewing position of the corresponding feature.
[0010] (3) A portable information terminal according to (1), characterized in that the portable information terminal displays a virtual reality object indicating feature information of a feature within the user's field of view in the upper area of the display screen in a manner corresponding to the feature's position.
[0011] (4) A mobile information terminal according to (3), characterized in that the mobile information terminal generates feature information of the feature in a reduced form of the feature's height as a virtual reality object, and displays the generated virtual reality object in the upper area of the display screen while maintaining its parallelism with the ground in the display.
[0012] (5) A portable information terminal according to (3), characterized in that the portable information terminal displays a virtual reality object showing feature information of a feature within the user's field of view at the user's line of sight.
[0013] (6) A portable information terminal according to (1), characterized in that the display screen displays only virtual reality objects located in a specific area, such as an area of interest to the user or an area within a predetermined radius from a specific location.
[0014] (7) A mobile information terminal according to (1), characterized in that the mobile information terminal generates feature information showing a miniature of a feature within the user's field of view as a virtual reality object, and displays the generated virtual reality object near the entrance to the feature.
[0015] (8) A portable information terminal as described in (1), characterized in that the portable information terminal generates internal information of a specified feature indicating the internal state of the specified feature as a virtual reality object, and displays the generated virtual reality object at a corresponding internal position within the virtual reality object representing the specified feature.
[0016] (9) A mobile information terminal according to (1), characterized in that the mobile information terminal generates internal information of a specified feature indicating the internal state of the specified feature as a virtual reality object, and when a user pulls the specified feature or the virtual reality object of the specified feature in a specific direction, displays the virtual reality object of the internal information of the feature generated at the position in the specific direction of the pull.
[0017] (10) A mobile information terminal as described in (1), characterized in that the mobile information terminal generates feature entrance mark information indicating the entrance to a specified feature as a virtual reality object, and displays the generated virtual reality object at the entrance position within the virtual reality object representing the specified feature.
[0018] (11) A display control method for virtual reality objects in a mobile information terminal that allows a user to view the field of view ahead and has the function of generating and displaying feature information regarding features within the user's field of view as virtual reality objects, the display control method for virtual reality objects comprising the steps of: generating feature information representing the shapes of features within the user's field of view, including features within the user's field of view that cannot be directly seen, as virtual reality objects; and displaying the generated virtual reality objects on a display screen in a form that corresponds to the position of the features within the user's field of view.
[0019] (12) A display control method for a virtual reality object as described in (11), comprising a step of displaying a virtual reality object indicating feature information of a feature within the user's field of view so as to be superimposed on the viewing position of the corresponding feature.
[0020] (13) A display control method for a virtual reality object as described in (11), comprising a step of displaying a virtual reality object indicating feature information of a feature within the user's field of view in the upper area of the display screen in a manner corresponding to the feature position.
[0021] By using the technology of the present invention, it is possible to provide a mobile information terminal and a method for controlling the display of virtual reality objects that make it easier to understand the spatial positional relationships of features around the user, including features that are not directly visible from the user's viewpoint, thereby improving convenience. Note that problems, configurations, and effects other than those described above will become clear from the description of the following embodiments.
[0022] 1 is a block diagram illustrating an example of the configuration of an HMD as an example of a mobile information terminal according to the present embodiment. FIG. 2 is an example of a flowchart illustrating the basic operation of the mobile information terminal according to the present embodiment. FIG. 3 is an example of a flowchart illustrating display processing of the mobile information terminal according to the present embodiment. FIG. 4 is an example of a diagram illustrating an external schematic diagram of an implementation state of the mobile information terminal according to the present embodiment. FIG. 5 is an example of a diagram illustrating a field of view display screen showing a landscape of real features viewed by a user on the mobile information terminal of the embodiment described in FIG. 3. FIG. 6 is an example of a diagram illustrating a field of view display screen on the mobile information terminal of the embodiment described in FIG. 3, in which a virtual reality object showing information of a feature in the user's field of view is superimposed on the position of the real feature. FIG. 7 is an example of a diagram illustrating a field of view display screen on the mobile information terminal of the embodiment described in FIG. 3, in which a virtual reality object showing information of a feature in the user's field of view is displayed in the upper region of the display screen. FIG. 8 is another example of a diagram illustrating a field of view display screen on the mobile information terminal of the embodiment described in FIG. 3, in which a virtual reality object showing information of a feature in the user's field of view is displayed in the upper region of the display screen. FIG. 9 is another example of a diagram illustrating a field of view display screen on the mobile information terminal of the embodiment described in FIG. 3, in which a virtual reality object showing information of a feature in the user's field of view is displayed in the upper region of the display screen. FIG. 4 is an example of a view display screen on the mobile information terminal of the embodiment described in FIG. 3, in which a virtual reality object showing information about a feature within the user's field of view is displayed at the eye level of the display screen. FIG. 5 is an example of a view display screen on the mobile information terminal of the embodiment described in FIG. 3, in which a virtual reality object corresponding to a feature in an area of interest is displayed superimposed on the position of the real feature. FIG. 6 is an example of a view display screen on the mobile information terminal of the embodiment described in FIG. 3, in which a virtual reality object corresponding to a feature in an area of interest is displayed in the upper region of the display screen. FIG. 7 is an example of a view display screen on the mobile information terminal of the embodiment described in FIG. 3, in which a virtual reality object for a feature within a predetermined radius from the user's position is displayed superimposed on the position of the real feature. FIG. 8 is an example of a view display screen on the mobile information terminal of the embodiment described in FIG. 3, in which a virtual reality object showing a miniature of the feature is displayed near the entrance position of the feature.1 is an example of a diagram illustrating a field of view display screen on the mobile information terminal of the embodiment described in Fig. 3, in which feature interior information indicating the internal state of the feature is displayed as a virtual reality object at a position inside the feature. 2 is an example of a diagram illustrating a field of view display screen on the mobile information terminal of the embodiment described in Fig. 3, in which feature interior information indicating the internal state of a specified feature is displayed as a virtual reality object. 3 is an example of a diagram illustrating a field of view display screen on the mobile information terminal of the embodiment described in Fig. 3, in which feature entrance mark information indicating the entrance of the feature is generated and displayed as a virtual reality object.
[0023] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The same components are designated by the same reference numerals throughout the drawings, and duplicated explanations will be omitted.
[0024] The mobile information terminal according to this embodiment can easily grasp the spatial positional relationships of features around the user, including features that are not directly visible from the user's viewpoint, thereby improving convenience. Therefore, the present invention can increase the commercial value of mobile information terminals to which the present invention is applied, and is therefore expected to contribute to Goal 8.2 of the Sustainable Development Goals (SDGs) advocated by the United Nations (increasing economic productivity through diversification, technological improvement, and innovation, particularly in industries that increase the value of goods and services and in labor-intensive industries).
[0025] Hereinafter, an example of an embodiment of the present invention will be described with reference to the drawings. As a specific example of a mobile information terminal according to the present embodiment, a so-called optical see-through head mounted display (HMD) will be mainly described as an example. The head mounted display is worn on the user's head and displays virtual reality objects on a display, while allowing the user to visually recognize real objects in front of the user as features on the ground.
[0026] FIG. 1 is a block diagram showing an example of the configuration of an HMD according to this embodiment.
[0027] In Figure 1, the HMD 100 is configured appropriately using a camera 101, a distance measurement sensor 102, a positioning sensor 103, an acceleration sensor 104, a gyro sensor 105, a geomagnetic sensor 106, an operation input interface 107, a display 108, a processor 120, a memory 130 that stores a program 131 and information data 132, a vibrator 141, a microphone 142, a speaker 143, a network communication device 144, and a short-range wireless communication device 145, and each component is connected to each other via a bus 150.
[0028] The network communication device 144 wirelessly exchanges information with an information server 161 that stores map information, internal state information of features, and other related information via an external network 160 .
[0029] The short-range wireless communication device 145 is placed near the HMD 100 without being worn by the user 10 (see Figure 3), such as a smartphone or personal computer, and wirelessly communicates information with a non-wearable information server terminal 170 that stores map information, information related to features, etc.
[0030] The camera 101 captures the field of view in front of the camera 101, i.e., the field of view of the user 10 wearing the HMD 100, and acquires the camera-captured image by converting light incident from the lens into an electrical signal using an image sensor. The HMD 100 can capture the position, orientation, and size of features such as buildings and roads in the field of view from the field of view image generated by capturing the field of view scenery in front of the user 10 captured by the camera 101. Therefore, when generating shape information related to the shape and state of features as a virtual reality object, the HMD 100 can use the field of view image captured by the camera 101 while referring to map information from the information server 161 and the non-wearable information server terminal 170.
[0031] The ranging sensor 102 is a sensor that can measure the distance and angle to an object and capture the shape of the object in three dimensions. Examples of such sensors include a LiDAR (Light Detection and Ranging) sensor that irradiates an object with laser light such as infrared light, measures the reflected scattered light, and analyzes and detects the distance to a distant object and the state of the object; a TOF (Time of Flight) sensor that measures the distance by irradiating a subject with pulsed light and measuring the reflection time for each pixel; and a millimeter-wave radar that emits millimeter-wave radio waves, captures the reflected waves, and detects the distance to the object and the state of the object. The HMD 100 can use the three-dimensional shape of the object captured by the ranging sensor 102 to more accurately generate a virtual reality object that represents the shape information of the object.
[0032] The positioning sensor 103 receives radio signals from GPS (Global Positioning System) satellites in the sky to detect the current location, and can identify the current location of the user 10 wearing the HMD 100. Using the current location information of the user 10 detected by the positioning sensor 103, the HMD 100 can refer to map information acquired from the information server 161 and the non-wearable information server terminal 170 to identify where on the map the user 10 is located.
[0033] The acceleration sensor 104 is a sensor that detects acceleration, which is a change in speed per unit time, and can detect movement, vibration, shock, etc. The acceleration sensor 104 provided in the HMD 100 can detect the tilt and direction of the HMD 100 worn by the user 10.
[0034] The gyro sensor 105 is a sensor that detects angular velocity in a rotational direction and can detect vertical, horizontal, and diagonal orientations. Therefore, the acceleration sensor 104 and the gyro sensor 105 can be used to detect the orientation, such as the tilt and direction, of the HMD 100.
[0035] The geomagnetic sensor 106 is a sensor that detects the Earth's magnetic force and detects the direction in which the HMD 100 is facing. A three-axis type that detects geomagnetism in the up-down direction in addition to the front-back and left-right directions can be used, and the movement of the HMD 100 can be detected by capturing changes in the geomagnetic field in response to the movement of the HMD 100. This allows the HMD 100 to detect and determine in which direction the user 10 wearing the HMD 100 is facing, and by referring to map information acquired from the information server 161 and the non-wearable information server terminal 170, it becomes possible to identify features within the field of view in the direction in which the user 10 is facing.
[0036] The operation input interface 107 is an input means, for example, using gaze, hands, a pointer, gestures, or even a keyboard, key buttons, touch keys, etc., and is used by the user 10 to set and input information that the user 10 wants to input. An input operation screen, such as an operation menu, may be displayed on the display screen of the display 108, and input operation information may be captured depending on the position on the input operation screen to which the gaze is directed, or a hand or a pointer may be displayed on the input operation screen and the hand or pointer may be operated using the operation input interface 107 to capture input operation information. Furthermore, in the case of input using a keyboard, key buttons, touch keys, etc., the keyboard, key buttons, touch keys, etc. may be provided in a position or shape within the HMD 100 that makes it easy for the user 10 to perform input operations, or may be separated from the main body of the HMD 100 and connected via a wire or wireless connection.
[0037] Examples of operation inputs include display selection operations such as whether to superimpose a virtual reality object showing the shape information of a feature on an image of the feature within the user's field of view, display it at the top of the display screen, or display it lowered to the user's eye level, display a miniature of the feature near the entrance to the feature, or display internal information of the feature that shows the internal state of the feature.
[0038] Examples of cases include when pointing to a virtual reality object of a target feature to display internal information about the feature that indicates its internal state, when the user 10 sets and inputs an area of interest or an area within a specified radius of a specific feature, or when setting and adding a mark to indicate the entrance to a feature.
[0039] In the case of an optical see-through HMD, the display 108 includes a projection unit that projects feature information and notification information to the user 10, and a transparent half mirror that forms an image of the projected information in front of the user's eyes. This allows the user 10 to visually recognize both the features in front of the user's eyes within the field of view and the formed projection information, as if they were floating as virtual reality objects.
[0040] In addition, in the case of a video see-through type HMD that is worn on the user's head and displays both real space and virtual space for viewing, it is composed of a display such as an LCD panel that displays both the features in front of the user photographed by camera 101 and virtual reality objects that show feature information, etc.
[0041] This allows the user 10 to view on the display an augmented reality space in which the field of view image and the virtual reality object, which is the feature information, are superimposed.
[0042] The processor 120 is composed of a CPU, etc., and controls each component of the HMD 100 by executing an operating system (OS) 133 stored in the memory 130 and application programs 134 for operation control, etc., and realizes functions of the OS, middleware, applications, etc., and other functions.
[0043] When processor 120 loads a virtual reality object display program, which is a type of application program 134, into the work area of memory 130 and executes it, a feature information virtual reality object generation processing unit 121, a feature information virtual reality object display processing unit 122, a feature miniature generation and display processing unit 123, a feature interior information generation and display processing unit 124, and a feature entrance mark generation and display processing unit 125 are configured, and each functional operation is realized.
[0044] The memory 130 is configured with a non-volatile storage device or the like, and stores various programs 131 and information data 132 that are handled by the processor 120 or the like. The information data 132 stores virtual reality object information 135 to be generated and displayed, field of view image information 136 captured by a camera, feature shape information 137 that indicates the shape of features, feature interior information 138 that indicates the internal state of features such as buildings, feature entrance mark information 139 that indicates the entrance of features such as buildings with a mark, and the like.
[0045] In this specification, feature information refers to information about a feature, including at least one of the feature shape information 137 and feature interior information 138. The feature information may include location information where the feature is located. Furthermore, the feature interior information may include attribute information of the feature, such as the type of feature, the name of the feature, the names of tenants occupying each floor of the feature, a map of each floor, and the business hours of the tenants. It may also include information indicating the location of the entrance to the feature.
[0046] The vibrator 141 generates vibrations under the control of the processor 120, and converts notification instruction information to the user 10 transmitted by the HMD 100 into vibrations. The vibrator 141 transmits vibrations to the head of the user wearing the HMD 100, thereby notifying the user 10 of the notification instruction information, thereby improving usability.
[0047] The microphone 142 collects external sounds and the user's own voice and converts them into audio data. The user 10 speaks a voice indicating an instruction such as an input operation, and the microphone 142 collects the voice and captures the instruction information such as the input operation, thereby enabling the user 10 to conveniently execute an operation in response to the instruction information.
[0048] The speaker 143 outputs sound based on the sound data, and can provide notification information to the user 10 by sound. For example, the speaker 143 can emit sound to notify the user that a display selection operation has been performed, such as whether to superimpose a virtual reality object on a feature image in the user's field of view or display it at the top of the display screen, thereby improving convenience.
[0049] The network communication device 144 is a communication interface that performs wireless communication with an external information server 161 via an external network 160 by base station communication or the like. The network communication device 144 can receive map information, feature information, and the like stored in the information server 161, and can also transmit and receive operation control signal information and the like to and from the information server 161. Note that, as the base station communication, long-distance wireless communication such as W-CDMA (Wideband Code Division Multiple Access) or GSM (Global System for Mobile communications) (registered trademark) may be used.
[0050] The short-range wireless communication device 145 is a communication interface that performs short-range wireless communication with a non-wearable information server terminal 170 that is within a short-range wireless communication range. As with the network communication device 144, the short-range wireless communication device 145 can receive map information, feature information, and the like stored in the non-wearable information server terminal 170, and can also transmit and receive operation control signal information and the like to and from the non-wearable information server terminal 170. Note that short-range wireless communication is performed using, for example, an electronic tag, but is not limited to this, and may be performed using Bluetooth (registered trademark), IrDA (Infrared Data Association, registered trademark), Zigbee (registered trademark), HomeRF (Home Radio Frequency, registered trademark), or wireless LAN (IEEE802.11a, IEEE802.11b, IEEE802.11g) as long as it is at least capable of wireless communication when the HMD 100 is near the non-wearable information server terminal 170.
[0051] In this specification, the non-wearable information server terminal 170 is a terminal that provides information to the HMD 100 and is responsible for part of the display control processing of virtual reality objects by the HMD 100, and specific examples of such terminals include mobile information terminals such as smartphones, tablet terminals, and laptop computers.
[0052] The feature information virtual reality object generation processing unit 121 generates virtual reality objects representing the shapes of features, such as buildings and roads, within the field of view based on the feature shape information 137. In this process, the orientation and size of the virtual reality object of the feature are changed, if necessary, to match the orientation and size of the viewed building. Based on map information acquired from the information server 161 or the non-wearable information server terminal 170, the current position of the user 10 detected by the positioning sensor 103 and the orientation of the user 10 detected by the acceleration sensor 104, gyro sensor 105, and geomagnetic sensor 106 are determined to identify features within the field of view of the user 10, and feature information for the identified features within the field of view is acquired from the map information, feature-related information, etc. Furthermore, the feature information for the identified features is acquired with greater accuracy using field of view images captured by the camera 101 and the three-dimensional shapes of the features captured by the ranging sensor 102.
[0053] The feature information virtual reality object display processing unit 122 displays the virtual reality object generated by the feature information virtual reality object generation processing unit 121 in a display area on the display screen of the display 108 associated with the area in which the feature is viewed. As an example of "displaying in a display area associated with the area in which the feature is viewed," the virtual reality object may be displayed superimposed on the viewing position of the corresponding feature. Alternatively, the virtual reality object generated by the feature information virtual reality object generation processing unit 121 may be displayed in an upper area corresponding to the display area at the top of the display screen corresponding to the feature position within the user's field of view. Furthermore, the virtual reality object may be displayed lowered to the eye level of the user 10, i.e., displayed in a partial area on the display screen centered on the user's eye level. Furthermore, the virtual reality object may be displayed in the entire area or in a limited area, such as an area of interest or an area within a predetermined radius, and show feature information about identified features within that area.
[0054] The feature miniature generation and display processing unit 123 generates a virtual reality object showing a miniature of a feature within the user's field of view based on the feature shape information 137, and displays the generated virtual reality object near the entrance to the feature.
[0055] The feature interior information generating and displaying processing unit 124 generates a virtual reality object (corresponding to an internal state virtual reality object) that indicates the internal state of the specified feature based on feature interior information 138 that represents the internal state of the specified feature. Furthermore, the generated virtual reality object is displayed at a corresponding internal position within the virtual reality object that represents the specified feature. Furthermore, when the user 10 pulls the specified feature or a virtual reality object representing that feature in a specific direction, the virtual reality object that indicates the generated feature interior information is displayed at the position in the specific direction that the user 10 pulled. Note that feature interior information can be acquired in the same manner as feature shape information is acquired.
[0056] The feature entrance mark generation and display processing unit 125 generates a virtual reality object (equivalent to an entrance mark virtual reality object) indicating a feature entrance mark based on feature entrance mark information 139 indicating the entrance of a specified feature such as a building, and displays the generated virtual reality object at a position corresponding to the entrance within the virtual reality object representing the specified feature.
[0057] With the above configuration, the HMD 100 acquires feature information of features such as buildings and roads within the field of view and generates a virtual reality object representing the acquired feature information by the feature information virtual reality object generation processing unit 121. The target feature is determined based on the position of the user 10 detected by the positioning sensor 103 and the orientation and tilt of the user 10 detected by the acceleration sensor 104, gyro sensor 105, and geomagnetic sensor 106 on map information obtained from the information servers 161 and 170 or internal information data, and feature information for the determined feature is acquired from the map information, feature-related information, etc., from the information servers 161 and 170 or internal information data.
[0058] Furthermore, in HMD 100, the feature information virtual reality object display processing unit 122 displays the virtual reality objects generated by the feature information virtual reality object generation processing unit 121 in the upper area of the display screen, superimposed on the viewing position of the corresponding feature or in a manner that corresponds to the position of the feature in the user's field of view. By displaying the generated virtual reality objects superimposed on the viewing position of the corresponding feature, user 10 can confirm the position of the real feature that is directly visible, while also being able to view features that are not visible from the user's viewpoint, i.e., features that are in the shadow of buildings, etc., by having them project through the foreground features, and can clearly recognize the entire view of the feature.
[0059] Furthermore, by displaying virtual reality objects in the upper region of the display screen, objects that were in the shadow of a building or the entire view of a feature can be viewed. When virtual reality objects are displayed in the upper region of the display screen, positional overlap between the virtual reality objects that indicate feature information of features within the field of view and the real features can be eliminated, thereby preventing the real features from interfering with the visibility of the virtual reality objects. Furthermore, when displaying virtual reality objects in the upper region, the virtual reality objects of features such as buildings are moved to the upper region as they are while maintaining their parallelism with the actual ground, and the height of the features such as buildings is reduced as necessary to generate and display the virtual reality objects.
[0060] Therefore, the user 10 can easily confirm the correspondence with the location of features within the user's field of view and visually recognize the generated and displayed virtual reality object in an easy-to-see manner. The virtual reality object may also be displayed in an upper region of the display screen, lowered to the eye level of the user 10, for easy viewing.
[0061] In addition, instead of displaying virtual reality objects across the entire field of view, virtual reality objects may be displayed in a limited area, such as an area of interest or an area within a specified radius, making it easier for user 10 to focus and view the area of interest.
[0062] For example, virtual reality objects based on feature information may be displayed within a radius of 50 meters from the user's current location. Furthermore, if multiple virtual reality objects exist on the same line of sight from the user's current location, a maximum overlap limit may be set in advance, and overlapping display of virtual reality objects exceeding that limit may be prohibited. This improves the visibility of the virtual reality objects.
[0063] In addition, in the HMD 100, the feature miniature generation and display processing unit 123 generates a miniature of the feature as a virtual reality object that indicates feature information of the feature within the user's field of view, and displays the generated miniature of the feature near the entrance to the feature. By indicating the feature information of the feature within the user's field of view with a feature miniature, the complexity of distinguishing between virtual reality objects and real features can be reduced.
[0064] As described above, by generating and displaying a virtual reality object showing feature information of a feature through the above-mentioned processing, it becomes possible to conveniently view the entire view of features within the field of view, including features that are not visible from the user's viewpoint, in a see-through manner.
[0065] Furthermore, in the HMD 100, the feature interior information generating and displaying processing unit 124 generates a virtual reality object that indicates the internal state of the feature based on the feature interior information 138. Furthermore, the generated virtual reality object is displayed at a corresponding internal position within the virtual reality object that indicates the feature for which the feature interior information is to be displayed. Furthermore, when the user 10 pulls the virtual reality object of the feature for which the feature interior information is to be displayed in a specific direction, the virtual reality object of the generated feature interior information is displayed at the position in the specific direction that the user 10 pulled. This makes it possible to see through the interior of a feature such as a building that cannot be seen directly, for example, the state of a store inside the building.
[0066] Furthermore, in the HMD 100, the feature entrance mark generating and display processing unit 125 displays a virtual reality object consisting of a feature entrance mark indicating the location of the entrance to a feature at a position corresponding to the feature entrance, based on the feature entrance mark information 139, thereby enabling the user 10 to easily find a location of interest, such as the entrance to a building. Note that marks indicating specific locations other than building entrances, such as a parking lot or emergency exit of a building, or a bus stop outside a building, may also be generated and displayed as virtual reality objects.
[0067] Thus, the HMD 100 can display the spatial positional relationships of features around the user in an easy-to-understand manner over a wide area, including features that cannot be seen directly from the user's viewpoint, and can also focus and clearly display the positional relationships of features in certain areas, such as areas of interest or areas within a specified radius, and can also clearly display the interior of the feature in question and the location of specific points on the feature.
[0068] Next, the basic operation of the mobile information terminal according to this embodiment will be described using a flowchart. Fig. 2A is an example of a flowchart explaining the basic operation of the mobile information terminal according to this embodiment. Fig. 2B is an example of a flowchart explaining the display process of the internal information of the feature and the entrance mark information when a virtual reality object showing the shape information of the feature is displayed in the basic operation of the mobile information terminal according to this embodiment.
[0069] In Figure 2A, when the HMD 100 displays the shapes of features within the field of view, such as buildings and roads, as virtual reality objects, it obtains feature shape information indicating the shapes of features within the field of view from an external information server 161, 170 or internal information data (S201).
[0070] Furthermore, the HMD 100 generates a virtual reality object that represents the shape of the feature within the acquired field of view (S202), and changes the orientation and size of the virtual reality object of the feature, if necessary, to match the orientation and size of the viewed building.
[0071] When the generated virtual reality object is displayed superimposed on the viewing position of the corresponding feature (S203: superimposed display), when the generated virtual reality object is displayed across the entire area within the field of view (S204: all areas), the virtual reality objects of all areas are displayed superimposed on the viewing position of the feature (S205).
[0072] When the generated virtual reality object is displayed in a limited area, such as an area of interest to the user 10 or an area within a predetermined radius (S204: partial area), the virtual reality object of the partial area is displayed superimposed on the feature viewing position (S206).
[0073] On the other hand, when the generated virtual reality object is displayed in the upper area of the display screen (S203: Display in upper area of display screen), and when the generated virtual reality object is displayed across the entire area within the field of view (S211: All areas), the virtual reality objects of all areas are displayed in the upper area of the display screen (S212).
[0074] When the generated virtual reality objects are displayed in a limited area, such as an area of interest to the user 10 or an area within a predetermined radius (S211: partial area), the virtual reality objects for the limited area are displayed in the upper area of the display screen (S213). If the display of these virtual reality objects continues (S214: Y), the process returns to the initial sequence S201. This allows new features to be updated while the virtual reality objects are still displayed.
[0075] On the other hand, if the display of these virtual reality objects is not to be continued (S214: N), the display of the virtual reality objects is stopped (S215) and this sequence flow is terminated.
[0076] Therefore, it is possible to recognize the overall spatial positional relationship of features around the user, including features that cannot be directly seen from the user's viewpoint, or to focus on a specific area, such as an area of interest or an area within a specified radius, and to concentrate on the positional relationship of features within that specific area.
[0077] Also, Figure 2B shows part of the processing of sequences S205, S206, S212, and S213 shown in Figure 2A, and as a representative example, we will explain the case of sequence S205, in which virtual reality objects of all areas are displayed superimposed on the feature viewing position.
[0078] 2B shows a case where a virtual reality object showing the shape of the feature in the entire area within the field of view is generated, and the virtual reality object is displayed superimposed on the feature viewing position (S221), and then feature interior information or feature entrance mark information is displayed (S222: Y). When the information to be displayed is feature interior information (S223: Display feature interior information), a virtual reality object showing the feature interior information is displayed at a position corresponding to the internal state of the feature or at a position in a specific direction pulled by the user 10 (S224).
[0079] If the display of the virtual reality object showing the feature interior information is to continue (S225: Y), this flow ends and the process returns to the flowchart of FIG. 2A, and the display of the virtual reality object showing the feature interior information continues.
[0080] When the display of the virtual reality object showing the feature interior information is finished (S225: N), the display of the virtual reality object showing the feature interior information is stopped (S226), and this flow ends.
[0081] On the other hand, when the information to be displayed is feature entrance mark information (S223: Display feature entrance mark information), a virtual reality object showing the feature entrance mark information is displayed at the entrance position of a feature such as a building (S234).
[0082] If the display of the virtual reality object indicating the feature entrance mark information is to continue (S235: Y), this flow ends and the process returns to the flow chart of FIG. 2A, and the display of the virtual reality object indicating the feature entrance mark information continues.
[0083] When the display of the virtual reality object showing the feature entrance mark information is finished (S235: N), the display of the virtual reality object showing the feature entrance mark information is stopped (S236), and this flow ends.
[0084] Furthermore, when the information to be displayed is both feature interior information and feature entrance mark information (S223: display both information), both a virtual reality object showing the feature interior information and a virtual reality object showing the feature entrance mark information are displayed (S244).
[0085] In this state, when the display of the virtual reality object showing the feature interior information is to be ended (S245: N, S246: Stop display of feature interior information), the display of the virtual reality object showing the feature interior information is stopped (S247).
[0086] Furthermore, when the display of the virtual reality object showing the feature entrance mark information is finished (S248: N), the display of the virtual reality object showing the feature entrance mark information is stopped (S249), and this flow is ended.
[0087] On the other hand, when the display of the virtual reality object showing the feature entrance mark information is to be ended from the state in which both are displayed (S245: N, S246: Display of feature entrance mark information stopped), the display of the virtual reality object showing the feature entrance mark information is stopped (S250).
[0088] Furthermore, when the display of the virtual reality object indicating the feature interior information is finished (S251: N), the display of the virtual reality object indicating the feature interior information is stopped (S252), and this flow is terminated. Note that this flow is terminated when the feature interior information and feature entrance mark information are not displayed (S222: N), when the display of the virtual reality object for both the feature interior information and the feature entrance mark information is continued (S245: Y), when the display of the virtual reality object indicating the feature entrance mark information is continued (S248: Y), or when the display of the virtual reality object indicating the feature interior information is continued (S251: Y).
[0089] Therefore, it is possible to see through and grasp the state of the inside of a feature such as a building that cannot be seen directly, and also to easily check the entrance of a feature such as a building.
[0090] It goes without saying that the same effect can be achieved with a similar sequence flow in the case of sequence S206, in which virtual reality objects of a certain area are superimposed on the feature viewing position, sequence S212, in which virtual reality objects of all areas are displayed in the upper area of the display screen, and sequence S213, in which virtual reality objects of a certain area are displayed in the upper area of the display screen.
[0091] Next, an embodiment of the portable information terminal according to the present embodiment will be described with reference to the drawings. Fig. 3 is an example of a diagram illustrating the appearance of an HMD 100 as a portable information terminal according to the present embodiment. Fig. 4 shows an example of a field of view display screen 400 when an optical see-through HMD is used in the embodiment shown in Fig. 3 and features such as buildings and roads in front of the user are directly viewed, with the outside world being visible through a half mirror.
[0092] In the case of a video see-through HMD, the field of view display screen 400 (see FIG. 4) can be obtained by displaying on the display a camera image of a feature within the field of view. Figures 5, 6A, 6B, 6C, and 6D show examples of the field of view display screen 400 in the embodiment shown in Figure 3, in which feature information indicating the shape of a feature such as a building is displayed as a virtual reality object together with the real feature. Figure 5 shows a case in which a virtual reality object is displayed superimposed on the viewing position of the real feature, while Figures 6A, 6B, 6C, and 6D show a case in which the virtual reality object is displayed in the upper region of the field of view display screen 400.
[0093] 3 , a user 10 positioned on a road in the center of the field of view is wearing an HMD 100 on his head. The HMD 100 is equipped with a camera 101 that captures the field of view ahead, a distance measurement sensor 102 that measures the distance and angle to features within the field of view and captures the shapes of the features in three dimensions, a positioning sensor 103 that detects the current position, an acceleration sensor 104 that detects the orientation and tilt of the user 10, a gyro sensor 105, a geomagnetic sensor 106, a network communication device 144 that wirelessly communicates with an external information server 161 via base station communication or the like, and a short-range wireless communication device 145 that wirelessly communicates with a non-wearable information server terminal 170.
[0094] The user 10 can see the following features in his or her field of vision ahead: a first building 301, a second building 302, a third building 303, a fourth building 304, a fifth building 305, a cylindrical sixth building 306, a seventh building 307, a swing 308 in the park, and a road 309 connecting these. Each building has an entrance marked "IN."
[0095] In the state shown in Figure 3, the field of view display screen 400 of the HMD 100 displays a road 309 in the center of the field of view display screen 400, as shown in Figure 4, with a first building 301, a second building 302, and a third building 303 on the left side of the road 309, and a fourth building 304, a fifth building 305, and a seventh building 307 on the right side of the road 309.
[0096] However, the swing 308 in the park in the shadow of the third building 303, the cylindrical sixth building 306 in the shadow of the fifth building 305, and even the full view of each building do not appear on the field of view display screen 400, and the user 10 is unable to grasp the spatial positional relationships of the buildings and other features within the field of view.
[0097] In contrast, as shown in Figure 5, HMD 100 generates feature information indicating the shapes of features such as buildings and roads as virtual reality objects, and displays the generated virtual reality objects (shown by dashed lines, dotted lines, dotted lines, and dotted lines) superimposed on the viewing position of the real feature.
[0098] In Figure 5, the virtual reality object of the first building 301 is the first virtual building 501 shown with a dotted line, the virtual reality object of the second building 302 is the second virtual building 502 shown with a dashed line, the virtual reality object of the third building 303 is the third virtual building 503 shown with a two-dot dashed line, the virtual reality object of the fourth building 304 is the fourth virtual building 504 shown with a dotted dashed line, the virtual reality object of the fifth building 305 is the fifth virtual building 505 shown with a dashed line, the virtual reality object of the cylindrical sixth building 306 is the sixth virtual building 506 shown with a dotted dashed line, the virtual reality object of the seventh building 307 is the seventh virtual building 507 shown with a two-dot dashed line, the virtual reality object of the swing 308 in the park is the first virtual swing 508 shown with a dotted line, and the virtual reality object of the road 309 is the first virtual road 509 shown with a dashed line.
[0099] As shown in Figure 5, the HMD 100 displays the generated virtual reality object superimposed on the viewing position of the real feature, allowing the user 10 to view the entire view of the feature at the position of the real feature that is directly visible, and also allowing the user 10 to clearly see objects that are in the shadow of the feature, such as buildings.
[0100] Thus, the user 10 can visually recognize the spatial positional relationships of features around the user, including features that are not directly visible from the user's viewpoint, in a wide, easy-to-understand manner. Note that feature information indicating the shapes of features within the field of view is obtained by determining the current position of the user 10 detected by the positioning sensor 103 and the facing direction of the user 10 detected by the acceleration sensor 104, gyro sensor 105, and geomagnetic sensor 106 on map information acquired from the information server 161 or the non-wearable information server terminal 170, thereby identifying features within the user's field of view and acquiring shape information of the identified features from the map information, etc.
[0101] In addition, virtual reality objects of features may be generated by enlarging or reducing them in the vertical or horizontal directions, thereby reducing overlap with the real features that are displayed on top of them and making the virtual reality objects easier to view.
[0102] Figure 6A shows a case where the generated virtual reality object (shown by dashed lines, dotted lines, dotted lines, and dotted lines) is displayed in the upper area of the field of view display screen 400, as if drawing a bird's-eye view of the features within the user's field of view, as if looking down diagonally from above.
[0103] In FIG. 6A , the virtual reality object of the first building 301 is an eleventh virtual building 611 shown by a dashed line, the virtual reality object of the second building 302 is a twelfth virtual building 612 shown by a broken line, the virtual reality object of the third building 303 is a thirteenth virtual building 613 shown by a two-dot dash line, the virtual reality object of the fourth building 304 is a fourteenth virtual building 614 shown by a dashed line, and the virtual reality object of the fifth building 305 is a fourteenth virtual building 615 shown by a dashed line. The virtual reality object of the sixth cylindrical building 306 is the 15th virtual building 615 shown in a dashed line, the virtual reality object of the sixth cylindrical building 306 is the 16th virtual building 616 shown in a dashed line, the virtual reality object of the seventh building 307 is the 17th virtual building 617 shown in a dashed line, the virtual reality object of the swing 308 in the park is the second virtual swing 618 shown in a dotted line, and the virtual reality object of the road 309 is the second virtual road 619 shown in a dashed line.
[0104] 6A , HMD 100 displays the generated virtual reality object in the upper region of field of view display screen 400, thereby allowing user 10 to view the entire view of the feature while avoiding positional overlap between the real feature and the virtual reality object display in the field of view, and also allowing user 10 to clearly see objects that are in the shadow of the feature, such as a building. In this case, HMD 100 displays the virtual reality object of a distant feature in the upper region of field of view display screen 400, allowing user 10 to view the virtual reality object in a manner that makes it easier to see the distant feature.
[0105] Figure 6B shows a case in which a virtual ground surface is assumed to be parallel to the actual ground surface, and virtual reality objects (shown by dashed lines, dotted lines, dotted lines, and dotted lines) are created by translating features such as roads and buildings on the actual ground surface in real space and placing them on the virtual ground surface, and then displaying them in the upper area of the field of view display screen 400.
[0106] In FIG. 6B , the virtual reality object of the first building 301 is a 21st virtual building 621 shown in a dashed line, the virtual reality object of the second building 302 is a 22nd virtual building 622 shown in a broken line, the virtual reality object of the third building 303 is a 23rd virtual building 623 shown in a two-dot dash line, the virtual reality object of the fourth building 304 is a 24th virtual building 624 shown in a dashed line, and the virtual reality object of the fifth building 305 is a 25th virtual building 625 shown in a dashed line. The virtual reality object of the sixth cylindrical building 306 is the 26th virtual building 626 shown in a dashed line, the virtual reality object of the seventh building 307 is the 27th virtual building 627 shown in a dashed line, the virtual reality object of the swing 308 in the park is the third virtual swing 628 shown in a dotted line, and the virtual reality object of the road 309 is the third virtual road 629 shown in a dashed line.
[0107] In Figure 6B, the user 10 stands on the actual ground and looks down at a virtual reality object placed in the air. In this case, the bottom of the virtual building in the air is visible. The height of the virtual ground surface may be set by the user 10 as appropriate, and the height of the building or other feature may be reduced or enlarged as needed.
[0108] Therefore, by displaying the generated virtual reality object in the upper area of the field of view display screen 400, the HMD 100 allows the user 10 to clearly see the entire view of the feature and objects in the shadow of the feature while avoiding positional overlap between the real feature in the field of view and the virtual reality object display, as in the case shown in Figure 6A.
[0109] Since the HMD 100 displays virtual reality objects of distant features in the upper area of the field of view display screen 400, the user 10 can view the virtual reality objects with the same perspective as when viewing actual features. In addition, since features such as buildings are translated upward, the correspondence between the virtual reality objects and the positions of the features in the user's field of view can be easily recognized.
[0110] Figure 6C shows a case in which a virtual mirror parallel to the actual ground surface is assumed, and virtual reality objects (shown by dashed lines, dotted lines, dotted lines, and dotted lines) are created in the sky in such a way that features such as roads and buildings on the actual ground surface are reflected in the mirror, and are displayed in the upper area of the field of view display screen 400.
[0111] In FIG. 6C , the virtual reality object of the first building 301 is the 31st virtual building 631 shown in a dashed line, the virtual reality object of the second building 302 is the 32nd virtual building 632 shown in a broken line, the virtual reality object of the third building 303 is the 33rd virtual building 633 shown in a two-dot dash line, the virtual reality object of the fourth building 304 is the 34th virtual building 634 shown in a dashed line, and the virtual reality object of the fifth building 305 is the 35th virtual building 635 shown in a dashed line. The virtual reality object of the sixth cylindrical building 306 is the 36th virtual building 636 shown in a dashed line, the virtual reality object of the seventh building 307 is the 37th virtual building 637 shown in a dashed line, the virtual reality object of the swing 308 in the park is the fourth virtual swing 638 shown in a dotted line, and the virtual reality object of the road 309 is the fourth virtual road 639 shown in a dashed line.
[0112] In Figure 6C, user 10 stands on the actual ground surface and views virtual reality objects placed in the sky from above, the opposite of Figure 6B. In this case, the rooftops of virtual buildings in the sky are visible. Note that user 10 may set the height of the virtual ground surface as appropriate, and may reduce or increase the height of features such as buildings as needed.
[0113] Therefore, by displaying the generated virtual reality object in the upper region of the field of view display screen 400, the user 10 can clearly see the entire view of the feature and objects hidden in the shadow of the feature while avoiding positional overlap between the real feature in the field of view and the virtual reality object display, as in the cases shown in Figures 6A and 6B. Because the HMD 100 displays the virtual reality object of a distant feature in the upper region of the field of view display screen 400, the user 10 can view the virtual reality object with the same perspective as when viewing the actual feature. Furthermore, because a mirror image of a feature such as a building is displayed in the sky, and its position on the ground surface is simply translated upward, the correspondence between the virtual reality object and the feature position in the user's field of view can be easily recognized.
[0114] Figure 6D shows a case where the generated virtual reality object is lowered to eye level compared to the virtual reality object display shown in Figure 6A. By displaying the virtual reality object at eye level, user 10 can easily view the virtual reality object. It goes without saying that the same effect can be achieved even if the virtual reality object is lowered to eye level compared to the virtual reality object display shown in Figures 6B and 6C. Conversely, to prioritize visibility, the virtual reality object of the feature may be displayed at eye level from the beginning, and when the user has finished viewing it or wishes to move it away from eye level, the user may move it to an upper area by operating the device. Furthermore, the device may detect that the user is looking away from the virtual reality object and automatically adjust the display position of the virtual reality object to an upper area so that it does not overlap with the real feature display.
[0115] As described above, in the embodiments shown in Figures 6A, 6B, 6C, and 6D, as in the case shown in Figure 5, the user 10 can visually recognize the spatial positional relationships of features around the user, including features that are directly visible from the user's viewpoint and features that are not directly visible, in an easy-to-understand manner over a wide area.
[0116] The virtual reality object corresponding to the feature drawn in the upper area of the field of view display screen 400 is displayed as if it exists at a fixed position in the outside world unless an operation to change the display position is performed. As an example of the processing and data structure for realizing this display, coordinates (x, y, z) indicating the display position of the virtual reality object representing the feature existing in the real world are used. v ,y v ,z v ) (where x is the left-right direction, y is the depth direction, and z is the height direction) are expressed as the position coordinates (x r ,y r ,z r ) is used to define equation (1). v ,y v ,z v ) = (x r ,y r ,z r +Δz) (1) Δz: a value indicating the lift height when the virtual reality object is lifted and displayed in the information area of the real feature
[0117] As a result, the positional relationship between the actual building and the virtual building is maintained even if the user 10 changes the direction of their face or moves on the actual ground surface while the virtual reality object is displayed in the upper region of the field of view display screen 400. When a bird's-eye view is displayed in the upper region of the field of view display screen 400 as shown in Figure 6A, the bird's-eye view direction and bird's-eye view point are changed in accordance with changes in the direction of the user 10's face or movement to a different location. Furthermore, when displaying virtual reality objects, the display color or transparency of the features to be displayed may be changed for each type, and the display may be turned on or off for each type, making it easier to distinguish between the features.
[0118] As an example of the process for displaying a mirror image of a virtual reality object in the upper region as shown in FIG. 6C, the virtual reality object is considered to be in a symmetrical position with respect to the xy plane, the z value in the height direction is converted to a negative value, and a value indicating the lift height is added (Equation (2)). (x v ,y v ,z v ) = (x r ,y r ,-z r+Δz) ... (2)
[0119] Next, using figures, we will explain how to narrow down the processing area to a specific area, such as an area of interest to the user 10 or an area within a specified radius from a specific position, and display feature information for features within that area as a virtual reality object.
[0120] Figures 7A and 7B are examples of displaying a virtual reality object by narrowing the partial area to an area of interest to user 10, and Figure 7C is an example of displaying a virtual reality object by narrowing the partial area to an area within a predetermined radius from a specific position. In Figures 7A, 7B, and 7C, parts with the same symbols as those shown in Figures 4, 5, 6A, 6B, and 6C have the same operations as those already described, so their description will be omitted.
[0121] 7A shows an example in which the area of interest to user 10 is an area surrounded by a frame line 700 on the left side of the field of view as seen from user 10. HMD 100 displays first virtual building 501, second virtual building 502, third virtual building 503, first virtual swing 508, and the left side of virtual road 509 within frame line 700 as virtual reality objects.
[0122] The area indicated by the frame 700 may be specified by the user encircling a partial area with his / her hand, and the operation input interface 107 of the HMD 100 recognizing the gesture and accepting input of the partial area.
[0123] In Figure 7B, as in the case shown in Figure 7A, the area of interest to the user 10 is the area surrounded by a frame line 700, but the HMD 100 displays a virtual reality object showing the shape information of the features within this area in the upper area of the field of view display screen 400.
[0124] 7B , the virtual reality object of the first building 301 is the 71st virtual building 701 shown by a dotted line, the virtual reality object of the second building 302 is the 72nd virtual building 702 shown by a dashed line, the virtual reality object of the third building 303 is the 73rd virtual building 703 shown by a two-dot dashed line, the virtual reality object of the swing 308 in the park is the fifth virtual swing 708 shown by a dotted line, and the virtual reality object of the left side of the road 309 is the left side of the fifth virtual road 709 shown by a dashed line. In this case, positional overlap with real features is eliminated, and the virtual reality objects can be displayed while avoiding visual obstruction to the virtual reality objects by the real features.
[0125] In addition, the virtual reality object may be enlarged for easier viewing. The superimposed virtual reality object shown in FIG. 7A may also be displayed. When displayed in the upper area, the display position may be directly in front of the user, as shown in FIG. 7B, or may be above the actual object so as to correspond to the position of the actual object.
[0126] 7C shows an example in which the partial area for displaying feature information is an area surrounded by a frame line 710 within a predetermined radius from the position of the user 10. The HMD 100 displays, within the frame line 710, parts of the first virtual building 501, the second virtual building 502, the third virtual building 503, the fourth virtual building 504, the fifth virtual building 505, the cylindrical sixth virtual building 506, and the virtual road 509 as virtual reality objects.
[0127] In this case, virtual reality objects that are far from the position of user 10 are hidden, and the virtual reality objects around user 10 are expected to stand out and become easier to see. The display area may be set by section, such as up to two sections away or only the third section away, or only a specified number of features may be displayed in order of proximity to the current location, or a category of features, such as convenience stores and hospitals, may be selected and displayed.
[0128] Furthermore, the specific position does not necessarily have to be the position of the user 10. For example, the position may be specified by the operation input interface 107. Furthermore, when displaying a virtual building, if even a part of the building is within the display area, the entire virtual building may be displayed as a virtual reality object.
[0129] As explained above, by narrowing down the area to a specific area, such as an area of interest to user 10 or an area within a specified radius from a specific location, and displaying the shapes of features within that area as virtual reality objects, user 10 can focus on and view the relative positions of features within the area of interest.
[0130] Next, a case where a miniature of a feature is generated and displayed as a virtual reality object representing feature information of the feature will be described with reference to the drawings. Fig. 8 shows an example where a miniature of a feature is generated as a virtual reality object representing the shape of the feature in the user's field of view, and the generated miniature of the feature is displayed near the entrance of the feature.
[0131] 8, the parts shown in FIG. 4 and having the same reference numerals have the same operations as those already explained, and therefore the explanation thereof will be omitted. In FIG. 8, the virtual reality object generated as a miniature of the first building 301 is the 81st virtual building 801 shown by a dashed line, the virtual reality object generated as a miniature of the second building 302 is the 82nd virtual building 802 shown by a broken line, the virtual reality object generated as a miniature of the third building 303 is the 83rd virtual building 803 shown by a two-dot dashed line, the virtual reality object generated as a miniature of the fourth building 304 is the 84th virtual building 804 shown by a dashed line, The virtual reality object generated as a miniature of the fifth building 305 is the 85th virtual building 805 shown in dashed lines, the virtual reality object generated as a miniature of the sixth cylindrical building 306 is the 86th virtual building 806 shown in dashed lines, the virtual reality object generated as a miniature of the seventh building 307 is the 87th virtual building 807 shown in dashed lines, and the virtual reality object generated as a miniature of the swing 308 in the park is the sixth virtual swing 808 shown in dotted lines.
[0132] 8 , the HMD 100 displays, as virtual reality objects indicating shape information of features within the user's field of view, miniatures of real features, such as an 81st virtual building 801, an 82nd virtual building 802, an 83rd virtual building 803, an 84th virtual building 804, an 85th virtual building 805, an 86th virtual building 806, an 87th virtual building 807, and a sixth virtual swing 808, near the entrances of the features, for example. Since the user 10 normally enters through the entrances of features, the entrances of features are the most easily recognizable feature locations for the user 10, and this shows an example of displaying miniatures of features near the entrances of the features.
[0133] In the case of swing 308 in a park, sixth virtual swing 808 may be displayed at the position of swing 308. Therefore, by displaying a virtual reality object that represents feature information of a feature within the user's field of view as a miniature of the feature at a feature position that is most easily recognizable to user 10, user 10 can easily and uncomplicatedly identify the miniature-sized virtual reality object that is displayed superimposed on the position of the real feature.
[0134] In addition to miniatures of features, original characters or logos representing the features may be created and displayed as virtual reality objects. The display position is also not limited to near the entrance, and may be, for example, on a rooftop, in the center of a building, or on a wall facing the street where the user 10 is located.
[0135] Next, a case where feature interior information indicating the internal state of a feature is generated as a virtual reality object and displayed in conjunction with a virtual reality object indicating feature information will be described with reference to the drawings. Fig. 9A shows an example where a virtual reality object indicating the generated feature interior information is displayed at a corresponding internal position within the virtual reality object of the feature information of the feature for which the feature interior information is to be displayed. Note that virtual reality objects indicating feature information other than the target for which the feature interior information is to be displayed are not shown. Fig. 9B shows an example where, when a user 10 pulls the virtual reality object of the feature information of the feature for which the feature interior information is to be displayed in a specific direction, a virtual reality object indicating the generated feature interior information is displayed at a position in the specific direction of the pull.
[0136] 9A and 9B, the parts denoted by the same reference numerals as those shown in FIGS. 4, 5 and 6A have the same operations as those already explained, and therefore the explanation thereof will be omitted.
[0137] 9A , the HMD 100 acquires feature interior information representing the internal state of the first building 301 from information stored in the information server 161 or the non-wearable information server terminal 170, or from information stored in the HMD 100, in a similar manner to the feature information acquisition process. For example, store information indicating the stores located within the first building 301 is acquired as the feature interior information of the first building 301. The acquired store information indicates that, relative to the building entrance, the first building 301 is arranged with Boutique A on the front side of the first floor, Cake Shop B on the back side of the first floor, Bookstore C on the front side of the second floor, Florist D on the back side of the second floor, Restaurant E on the front side of the third floor, and Sportswear Store F on the back side of the third floor.
[0138] Based on the location information of each store, the HMD 100 generates virtual reality objects 901, 902, 903, 904, 905, and 906 indicating feature interior information for representing the location of the stores, and displays the virtual reality objects indicating the feature interior information at corresponding interior positions within the virtual building 501, as shown in FIG. 9A . The virtual reality objects indicating feature interior information are not limited to store location information. For example, virtual reality objects may be created using text or illustrations to represent the store name, products sold, sale information, etc., and displayed together with the virtual reality objects indicating the feature interior information for representing the store location. Thus, the user 10 can see through the interior of a feature such as a building that cannot be seen directly, for example, the state of the stores within the building, from that position. Note that virtual reality objects indicating feature information for features other than those for which feature interior information is displayed may also be displayed.
[0139] Also, as shown in Figure 9B, the HMD 100 generates the contents of the layout status of each store as a virtual reality object 910, and when the user 10 pulls downward the 11th virtual building 611, which is the virtual reality object of the first building 301, the virtual reality object 910 of the internal information of the feature generated is displayed at the position in the downward direction of the pull.
[0140] Therefore, when user 10 wants to know the interior of a feature such as a building, for example, the status of stores within the building, the user 10 can easily view the interior information of the feature with a simple operation. While the above example illustrates pulling downward, the pulling direction may be selected to make the generated virtual reality object easier to view. While the above example focuses on the interior status of a building as feature interior information, an example of indoor use is a floor map display of a shopping mall. When displayed in the upper area, the display position can be determined based on the floor ceiling. Furthermore, when a virtual reality object showing interior information of a feature is touched, the feature and its virtual reality objects may be highlighted. Information displayed by a virtual reality object is not limited to information related to stores within the feature. Information about the feature itself may be generated as a virtual reality object and displayed linked to the virtual building. Furthermore, the display of interior information may be turned on and off for each building.
[0141] In the example shown in FIG. 10, a mark indicating a specific location of a feature is generated as a virtual reality object and displayed at a position corresponding to the specific location in virtual space. The example shown in FIG. 10 illustrates a case in which a mark indicating an entrance to a feature is generated as a virtual reality object and the generated virtual reality object is displayed at the corresponding entrance position of the feature.
[0142] In FIG. 10, the parts shown in FIGS. 4 and 6A and denoted by the same reference numerals have the same operations as those already explained, and therefore the explanation thereof will be omitted.
[0143] In Figure 10, HMD 100 generates entrance marks indicating the entrances to the first building 301, the second building 302, the third building 303, the fourth building 304, the fifth building 305, the sixth building 306, and the seventh building 307 as virtual reality objects 1001, 1002, 1003, 1004, 1005, 1006, and 1007, and displays the generated virtual reality objects 1001, 1002, 1003, 1004, 1005, 1006, and 1007 at the entrance positions of the eleventh virtual building 611, the twelfth virtual building 612, the thirteenth virtual building 613, the fourteenth virtual building 614, the fifteenth virtual building 615, the sixteenth virtual building 616, and the seventeenth virtual building 617, respectively. Thus, the user 10 can easily find the entrance location of the building.
[0144] In Figure 10, the entrance to a building is used as an example of a specific location on a geographical feature, but marks indicating locations that the user 10 wants to know about, such as a building's parking lot, emergency exit, or a bus stop outside of a building, may also be generated and displayed as virtual reality objects.
[0145] In the embodiments described above, the displayed virtual reality object is composed of a building and a road. However, it is not necessary to display both the building and the road; only one of them may be displayed. For example, a virtual reality object of only the road may be generated and displayed without displaying the building. Furthermore, in this case, a virtual reality object related to information about the building itself, such as the building's name, may be displayed at the position where the virtual reality object for the building is displayed, and a virtual reality object such as a 2D map may be displayed.
[0146] In the above description, an HMD has been used as a specific example of a mobile information terminal, but all devices with similar functions are applicable. It goes without saying that similar functions and effects can be achieved with mobile information terminals, such as smartphones and tablets, that allow a user 10 to visually perceive a forward field of view and generate and display information about features within the user's field of view as virtual reality objects. Furthermore, shape information and internal information of virtual features may be generated as virtual reality objects for virtual features in a virtual space, and the generated virtual reality objects may be superimposed on the virtual features in the virtual space or displayed in an area above the virtual feature display. In this case, a video see-through HMD, such as VR goggles, would be used.
[0147] The present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.
[0148] Furthermore, the numerical values, messages, etc. appearing in the text and figures are merely examples, and the effects of the present invention will not be impaired even if different ones are used.
[0149] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations, functions, etc. may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the program, table, and file that implements each function may be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD, or may be stored in a device on a communication network.
[0150] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected.
[0151] The above embodiment includes the following aspects.
[0152] (Supplementary Note 1) A portable information terminal comprising: a camera that captures a user's field of view and outputs a field of view image; a display; and a processor, wherein the processor identifies, based on the field of view image, at least one of features that are within the user's field of view and that the user can directly see, and features that are within the user's field of view but that the user cannot directly see; obtains feature information including shape information of the identified feature; generates a virtual reality object that indicates the shape of the identified feature based on the feature information; and displays the virtual reality object in a display area on the display screen associated with the area in which the identified feature is visible.
[0153] (Supplementary Note 2) A display control method for virtual reality objects, comprising: a feature identification step of capturing a user's field of view and identifying, based on the field of view image, at least one of a feature that is within the user's field of view and can be directly seen by the user and a feature that is within the user's field of view but cannot be directly seen by the user; a feature information acquisition step of acquiring feature information including shape information of the identified feature; a virtual reality object generation step of generating a virtual reality object that indicates the shape of the identified feature based on the feature information; and a display control step of displaying the virtual reality object in a display area on a display screen associated with an area in which the identified feature is visible.
[0154] 10: User 100: HMD 101: Camera 102: Distance measurement sensor 103: Positioning sensor 104: Acceleration sensor 105: Gyro sensor 106: Geomagnetic sensor 107: Operation input interface 108: Display 120: Processor 121: Feature information virtual reality object generation processing unit 122: Feature information virtual reality object display processing unit 123: Feature miniature generation and display processing unit 124: Feature interior information generation and display processing unit 125: Feature entrance mark generation and display processing unit 130: Memory 131: Program 132: Information data 134: Application program 135: Virtual reality object information 136: View image information 137: Feature shape information 138: Feature interior information 139: Feature entrance mark information 141: Vibrator 142: Microphone 143: Speaker 144: Network communication device 145: Near field communication device 150: Bus 160: External network 161: Information server 170: Non-wearable information server terminal 301: First building 302: Second building 303: Third building 304: Fourth building 305: Fifth building 306: Sixth building 307: Seventh building 308: Swing 309: Road 400: View display screen 501: First virtual building 502: Second virtual building 503: Third virtual building 504: Fourth virtual building 505: Fifth virtual building 506: Sixth virtual building 507: Seventh virtual building 508 : First virtual swing 509 : First virtual road 611 : Eleventh virtual building 612 : Twelfth virtual building 613 : Thirteenth virtual building 614 : Fourteenth virtual building 615 : Fifteenth virtual building 616 : Sixteenth virtual building 617 : Seventeenth virtual building 618 : Second virtual swing 619 : Second virtual road 621 : Twenty-first virtual building 622 : Twenty-second virtual building 623 : Twenty-third virtual building 624 : Twenty-fourth virtual building 625 : Twenty-fifth virtual building 626 : Twenty-sixth virtual building 627 : Twenty-seventh virtual building628: Third virtual swing 629: Third virtual road 631: Thirty-first virtual building 632: Thirty-second virtual building 633: Thirty-third virtual building 634: Thirty-fourth virtual building 635: Thirty-fifth virtual building 636: Thirty-sixth virtual building 637: Thirty-seventh virtual building 638: Fourth virtual swing 639: Fourth virtual road 700: Frame line 701: Seventy-first virtual building 702: Seventy-second virtual building 703: Seventy-third virtual building 708: Fifth virtual swing 709: Fifth virtual road 710: Frame line 801: Eighty-first virtual building 802: Eighty-second virtual building 803: Eighty-third virtual building 804 : 84th virtual building 805 : 85th virtual building 806 : 86th virtual building 807 : 87th virtual building 808 : 6th virtual swing 901 : Virtual reality object 902 : Virtual reality object 903 : Virtual reality object 904 : Virtual reality object 905 : Virtual reality object 906 : Virtual reality object 910 : Virtual reality object 1001 : Virtual reality object 1002 : Virtual reality object 1003 : Virtual reality object 1004 : Virtual reality object 1005 : Virtual reality object 1006 : Virtual reality object 1007 : Virtual reality object
Claims
1. A portable information terminal comprising: a camera that captures a user's field of view and outputs a field of view image; a display; and a processor, wherein the processor identifies, based on the field of view image, at least one of features that are within the user's field of view and that the user can directly see, and features that are within the user's field of view but cannot directly see, obtains feature information including shape information of the identified features, generates a virtual reality object indicating the shape of the identified feature based on the feature information, and displays the virtual reality object in a display area on the display screen associated with the area in which the identified feature is visible.
2. A portable information terminal as claimed in claim 1, wherein the processor causes the virtual reality object to be superimposed on the area in which the identified feature is visible on the display screen of the display.
3. A portable information terminal as described in claim 1, wherein the processor causes the virtual reality object to be displayed in an upper area located above the area in which the identified feature is visible on the display screen of the display.
4. A portable information terminal as described in claim 3, wherein the processor generates the virtual reality object having a shape that reduces the height of the identified feature, and displays the virtual reality object in the upper area on the display screen of the display while maintaining it parallel to the ground of the display screen.
5. A portable information terminal as described in claim 3, wherein the processor causes the virtual reality object to be displayed on the display screen within a partial area centered on the user's line of sight.
6. A portable information terminal as described in claim 1, wherein the processor causes only the virtual reality object that is within a portion of the field of view image specified by the user to be displayed on the display screen of the display.
7. A portable information terminal as described in claim 1, wherein the processor causes only a predetermined number of the virtual reality objects or less to be displayed on the display screen of the display.
8. A portable information terminal as described in claim 1, wherein the processor generates the virtual reality object showing a miniature of the identified feature, and displays the virtual reality object near an entrance to the identified feature.
9. A portable information terminal as described in claim 1, wherein the processor further generates an internal state virtual reality object indicating the internal state of the identified feature, and displays the internal state virtual reality object superimposed on a corresponding internal position within the virtual reality object on the display screen of the display.
10. A portable information terminal as described in claim 1, wherein the processor further generates an internal state virtual reality object indicating the internal state of the identified feature, and when an operation of pulling the virtual reality object indicating the identified feature in a specific direction is detected, the processor displays the internal state virtual reality object in a display area corresponding to the specific direction, based on the display position at which the virtual reality object of the identified feature is displayed on the display screen of the display.
11. A portable information terminal as described in claim 1, wherein the processor further generates an entrance mark virtual reality object indicating the entrance to the identified feature, and displays the entrance mark virtual reality object at a position corresponding to the entrance of the virtual reality object on the display screen of the display.
12. A display control method for virtual reality objects, comprising: a feature identification step of capturing an image of a user's field of view and identifying, based on the field of view image, at least one of a feature that is within the user's field of view and can be directly viewed by the user and a feature that is within the user's field of view but cannot be directly viewed by the user; a feature information acquisition step of acquiring feature information including shape information of the identified feature; a virtual reality object generation step of generating a virtual reality object indicating the shape of the identified feature based on the feature information; and a display control step of displaying the virtual reality object in a display area on a display screen associated with the area in which the identified feature is viewed.
13. A display control method for a virtual reality object as described in claim 12, wherein the display control step comprises displaying the virtual reality object superimposed on an area on the display screen of the display in which the identified feature is visible.
14. A display control method for a virtual reality object as described in claim 12, wherein the display control step displays the virtual reality object in an upper area located above an area in which the identified feature is visible on the display screen of the display.
Citation Information
Patent Citations
On-vehicle information provision apparatus
JP2005098916A
Guidance point guiding information providing system
JP2010060712A
Head-mounted display, control method thereof, computer program for controlling head-mounted display, and recording medium with computer program recorded thereon
JP2015069362A
Controller, electronic apparatus, control method, and control program
JP2019174548A
Display device, display method, and vehicle
JP2021148691A
