Head-mounted display and portable information terminal

Abstract

The present invention provides a head-mounted display that can enable a user to view virtual reality content, the head-mounted display being characterized by comprising a processor, wherein the processor acquires a desired destination of a person other than the user, generates a two-dimensional schematic diagram including the user and the person other than the user, adds to the two-dimensional schematic diagram a recommended movement path to the desired destination, and provides the two-dimensional schematic diagram with the recommended movement path added thereto to a portable information terminal carried by the person other than the user.

Description

Head-mounted displays and mobile information terminals

[0001] The present invention relates to a head-mounted display and a portable information terminal.

[0002] A virtual space image is displayed on a head-mounted display (sometimes referred to as an HMD) to allow users to view and experience VR (Virtual Reality) content, such as games, that unfold in a virtual space. At this time, a user wears the HMD on their head to limit their view of the real space and concentrate on the VR content.

[0003] Generally, when a user is experiencing VR content, they may move slowly or violently in accordance with the VR content. If there are real objects (furniture, walls, etc.) in the real space, the user's movements may cause a collision or other problem, forcing the user to interrupt the VR content viewing experience, and even causing damage to the real objects or injury to the user. Furthermore, in places where there are external people (people other than the user in the same real space as the user experiencing the content), such as in a living room at home, there is a problem that the external people may not be able to predict the user's movements and may approach the user, disrupting the user's VR content viewing or even injuring the external people.

[0004] To address the above-mentioned problem, a known technique is disclosed in Japanese Patent Application Laid-Open No. 2017-119031. In this known technique, a play area is set in which a user can view and experience VR content, and when the user approaches the boundary of the play area or leaves the play area, the VR content is switched or interrupted, prompting the user to avoid collision with real objects.

[0005] However, in the prior art, when a user is viewing VR content, virtual objects unrelated to the VR content are displayed or the VR content is interrupted, which hinders the user's viewing experience of the VR content. Furthermore, the prior art does not take into consideration moving outsiders.

[0006] Japanese Patent Application Laid-Open No. 2017-119031

[0007] Therefore, in the relationship between a user and an external person, the challenge is to provide a head-mounted display and a mobile information terminal that provide the external person with recommended travel route information that takes into account the user's play area, and enable the external person to move around without interfering with the user's VR content viewing experience.

[0008] In order to solve the above problem, for example, the following HMD is provided. This HMD allows a user to view virtual reality content. The HMD includes a processor. The processor acquires a desired destination of a person other than the user. The processor generates a two-dimensional schematic diagram including the user and the person other than the user. The processor assigns a recommended travel route to the desired destination to the two-dimensional schematic diagram. The processor provides data of the two-dimensional schematic diagram with the assigned recommended travel route to a mobile information terminal carried by the person other than the user.

[0009] Also, for example, the following mobile information terminal is provided. This mobile information terminal includes a processor. The processor is capable of performing a process for creating a travel route and a display process. In the process for creating a travel route, the processor acquires a three-dimensional spatial relationship diagram of real space from a head-mounted display on which virtual reality content is viewed, generates a two-dimensional schematic diagram from the three-dimensional spatial relationship diagram, and assigns a recommended travel route to the two-dimensional schematic diagram. In the display process, the processor displays the two-dimensional schematic diagram to which the recommended travel route has been assigned.

[0010] According to the present invention, it is possible to display recommended travel routes to a desired destination that have a low probability of collision with the user on an outside person's mobile information terminal, thereby preventing interference with the user's VR content viewing experience.

[0011] 1 is a block diagram showing an example of an HMD. FIG. 2 is an external view of an example of an HMD. FIG. 3 is a view showing an example of the relationship between a user and an external person. FIG. 4 is a view corresponding to FIG. 3 and showing an example of the relationship between a user and an external person. FIG. 5 is a view showing an example of a recommended movement path that passes outside the play area. FIG. 6 is a view showing an example of a recommended movement path that passes outside the play area. FIG. 7 is a view showing an example of a recommended movement path that passes through the play area. FIG. 8 is a view showing an example of a recommended movement path taking into account predicted movement in the play area. FIG. 9 is a view showing an example of a recommended movement path taking into account predicted movement in the play area. FIG. 10 is a first sequence diagram showing an example of processing between an HMD and a mobile information terminal. FIG. 11 is a first flowchart of a recommended movement path program for an HMD taking into account predicted movement in the play area. FIG. 12 is a second sequence diagram showing an example of processing between an HMD and a mobile information terminal. FIG. 13 is a second flowchart of a recommended movement path program for an HMD taking into account predicted movement in the HMD. FIG. 14 is a view showing an example of a display screen in which an AR image for recommended route guidance is superimposed on a camera image. FIG. 15 is a sequence diagram showing an example of processing between an HMD, a mobile information terminal, and a display device. FIG. 16 is a view showing an example of approach alert information display on a display device. FIG. 17 is a third flowchart of a recommended movement path program for an HMD taking into account predicted movement in the HMD. 1 is a diagram showing an example of a recommended movement route when a plurality of HMDs are present. FIG. 2 is a diagram showing an example of a recommended movement route when a plurality of HMDs are present. FIG. 3 is a diagram showing an example of a recommended movement route when a plurality of HMDs are present. FIG. 4 is a sequence diagram showing an example of a process including a plurality of HMDs. FIG. 5 is a fourth flowchart of a recommended movement route program for an HMD showing an example of a process. FIG. 6 is a block diagram showing an example of a mobile information terminal. FIG. 7 is a sequence diagram showing an example of a process between a mobile information terminal and an HMD. FIG. 8 is a flowchart of a movement route creation program for a mobile information terminal showing an example of a process. FIG. 9 is a diagram explaining an example of display control due to tilt of a mobile information terminal. FIG. 10 is a diagram explaining an example of display control due to tilt of a mobile information terminal. FIG. 11 is a diagram explaining an example of a destination route input method. FIG. 12 is a diagram explaining a modified example of a destination route input method. FIG. 13 is a diagram explaining a modified example of a destination route input method.

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the embodiment, a technology will be described that, in relation to a relationship between an HMD user (sometimes referred to as a user) wearing a head-mounted display (sometimes referred to as an HMD) and viewing and experiencing virtual reality (VR) content unfolding in a virtual space, and a person (hereinafter sometimes referred to as an external person) other than the user in the same real space, provides the external person with recommended movement route information that takes into account the user's play area, enabling the external person to move without interfering with the user's VR content viewing experience.

[0013] [First Example] A first example will be described with reference to Figures 1 to 8. Figure 1 is a block diagram showing an example of an HMD, and is applicable to the second example and subsequent examples in addition to the first example.

[0014] The HMD 1 in Figure 1 includes a camera unit 10, a distance measurement sensor unit 11, an image display unit 12, a speaker 13, a microphone 14, a group of sensors 15, a control unit 16, an internal bus 17, a communication unit 20, a main calculation unit 21, a memory unit 22, a storage unit 23, a basic operation program 24, a VR viewing program 25, and a route recommendation program 26.

[0015] The main calculation unit 21 is configured using a processor such as a CPU, and the memory unit 22 is configured from RAM or the like. The storage unit 23 is configured from a non-volatile storage medium such as flash ROM, and stores a basic operation program 24, a VR viewing program 25, and a route recommendation program 26. The basic operation program 24, the VR viewing program 25, and the route recommendation program 26 are expanded in the memory unit 22 and executed by the main calculation unit 21. Note that in FIG. 1, the basic operation program 24, the VR viewing program 25, and the route recommendation program 26 are each shown as separate programs. However, this is not limiting, and a single program may include and provide the basic operation function, the VR viewing function, and the route recommendation function. Alternatively, different divided programs may provide the above functions.

[0016] When a user wears the HMD 1 and experiences viewing VR content using the VR viewing program 25, the camera unit 10 captures the real space in front of the HMD 1. If the camera unit 10 is equipped with multiple cameras, it may capture a 360-degree view of the surroundings of the HMD 1. If the camera unit 10 is equipped with a 360-degree camera, it may capture a 360-degree view of the surroundings of the HMD 1 using the 360-degree camera. The main processing unit 21 detects real objects such as furniture and external people from the images captured by the camera unit 10. The distance measurement sensor unit 11 acquires the distance to the detected real object, and the HMD 1 can create a three-dimensional spatial relationship diagram of the real space using the data from the distance measurement sensor unit 11. The image display unit 12 displays images of the VR content, and the speaker 13 outputs the sound of the VR content. If necessary, the microphone 14 acquires the user's voice. The sensor group 15 performs various sensing operations, for example, sensing the user's up / down, forward / backward, left / right movements, and rotational movements that accompany changes in the direction of the user's line of sight.

[0017] The camera unit 10 may be equipped with an infrared camera or infrared light illumination to improve infrared sensitivity in consideration of shooting in dark areas. Furthermore, the distance measurement sensor unit 11 may be configured to perform three-dimensional measurement of real space using laser light, such as LiDAR, to improve detection accuracy.

[0018] Furthermore, the communication unit 20 has multiple communication protocols, such as Internet communication and near-field communication, which are used depending on the purpose. The VR viewing program 25 in the HMD 1 receives VR content using Internet communication and allows the user to view the VR content. Furthermore, the HMD 1 directly communicates with a mobile information terminal using near-field communication or the like, and transmits and receives three-dimensional space maps, camera images, and the like.

[0019] The calculation processing of the main calculation unit 21 may be performed in part or in whole by a server on the Internet via the communication unit 20, or by a personal computer connected by direct communication, for example.

[0020] 2 is an external view of the HMD 1, which is a fully immersive HMD, and corresponds to the block diagram of FIG.

[0021] In Figure 2, the same components as those shown in the block diagram of the HMD in Figure 1 are denoted by the same symbols. The camera unit 10 is denoted by the same symbol. The distance measurement sensor unit 11 for measuring the distance to a real object is denoted by the same symbol. The image display unit 12 has a plurality of display elements for the left eye and the right eye as display elements, displays an image for the left eye and an image for the right eye, and allows the user to recognize a three-dimensional image. Note that the image display unit 12 may be configured to use another method as long as it is capable of three-dimensional display.

[0022] The microphone 14, the sensor group 15, and the control unit 16 are denoted by the same reference numerals. The speakers (13a, 13b) correspond to the speaker 13 in Fig. 1. The HMD 1 is formed with housings (18a, 18b) for wearing.

[0023] FIG. 3 is a diagram in which a network and a VR service server are added in a three-dimensional spatial relationship diagram including a user and an external person.

[0024] Figure 3 shows an HMD 1, a user 2, an external person 3, a mobile information terminal 4 such as a smartphone or a wearable device, an access point 5 for Internet communication, a network 6, a VR service server 7 that provides VR content, a bookshelf 30, a television 31, a desk lamp 32, and a play area 33.

[0025] The HMD 1 worn by the user 2 receives VR content distributed from a VR service server 7 via an access point 5 and a network 6, and the user 2 views and experiences the VR content. The Internet 6 may be an intranet, and the VR service server 7 may be an external device such as a PC connected via direct communication. Furthermore, the VR content stored in the HMD 1 may be viewed and experienced.

[0026] The bookshelf 30, television 31, and desk lamp 32 are examples of fixed real objects in real space. The types of real objects are not particularly limited, and other real objects such as a sofa and a table may also be used. These real objects are real objects that user 2 should avoid coming into contact with while viewing and experiencing VR content. The play area 33 is set as an area in which these real objects should be avoided. If user 2 moves outside the play area 33, the HMD 1 interrupts viewing of the VR content to avoid a collision. In addition, just before user 2 moves outside the play area 33, the HMD 1 superimposes a VR image on the VR content to alert the user, thereby encouraging the user to avoid contact.

[0027] Figure 4 is a two-dimensional schematic diagram corresponding to the three-dimensional spatial relationship diagram in Figure 3. The two-dimensional schematic diagram in Figure 4 corresponds to the entire three-dimensional space, and the same components as in Figure 3 are assigned the same numbers. In Figure 4, reference numeral 34 indicates a real space, for example, the entire living room. Note that the two-dimensional schematic diagram may also be a two-dimensional schematic diagram of a partial area including an external person and the external person's desired destination.

[0028] 5A and 5B show an example of a recommended path of travel that passes outside the play area, with a recommended path of travel 40 added to the two-dimensional schematic diagram of FIG.

[0029] 5A, there is space between the boundary (such as a wall) of the play area 33 and the real space 34, and only the outbound recommended movement route to the desired destination is presented. Here, the space is an area where the user 2 will not collide with a real object immediately if he or she leaves the play area 33. When the outside person 3 wishes to move from his or her current location to the bookshelf 30, the recommended movement route 40 is set within the space.

[0030] In addition, the desired destination may be presented directly or at the same time as a return route via a third destination. In the example of Fig. 5B, a recommended travel route 43 passing through the television 31 as the third destination and a return route 44 from the third destination are presented at the same time.

[0031] 6A is a diagram showing an example of a recommended movement route that passes through the play area 33 when the above-mentioned space is extremely narrow. In FIG. 6A, the movement history 41 of user 2, the predicted movement range 42, and the changed play area 35 are shown.

[0032] The user 2's movement history 41 is movement information corresponding to the travel time required for the outside person 3 to travel from their current location to their desired destination. Because the travel time is short compared to the overall viewing time of the VR content, the predicted movement range 42 is limited to a portion of the play area 33. While the outside person 3 moves, the HMD 1 changes the play area 33 to a play area 35 (narrowing the play area in the example of FIG. 6 ) to create space and sets a recommended movement path 40 within the space. Here, the range of the play area 35 is determined based on the predicted movement range 42. The predicted movement range 42 is merely a prediction based on trends and history, and it is possible that the user 2 may move beyond the predicted range. Therefore, it is desirable that the range of the play area 35 be wider than the predicted movement range 42, and that the play area 35 be within a range that reduces the possibility of the outside person 3 coming into contact with the play area 35. Note that when changing the play area 33 to the play area 35, the HMD 1 may notify the user of this change to a level that does not interfere with the VR content viewing experience. The changed play area 35 is returned to the play area 33 after the travel time has elapsed or after the external person 3 has reached the desired travel destination.

[0033] The user's movement history 41 may be data going back in time from the present, data from when the user viewed the same VR content in the past, or data reflecting the movement history of a third party who viewed the same VR content. Such history information is recorded in the VR service server 7, for example, in association with the content.

[0034] 6B and 6C are diagrams showing a recommended movement path based on predicted movement of the play area. FIGS. 6B and 6C show the current play area 35a and a play area 35b based on the results of the movement prediction. In some cases, the play area is set to move in accordance with the user's movement during a VR content viewing experience. Therefore, the HMD 1 may predict the user's movement and reflect the results in the recommended movement path.

[0035] FIG. 6B illustrates an example of presenting an outbound route. Currently, a recommended path 40a is set that does not pass through the play area 35a. If the user's movement from position 2a' to position 2b' is predicted, the HMD 1 may reflect a recommended path 40b that does not pass through the play area 35b. The external person 3 can reach the destination (in this example, the bookshelf 30) by traveling along the current recommended path 40a or by traveling along the recommended path 40b after the user moves. Note that the user's movement can be predicted based on an appropriate method. For example, if the user is currently viewing a scene in the VR content where the user's movement is predicted, the HMD 1 can predict the user's movement based on the predicted direction and amount of movement of the user in that scene. Furthermore, the HMD 1 can obtain, for example, from the VR service server 7, the history of other viewers who have viewed the same VR content and predict the user's movement based on the tendency of the actions taken by the other viewers in the scene currently being viewed.

[0036] FIG. 6C shows an example of presenting an outbound route and a return route. Currently, a recommended movement path 40a that does not pass through the play area 35a is set as the outbound route. Here, if the user's movement from position 2a' to position 2b' is predicted, the HMD 1 may reflect a recommended movement path 40b that does not pass through the play area 35b as the return route. The external person 3 can proceed along the current recommended movement path 40a, and then, after the user moves, proceed along the recommended movement path 40b to return to the original position. Note that the user's movement can be predicted using a method similar to that described with reference to FIG. 6B.

[0037] FIG. 7 is a first sequence diagram showing an example of processing between the HMD 1 and the mobile information terminal 4.

[0038] In S10, the HMD 1 receives a travel route recommendation request from the portable information terminal 4 in the route recommendation program 26. The travel route recommendation request includes parameters such as a desired destination and a second desired destination such as a return trip. The second desired destination may include even more points as stopovers. In S11, the HMD 1 transmits to the portable information terminal 4 from the route recommendation program 26 that the external person and desired destination have been confirmed, and in S12, transmits a two-dimensional schematic diagram including the external person, desired destination, and recommended travel route.

[0039] When the HMD 1 receives a notification that the external person has reached the desired destination in S13, if there are return route parameters, the HMD 1 transmits a two-dimensional schematic diagram including a recommended return route in S14, and further receives a notification that the desired destination has been reached in S15. Note that, although the notification of arrival at the desired destination is received from the mobile information terminal 4 in S13 and S15, this may also be detected within the route recommendation program 26. Furthermore, if no return route parameters are set, the sequence ends when the route recommendation program 26 receives a notification that the desired destination has been reached in S13. At this time, it is conceivable that the external person 3 will return from the arrival destination to the original location (return route) or move to another location. In this case, the external person 3 can obtain the same effect as when return route parameters are set by transmitting a new travel route recommendation request from the mobile information terminal 4 to the HMD 1.

[0040] FIG. 8 is a first flowchart of the route recommendation program 26 of the HMD, showing an example of processing.

[0041] The process starts at S20. At S21, the HMD 1 generates a three-dimensional spatial relationship diagram of the real space from the camera image and distance measurement data. This three-dimensional spatial relationship diagram shows the real space and includes data on the positions of real objects, etc.

[0042] In S22, the HMD 1 sets a play area so as to avoid real objects. The setting of the play area is notified to the VR viewing program 25, and the user begins viewing the VR content. In S23, the route recommendation program 26 confirms that the user is currently viewing the VR content.

[0043] The HMD 1 checks in S24 whether there is a request for a recommended travel route. If there is a recommendation request, the HMD 1 checks the external person in S25 and the real object of the desired destination in the three-dimensional space diagram in S26, and creates a two-dimensional schematic diagram including the external person and the desired destination in S27. This two-dimensional schematic diagram corresponds to the three-dimensional schematic diagram and includes data related to the positions of the real objects, etc. If there is no recommendation request, the process proceeds to checking the termination conditions in S33.

[0044] The HMD 1 determines a recommended movement path in S28. The method for determining the recommended movement path may be the method described in FIGS. 5A, 5B, 6A, 6B, and 6C. In S29, if a temporary change in the play area is involved, the HMD 1 notifies the user of this. Examples of notification methods include superimposing a special virtual object image on a portion of the display image of the VR content on the HMD 1, or superimposing a special sound on the sound of the VR content. Note that a user who feels that notifications interfere with viewing the VR content may be able to set the notification not to be sent. Furthermore, in S30, the HMD 1 superimposes the recommended movement path on a two-dimensional schematic diagram and transmits the diagram to the mobile information terminal 4.

[0045] If it is not confirmed in S31 that the outside person has reached the desired destination, and if it is confirmed in S31 that the outside person has reached the desired destination and the return route is included in the recommendation request in S32, the process returns to before S25.

[0046] If S31 returns NO or S32 returns YES, the HMD 1 will execute the processes of S25 to S30 again. At this time, the processing content may be changed from the first time the processing is executed after receiving a recommended travel route request in S24 to reduce the processing load and processing time. Specifically, for example, when checking a foreign person in S25 or a physical object at the desired destination in S26 in a three-dimensional space diagram, searching using the previous position as a base point allows the foreign person or the physical object at the desired destination to be confirmed more efficiently than searching the entire three-dimensional space. Furthermore, the two-dimensional schematic diagram created in S27 can also be created efficiently by updating only the differences from the previous creation data, rather than recreating it anew.

[0047] Furthermore, if the return route is not included in the recommendation request in S32, the process proceeds to checking the termination condition in S33. If there is a termination condition such as a notification to stop viewing VR content in S33, the HMD 1 ends the program in S34. If there is no termination condition, the process returns to before S23, and the HMD 1 continues the VR viewing experience while checking whether there is a recommendation request for a new travel route in S24.

[0048] In this flowchart, the VR viewing experience by the VR viewing program 25 monitored in S23 and the processing of S24 to S33 are performed in parallel, and the VR viewing experience is not interrupted during the processing of S24 to S33. This can be achieved, for example, by having the control unit 16 execute the VR viewing program 25 and the route recommendation program 26 in separate threads.

[0049] As described above, the head-mounted display of the first embodiment makes it possible to display on the mobile information terminal of an outside person a recommended travel route to a desired destination that has a low probability of collision with the user, and the outside person can travel based on the displayed recommended travel route, thereby ensuring safety and preventing interference with the user's VR content viewing experience.

[0050] [Second Example] A second example will be described with reference to Figures 9 to 11. Note that descriptions similar to those above may be omitted. Figure 9 shows an example of processing, and is a second sequence diagram between the HMD 1 and the mobile information terminal 4. The same sequence elements as those in the first sequence diagram shown in Figure 7 are assigned the same numbers. In Figure 9, from the perspective of the route recommendation program 26 of the HMD 1, S40 is reception of a camera image, and S41 and S42 are conditional transmission of an AR image for route guidance.

[0051] In S10, the HMD 1 receives a travel route recommendation request from the portable information terminal 4. The travel route recommendation request includes parameters such as a desired destination and a second desired destination such as a return trip. Furthermore, in S40, the HMD 1 receives a camera image captured by a camera provided in the portable information terminal 4. In S11, the HMD 1 transmits to the portable information terminal 4 that the external person and the desired destination have been confirmed, and in S12, transmits a two-dimensional schematic diagram including the external person, the desired destination, and the recommended travel route. Furthermore, in S41, the HMD 1 transmits an AR image for travel route guidance to be superimposed on the camera image.

[0052] When the HMD 1 receives a notification that the external person has reached the desired destination in S13, if there are return route parameters, the HMD 1 transmits a two-dimensional schematic diagram including a recommended return route in S14, and further transmits an AR image for route guidance to be superimposed on the camera image in S42. The HMD 1 receives the notification that the desired destination has been reached in S15. If return route parameters are not set, the sequence ends once the route recommendation program 26 receives a notification that the desired destination has been reached in S13, as in FIG. 7, and the external person returns from the arrival destination to the original location (return route). Note that by transmitting a new route recommendation request when moving to another location, it is possible to obtain the same effect as when return route parameters are set.

[0053] 10 shows an example of processing and is a second flowchart of the HMD route recommendation program 26. The same steps as those in the first flowchart shown in FIG. 8 are assigned the same numbers, and duplicated explanations may be omitted.

[0054] If there is a request for recommended travel routes in S24, the HMD 1 receives camera images from the portable information terminal in S45. The HMD 1 determines a recommended travel route in S28, and then superimposes the recommended travel route onto a two-dimensional schematic diagram and transmits the diagram to the portable information terminal in S30.

[0055] Furthermore, in S46, the HMD 1 identifies the positional relationship between the camera image of the portable information terminal and the three-dimensional spatial relationship diagram, and in S47, checks whether a physical object of the desired destination is captured in the camera image of the portable information terminal. If a physical object is captured, the HMD 1 generates and transmits an AR image for route guidance to be superimposed on the camera image of the portable information terminal in S48. Note that the check in S47 as to whether a physical object of the desired destination is captured in the camera image may be performed in the same way even if the physical object of the desired destination is not captured in the camera image, but is present near the camera image.

[0056] If the physical object of the desired destination is not captured in the camera image in S47, the process proceeds to S31.

[0057] 8, in this flowchart, the user's VR viewing experience, which starts in S23, and the processes of S24 to S33 are performed in parallel, and the VR viewing experience is not interrupted during the processes of S24 to S33. This can be achieved, for example, by having the control unit 16 execute the VR viewing and the creation of the recommended route, including the generation of AR images, in separate threads.

[0058] 11 is an example of the display screen of the mobile information terminal 4 in which a recommended route guidance AR image is superimposed on a camera image. Fig. 11 shows a camera image 45 in which image information captured by a camera included in the mobile information terminal 4 is displayed on the display screen of the mobile information terminal 4. The camera image 45 captures an image of the user 2 and a bookshelf 30, which is a physical object at the desired destination, and a recommended travel route guidance AR image 46 is superimposed on the captured image. The recommended travel route guidance AR image 46 guides the outside person 3 holding the mobile information terminal 4 to the desired destination 30.

[0059] As described above, the head-mounted display of the second embodiment has the same features as the first embodiment, and can use a camera image to display, for example, recommended travel route guidance superimposed on an image of the real world while indicating the user's position and objects that exist in the real world, thereby making it possible to realize recommended travel route guidance that is easy to see. In other words, since the recommended travel route guidance can be displayed on the scenery that the user is actually viewing, it is possible to clearly show where and how to move in the real world compared to a schematic display.

[0060] [Third Example] A third example will be described with reference to Figures 12, 13, and 14. Note that descriptions similar to those above may be omitted. Figure 12 shows an example of processing, and is a sequence diagram between the HMD 1, the portable information terminal 4, and the display device 31. The same sequence elements as those in the first sequence diagram between the HMD 1 and the portable information terminal 4 shown in Figure 7 are assigned the same numbers, and duplicate descriptions will be omitted. In Figure 12, the HMD 1 transmits information to the portable information terminal 4 and the display device 31 from the path recommendation program 26 of the HMD 1 in S50 and S51 to warn of approach to the HMD.

[0061] 13 shows an example in which an image 36 for alerting users to approaching is displayed on the television 31 shown in FIG. 3 as a display device. In this example, a text image stating "User A is playing in the living room" is displayed. If the television 31 is turned off, the television 31 may be turned on remotely to display the image. The display may also be stopped upon checking a timer. The display may also be stopped upon checking that the mobile information terminal has requested a recommended travel route.

[0062] 12 and 13, the television 31 shown in FIG. 3 is an example of a display device. However, other display devices such as an intercom or a home security monitor may also be used. The image for alerting the user 2 to approach may be an illustrated image or text. Furthermore, for example, a schematic diagram of a living room including a play area may be displayed, other than the example of FIG. 13. It is sufficient if the purpose of making the user 2 watching VR content aware that they need to pay attention can be achieved without the outside person 3 operating the mobile information terminal 4.

[0063] Furthermore, information warning the user 2 when approaching the user 2 is provided not only to the external person 3 but also to anyone who may approach the user 2 viewing the VR content. The HMD 1 may transmit information to, for example, a mobile information terminal other than the mobile information terminal 4 possessed by the external person 3. When transmitting information to the display device 31, it is preferable to transmit the information preferentially to a device that is likely to be seen by many people. That is, if there is a television 31 in both the living room and the bedroom, transmitting information preferentially to the television 31 in the living room, where many people tend to gather, allows multiple people to be warned with a single transmission. Furthermore, if a device that emits audio, such as a smart speaker, is installed instead of a display device, information may be transmitted to the device to warn the user by audio. The audio emitted here is, for example, audio information of text displayed on the display device 31.

[0064] 14 shows an example of processing and is a third flowchart of the HMD route recommendation program 26. The same steps as those in the first flowchart shown in FIG. 8 are given the same numbers, and redundant explanations will be omitted.

[0065] In the third flowchart of Fig. 14, S50 is added to the flowchart of Fig. 8. In S50, the HMD 1 transmits an approach warning image to the HMD 1 to the display device.

[0066] As described above, the head-mounted display of the third embodiment has the same features as the first embodiment, and can make outsiders aware of the need to be vigilant about approaching the HMD early on. More specifically, when an outsider enters a room where an HMD user is viewing VR content, the outsider does not realize for the first time that caution is needed when approaching after entering the room, but rather knows the need for caution before entering the room, making it possible to make advance preparations, such as by always carrying a mobile information terminal with them when entering the room so that they can request and receive a recommended travel route.

[0067] [Fourth Example] A fourth example will be described with reference to Figures 15 to 17. This example is an example in which multiple HMD users exist. Note that the same explanation as above may be omitted. Figures 15A to 15C are two-dimensional schematic diagrams corresponding to the two-dimensional schematic diagram of Figure 4. However, there are two users (2a, 2b), and the two HMD users share the real space and set their own play areas (33a, 33b). Alternatively, the two users (2a, 2b) set a play area 33c that combines their respective play areas (33a, 33b).

[0068] In the example of Fig. 15A, there is a gap between the play areas (33a, 33b). The gap is provided so that users 2a and 2b do not come into contact immediately when they cross the play area, and a recommended movement path 40 is set within the gap. If the gap is narrow, the gap is provided taking into account the movement histories of users 2a and 2b, as shown in Fig. 6A.

[0069] In contrast, in the example of FIG. 15B , there is no space between the play areas (33a, 33b) set by the two users (2a, 2b). Furthermore, as shown in FIG. 15C , when a play area 33c is set by combining the play areas (33a, 33b), there is also no space between the play areas. In such a case, a recommended travel route 40 is set to avoid the play areas. That is, in the example of FIG. 15B , a recommended travel route 40 is set to avoid both the two play areas 33a and 33b, and in the example of FIG. 15C , a recommended travel route 40 is set to avoid the play area 33c, which is the combined play area (33a, 33b). Note that while the example described here uses a case where there are two users, a recommended travel route 40 is similarly set to avoid the play areas in the case of two or more users, taking into account the play areas of each user. Furthermore, if there is insufficient room to avoid multiple play areas or combined play areas (for example, if each individual play area is small but the combined play area takes up the entire room), a recommended travel route 40 is set that does not interfere with the user's VR viewing experience and does not result in contact, taking into account the user's movement history, as described above.

[0070] In Fig. 16, the two HMDs are denoted by reference numerals 1a and 1b. Fig. 16 shows an example of processing and is a sequence diagram between the HMDs 1a and 1b and the portable information terminal 4. The same sequence elements as those in the first sequence diagram between the HMD 1 and the portable information terminal 4 shown in Fig. 7 are assigned the same numbers.

[0071] S60 and S61 are sequence elements for negotiating the play area between the path recommendation program 26a of the HMD 1a and the path recommendation program 26b of the HMD 1b. For example, as shown in FIG. 15A, the two play areas can be adjusted so that they do not overlap (so that there is space between the play areas). As a special example, when users (2a, 2b) watch the same content (such as a game played by both users cooperatively), the play area may be shared. S10a and S10b are travel path recommendation requests to the path recommendation program 26a of the HMD 1a and the path recommendation program 26b of the HMD 1b, and the travel path recommendation requests are issued from the mobile information terminal 4. In S62 and S63, the route recommendation program 26a of the HMD 1a and the route recommendation program 26b of the HMD 1b share their position information with each other, so the HMD 1a transmits the position information of the HMD 1a to the HMD 1b (S62), and the HMD 1b transmits the position information of the HMD 1b to the HMD 1a (S63). The position information is, for example, a position based on the desired destination received as a parameter in S10a, and the route recommendation program 26 of the HMD closest to the desired destination provides the recommended travel route to the mobile information terminal 4. A rule may be determined in advance as to which HMD's route recommendation program 26 provides the recommended travel route, and this rule may be followed.

[0072] 16, the route recommendation program 26 of the HMD 1b provides a recommended travel route. This is because, as a result of determining which of the HMDs 1a and 1b is closer to the desired destination based on the shared position information, it is determined that the HMD 1b is closer. This determination can be made by either the HMD 1a or the HMD 1b, and here, as an example, the HMD 1a and the HMD 1b each make the determination (both determination results will be the same), and the HMD determined to be closer to the desired destination decides to create a recommended travel route and provide it to the mobile information terminal, and notifies the other HMD of this (S64).

[0073] If the return route parameter is not set, when the route recommendation program 26b receives a notification that the desired destination has been reached in S13, the sequence ends once, and the outside person returns from the arrival destination to the original location (return route), as in Fig. 7. Note that by sending a new travel route recommendation request when moving to another location, it is possible to obtain the same effect as when the return route parameter is set.

[0074] The procedure for providing a recommended travel route is from S11 to S15, which is the same as that shown in FIG.

[0075] 17 shows an example of processing and is a fourth flowchart of the HMD route recommendation program 26. The same steps as those in the first flowchart shown in FIG. 8 are given the same numbers, and duplicated explanations will be omitted.

[0076] In the fourth flowchart in Fig. 17, S65, S66, S67, S68, and S69 are added to the flowchart in Fig. 8. In S65, the HMD detects other HMDs in the real space.

[0077] S66 is a step corresponding to S60 and S61 for negotiating and adjusting the play area in the sequence diagram of Figure 16, and S67 is a step for exchanging position information corresponding to S62 and S63, in which the HMD transmits information about its own position to the other HMDs and receives information about the positions of the other HMDs. In S68, the HMD determines whether it will play a role in creating and providing a recommended movement path, and the HMD that plays that role notifies the other HMDs of this in S69. Thereafter, the HMD executes steps from S27 onwards, and in S30, when multiple HMDs are present, the recommended movement path is superimposed on a two-dimensional schematic diagram and transmitted to the mobile information terminal 4.

[0078] As described above, the head-mounted display of the fourth embodiment has the same features as the first embodiment, and is also capable of providing an appropriate recommended movement route even when multiple HMDs are present.

[0079] [Fifth Example] A fifth example will be described with reference to Figures 18 to 23. Note that descriptions similar to those above may be omitted. Figure 18 is a block diagram showing an example of a mobile information terminal. The mobile information terminal 4 in Figure 18 includes a camera unit 50, a display and operation unit 51, a speaker 52, a microphone 53, a group of sensors 54, an internal bus 55, a communication unit 56, an information processing unit 57, a memory unit 58, a storage unit 59, a basic operation program 60, and a route creation program 61.

[0080] The information processing unit 57 is configured using a processor such as a CPU, and the memory unit 58 is configured with a RAM or the like. The storage unit 59 is configured with a non-volatile storage medium such as a flash ROM, and stores a basic operation program 60 and a route creation program 61. The basic operation program 60 and the route creation program 61 are expanded in the memory unit 58 and executed by the information processing unit 57. Note that in FIG. 18 , the basic operation program 60 and the route creation program 61 are shown as separate programs. However, this is not limiting, and a form in which a single program includes a basic operation function and a route creation function and provides those functions may also be used. Alternatively, a form in which different divided programs provide the above functions may also be used.

[0081] The camera unit 50 captures images of the real space in front of the mobile information terminal 4. The display / operation unit 12 displays the execution results of the route creation program 61, camera images, etc. The display / operation unit 12 is equipped with a flat display, an input operation unit is stacked on top of it, and has touch panel functionality. The speaker 13 outputs sounds from application programs, etc., and the microphone 14 acquires the voices of external people as needed. The sensor group 15 senses the movement, tilt, etc. of the mobile information terminal 4. Furthermore, the communication unit 56 is equipped with multiple communication protocols such as mobile communication, Internet communication, and near-field communication, and supports direct communication with the HMD.

[0082] FIG. 19 shows an example of processing, and is a sequence diagram between the route creation program 61 of the portable information terminal and the route recommendation program 26 of the HMD.

[0083] In S70, the route creation program 61 sends a request for travel route related information to the route recommendation program 26. The request for travel route related information includes the desired destination as a parameter. In S71, the route creation program 61 receives information on all or part of the three-dimensional spatial relationship diagram.

[0084] FIG. 20 is a flowchart showing an example of processing of the route creation program 61 of the portable information terminal.

[0085] Processing begins in S80. The portable information terminal 4 issues a route-related information request in S81. The portable information terminal 4 receives a three-dimensional spatial relationship diagram in S82. The received three-dimensional spatial relationship diagram may be partial information of the three-dimensional spatial relationship diagram held by the HMD 1, including an external person and a physical object at the desired destination. The portable information terminal 4 identifies the external person's current location and desired destination within the three-dimensional spatial relationship diagram in S83, and creates a two-dimensional schematic diagram in S84. Furthermore, the portable information terminal 4 creates a recommended travel route in S85.

[0086] In S86, the portable information terminal 4 determines whether a camera is capturing an image of the area ahead, including the desired destination. If a camera is capturing an image, the portable information terminal 4 identifies the position of the camera image within the three-dimensional spatial relationship diagram in S87. In S88, the portable information terminal 4 checks whether a camera image is being displayed, and if so, the portable information terminal 4 creates an AR image for guiding the travel route in S89 and displays it superimposed on the camera image in S90.

[0087] If no camera image has been taken in S86, or if no camera image has been displayed in S88, the portable information terminal 4 creates and displays a two-dimensional schematic diagram with the recommended travel route superimposed thereon in S91.

[0088] If it is not confirmed in S92 that the desired destination has been reached, and if a recommended return route is to be generated in S93, the process returns to before S83. If a return route is not to be generated in S93, a termination condition determination is made in S94, and if the termination condition is met, the process ends in S95. If the termination condition is not met in S94, the process returns to before S81.

[0089] 21A and 21B are diagrams illustrating an example of display control based on the tilt of the mobile information terminal 4. FIG.

[0090] The mobile information terminal 4 detects the tilt of the mobile information terminal 4 using the sensor group 54 and controls the display pattern of the recommended travel route to be changed depending on the tilt of the mobile information terminal 4. As shown in FIG. 21A , when the tilt θ of the mobile information terminal 4 is smaller than a predetermined value θth, i.e., when the owner of the mobile information terminal 4 holds the mobile information terminal nearly horizontal to the ground, the mobile information terminal 4 displays the recommended travel route superimposed on a two-dimensional schematic diagram. On the other hand, as shown in FIG. 21B , when the tilt θ of the mobile information terminal 4 is greater than a predetermined value θth, i.e., when the owner of the mobile information terminal 4 holds the mobile information terminal nearly vertical to the ground, the mobile information terminal 4 determines that the camera of the mobile information terminal 4 is facing forward, and displays an AR image guiding the recommended travel route superimposed on the camera image. Such a determination of appropriate display control based on the tilt of the mobile information terminal 4 is performed in step S88 of the flowchart of FIG. 20. The display pattern may be changed when the tilt of the mobile information terminal 4 remains greater than or less than a predetermined value θth for a certain period of time or more. This prevents the display pattern from being changed too frequently when the tilt of the mobile information terminal 4 is near θth.

[0091] FIG. 22 is a diagram illustrating an example of a method for inputting a desired destination route.

[0092] A camera-captured image 45 is displayed on the display screen of the portable information terminal 4. When an external person registers a desired destination in S10 of FIG. 7, for example, if the desired destination can be set and registered by touching an object (e.g., bookshelf 30) displayed on the screen with fingertip 71b, the desired destination can be set and registered with a simple operation, making the device easy to use. As an example of a method for an external person to input a desired destination and a desired route to that destination, the external person may touch a mark 70 indicating the external person's current location displayed on the screen with fingertip 71a, drag the fingertip as indicated by reference numeral 72 to move it to the desired destination, bookshelf 30, and release the fingertip at the position indicated by reference numeral 71b. With this series of operations, the bookshelf 30, which is the end point of the fingertip movement, can be set and registered as the desired destination, and the route indicated by reference numeral 72, which is obtained by dragging the fingertip from the start point 71a to the end point 71b, can be set and registered as the desired route. If it is possible to request a travel route close to this desired route 72, the desired destination and desired travel route can be set, registered, and requested with a simple operation, which is very user-friendly. Upon receiving this request, the HMD creates a recommended travel route based on the wishes of the external person and the play area setting status of user 2, and secures the recommended travel route in the area to the right of user 2 (as seen from the external person), for example, and provides this recommended travel route to the external person.

[0093] 23A and 23B are diagrams illustrating a modified method for inputting a destination route. In FIG. 23 , when a user drags a fingertip from a mark 70 indicating the external person's current location to a desired destination (bookshelf 30 in this example) with their fingertip, the HMD provides a real-time response as to whether the route dragged with the fingertip is passable or a recommended route, and the response is reflected on the display. In FIG. 23A , the external person drags a fingertip 71c on the screen of the mobile information terminal 4 along the route they wish to travel. If the dragged route is passable without interfering with user 2's play area, a path is drawn based on the movement of the dragging fingertip 71d. If, for example, the external person gets too close to user 2 and interferes with the play area, an icon 73 indicating that further progress is prohibited is displayed, and the progress of the path based on the fingertip movement is stopped. The external person may also be notified of the impossibility of passage by vibrating their fingertip (reference numeral 74 in the figure). When an outsider finds out that the desired route in Fig. 23A is not passable, he / she considers an alternative route and drags a route different from that in Fig. 23A, as shown in Fig. 23B. At this time, if the route (the trajectory of the fingertip movement) that was previously dragged and found to be impassable is displayed, it becomes possible to use the display as a reference and drag with fingertip 71f from fingertip 71e to take a different route, which is convenient.

[0094] As described above, the mobile information terminal makes it possible to display recommended travel routes to a desired destination that have a low probability of collision with the user, thereby preventing interference with the user's VR content viewing experience.

[0095] According to the above description, for example, the following HMD is provided. This HMD includes a camera unit, a ranging sensor unit, an image display unit, a communication unit, a sensor unit, and a control unit. The camera unit captures images in front of the user of the HMD or the entire 360-degree range and detects the background and real objects in the real space. The ranging sensor unit measures the distance between the HMD and real objects and obtains data related to three-dimensional spatial relationship information. The image display unit displays VR content in three dimensions. The sensor unit detects the movement and orientation of the HMD and, in combination with the three-dimensional spatial relationship information, improves the accuracy of approach to real objects. The control unit is a computer system consisting of a CPU and memory, which controls the overall operation of the HMD and generates recommended travel route information. The communication unit is capable of Internet communication as well as near-field communication. The communication unit receives the external person's desired destination and camera images from a portable information terminal carried by the external person and transmits recommended travel route information.

[0096] Also, for example, the following HMD is provided. This HMD includes a camera unit, a distance measurement sensor unit, an image display unit, a communication unit, a sensor unit, and a control unit. The camera unit captures images in front of the user of the HMD or the entire 360-degree range, and detects the background and real objects in the real space. The distance measurement sensor unit measures the distance between the HMD and real objects and obtains data related to a three-dimensional spatial relationship diagram. The image display unit displays VR content in three dimensions. The control unit generates a recommended travel route. The communication unit receives the external person's desired destination, camera images, etc., from a mobile information terminal carried by the external person, and transmits a recommended travel route.

[0097] Furthermore, according to the above description, for example, the following mobile information terminal is provided. This mobile information terminal includes a camera unit, a display / operation unit, a communication unit, a sensor unit, and an information processing unit. The camera unit captures an image in front of an external person. The display / operation unit inputs a desired destination and displays the camera image and a recommended travel route. The communication unit has at least a communication function with an HMD and receives a three-dimensional spatial relationship diagram. The information processing unit controls the overall operation of the mobile information terminal and generates a recommended travel route.

[0098] Also, for example, the following mobile information terminal is provided. This mobile information terminal includes a camera unit, a display / operation unit, a communication unit, a sensor unit, and an information processing unit. The camera unit captures an image of an outside person in front of them. The display / operation unit inputs a desired destination and displays the camera image and a recommended travel route. The communication unit has at least a communication function with an HMD and receives a three-dimensional spatial relationship diagram. The information processing unit generates recommended travel route information.

[0099] The embodiments of the present invention described above with reference to the drawings are not limited to these, and it is possible to replace part of the configuration of one embodiment with another embodiment. It is also possible to add the configuration of another embodiment to the configuration of one embodiment. These all fall within the scope of the present invention, and the numerical values, messages, etc. appearing in the text and figures are merely examples, and the use of different ones does not impair the effects of the present invention.

[0100] Furthermore, some or all of the functions of the invention may be implemented in hardware, for example, by designing an integrated circuit. They may also be implemented in software by a microprocessor unit, CPU, etc., interpreting and executing an operating program. Furthermore, the scope of software implementation is not limited, and both hardware and software may be used.

[0101] 1: HMD, 2, 2a, 2b: User, 3: External person, 4: Portable information terminal, 6: Network, 7: VR service server, 10, 50: Camera unit, 11: Distance measurement sensor unit, 12, 51: Image display unit, 16: Control unit, 20, 56: Communication unit, 21: Main calculation unit, 57: Information processing unit, 23, 59: Storage unit, 25: VR viewing program, 26: Route recommendation program, 61: Route creation program, 30: Bookshelf, 31: Television, 33, 33a, 33b, 33c, 35: Play area, 40, 40a, 40b: Recommended movement route, 41: Movement history, 42: Movement prediction range, 45: Camera image, 46: AR image for recommended movement route guidance.

Claims

1. A head-mounted display that allows a user to view virtual reality content, comprising a processor that: acquires a desired destination of a person other than the user; generates a two-dimensional schematic diagram including the user and the person other than the user; adds a recommended travel route to the desired destination to the two-dimensional schematic diagram; and provides the two-dimensional schematic diagram with the recommended travel route added to a mobile information terminal carried by the person other than the user.

2. A head-mounted display as claimed in claim 1, characterized in that the processor creates a three-dimensional spatial relationship diagram of real space, and creates the two-dimensional schematic diagram from the three-dimensional spatial relationship diagram.

3. A head-mounted display as described in claim 1, characterized in that the processor sets a play area in which the user will view the virtual reality content, and sets a recommended movement route outside the play area.

4. A head-mounted display as described in claim 1, characterized in that the processor: sets a play area in which the user will view the virtual reality content; evaluates the user's movement history; temporarily changes the play area based on the evaluation; and sets a recommended movement route outside the changed play area.

5. A head-mounted display as described in claim 1, characterized in that the processor sets a play area in which the user views the virtual reality content, predicts the user's movements, and sets a recommended movement path outside the play area that moves in accordance with the user's movement.

6. A head-mounted display as described in claim 2, wherein the processor: acquires camera images from a portable information terminal carried by a person other than the user; recognizes a desired destination of the person other than the user from the camera images and the three-dimensional spatial relationship diagram; generates AR image data in which a recommended travel route is superimposed on the camera image; and provides the AR image data to the portable information terminal carried by the person other than the user.

7. A head-mounted display as described in claim 1, characterized in that the processor provides data to a display device near the head-mounted display for use in outputting warning information to warn of approach to the head-mounted display before obtaining a desired destination of a person other than the user.

8. A head mounted display as described in claim 3, wherein the processor performs processing to avoid overlapping of play areas between multiple head mounted displays, and sets a recommended movement path outside the play areas of the multiple head mounted displays.

9. A mobile information terminal comprising a processor, the processor being capable of performing a process of creating a travel route and a display process, the process of creating the travel route comprising obtaining a three-dimensional spatial relationship diagram of real space from a head-mounted display on which virtual reality content is viewed, generating a two-dimensional schematic diagram from the three-dimensional spatial relationship diagram, and adding a recommended travel route to the two-dimensional schematic diagram, and the display process comprising displaying the two-dimensional schematic diagram with the recommended travel route added.

10. A portable information terminal as described in claim 9, wherein the processor, in the process of creating the travel route, acquires a camera image, recognizes a desired destination from the camera image and the three-dimensional spatial relationship diagram, and generates an AR image showing a recommended travel route to the desired destination, and, in the display process, displays the camera image with the AR image superimposed.

11. A mobile information terminal as described in claim 10, wherein the processor controls display switching so that, in the display processing, either the two-dimensional schematic diagram with a recommended travel route added thereto or the camera image overlaid with an AR image showing the recommended travel route is displayed.

12. A portable information terminal according to claim 10, wherein said processor acquires a desired destination indicated on said camera image displayed in a display process.

Images