Video display device and video display method

Abstract

To provide a video display device and a video display method which show a backtracking route along a route which a user has actually moved along.SOLUTION: A head mounted display (HMD) 1 being a portable video display device comprises a sensor which detects a position and direction of a user, a track information acquisition unit which acquires information of a movement track of the user, a storage unit in which track information of the user and information of an avatar are stored, a display unit 72 which displays the movement track of the user with the avatar, and a main control unit. The main control unit creates the avatar and obtains user's current visible area by the sensor and displays the avatar on the movement track of the user in the display unit in accordance with the user's current visible area on the basis of the track information of the user stored in the storage unit. When being moved along the movement track of the user, the avatar is moved from the latest piece toward the oldest piece of the track information with respect to time.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a portable image display device and an image display method. [Background technology]

[0002] In recent years, portable image display devices, such as smartphones, have become widespread. Among them, a head mounted display (hereinafter, HMD) worn on the user's head displays an image of real space and an image of augmented reality (AR) created by a computer or the like on a glasses-type display screen by superimposing them. Furthermore, by attaching a sensor to the HMD, information acquired by the sensor can be displayed on the display screen as an AR image. For example, Patent Document 1 discloses a configuration of a portable terminal that depicts a user's behavior history, the configuration including a terminal information acquisition means for acquiring the position information of the terminal itself, a camera means for generating a camera image capturing an image of the surroundings, a depiction history calculation means for calculating behavior history depiction information (avatar) to be displayed based on the behavior history acquired in advance and the imaging range of the camera means, an image composition means for generating a composite image in which the behavior history depiction information is depicted in the camera image, and a display means for displaying the composite image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2011 / 093031 Summary of the Invention [Problem to be solved by the invention]

[0004] There are many elderly people who forget the places they stopped off along the way or the route back while traveling along a complicated route. In such cases, there are methods such as navigation devices that guide users to a typical route that returns to a set starting point, but the route guided is not necessarily the same as the route the user actually took, and it cannot be expected that the route along the way will be accurately reproduced.

[0005] In Patent Document 1, when a user instructs to record a behavior history, it is registered in a behavior history server. Then, the behavior history of the desired user is acquired from the behavior history server, and an avatar is displayed superimposed on a camera image according to the user's location information. However, Patent Document 1 assumes that the behavior history of any other user is acquired and displayed, and does not particularly consider displaying the user's own behavior history. In addition, when displaying the behavior history, the avatar is displayed in the order in which the user moved to the locations he or she moved to, so it is not suitable as a tool to guide a user who has forgotten the return route back from the current location.

[0006] An object of the present invention is to provide a video display device that guides a user on a return route along a route that the user has actually traveled. [Means for solving the problem]

[0007] In order to solve the above problems, the image display device of the present invention includes a sensor that detects the position and direction of a user carrying the image display device, a trajectory information acquisition unit that acquires information on the user's movement trajectory from the detection result of the sensor, a storage unit that stores the user's trajectory information acquired by the trajectory information acquisition unit and information on an avatar that is a virtual image representing the user, a display unit that displays the user's movement trajectory as an avatar, and a control unit that controls the trajectory information acquisition unit and the display unit. The control unit generates an avatar from the avatar information stored in the storage unit, obtains the current user's field of view using the sensor, and arranges the avatar on the user's movement trajectory in accordance with the current user's field of view based on the user's trajectory information stored in the storage unit, and displays the avatar on the display unit. When moving the avatar along the user's movement trajectory, the control unit moves the avatar from newer trajectory information toward older trajectory information. Effect of the Invention

[0008] According to the present invention, it is possible to provide easy-to-understand guidance of the return route to a user who has forgotten where he or she stopped along the way or the return route while traveling. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is an external view of an HMD according to a first embodiment. [Diagram 2] FIG. 2 is a block diagram showing the internal configuration of an HMD. [Diagram 3] FIG. 4 is a diagram showing an example of a connection configuration with an external device by a communication processing unit. [Figure 4] FIG. 2 is a diagram showing the functional block configuration of the HMD. [Diagram 5] FIG. 13 is a schematic diagram showing an example of user trajectory collection. [Figure 6] 13 is a flowchart showing a trajectory collection process. [Figure 7A] A trajectory information storage table that stores location information. [Figure 7B] A differential trajectory information storage table that stores position information as differential values. [Figure 7C] Start / end point coordinate table to store start / end position information. [Figure 8A] A still image data storage table for storing still images. [Figure 8B] A video data storage table for saving videos. [Figure 9A] 11 is a flowchart showing an entire avatar display process. [Figure 9B] 9B is a flowchart showing the relief process in FIG. 9A. [Figure 10A] FIG. 13 is a diagram showing an example of a direction change guide display. [Figure 10B] FIG. 13 is a diagram showing an example of a movement direction guide display. [Figure 11A] FIG. 13 is a diagram showing an example of how avatars are displayed. [Figure 11B] FIG. 13 is a diagram showing a display example in which an avatar changes its orientation. [Figure 12] A table showing examples of voice commands used to operate the HMD. [Figure 13] 1A and 1B are diagrams illustrating an operation instruction by a user's finger. [Figure 14A] 13 is a two-dimensional trajectory information storage table for storing position information according to the second embodiment. [Figure 14B] Two-dimensional start and end point coordinate table at start and end. [Figure 15] FIG. 13 is a schematic diagram of avatar display in the third embodiment. [Figure 16] FIG. 13 is a schematic diagram of avatar display in the fourth embodiment. [Figure 17A] 23 is a trajectory information storage table in the seventh embodiment. [Figure 17B] FIG. 13 is a diagram showing an example of how avatars are displayed. [Figure 18] FIG. 13 is a diagram showing the appearance of a smartphone according to an eighth embodiment. [Figure 19] FIG. 13 is a diagram showing the configuration of an image display system to which a plurality of HMDs are connected according to a ninth embodiment. [Figure 20] FIG. 1 is a schematic diagram showing an example of displaying a plurality of avatars. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described, focusing on an example of a head mounted display (HMD) which is a head-mounted image display device. EXAMPLES

[0011] 1 shows an external view of an HMD according to Example 1. The HMD 1 has a glasses-type shape, and a display unit 72 on which the user views an image and an imaging unit 71 that captures an external view are arranged on the front, and various processing units described below are stored in temple units (temples) 91. Also, some of the processing units in the temple units 91 may be separated from the HMD main body, stored in a separate housing, and connected to the HMD main body by a cable.

[0012] 2 is a block diagram showing the internal configuration of the HMD 1. The HMD 1 is made up of a main control unit 10, a system bus 30, a storage unit 40, a sensor unit 50, a communication processing unit 60, a video processing unit 70, a sound processing unit 80, and an operation input unit 90.

[0013] The main control unit 10 is a microprocessor unit that controls the entire HMD 1 according to a predetermined operation program. The system bus 30 is a data communication path for transmitting and receiving various commands and data between the main control unit 10 and each component block in the HMD 1.

[0014] The storage unit 40 has a work area 43 for storing various programs 41 for controlling the operation of the HMD 1, various data 42 such as operation setting values, detection values ​​from a sensor unit described later, and objects including contents, and is used for the operation of various programs. In this case, the storage unit 40 can store operation programs downloaded from a network, various data created by the operation programs, downloaded contents such as moving images, still images, and sounds. It can also store data such as moving images and still images captured by the imaging unit 71. The storage unit 40 needs to hold the stored information even when power is not supplied to the HMD 1 from the outside. Therefore, for example, devices such as semiconductor element memories such as flash ROM and SSD (Solid State Drive), and magnetic disk drives such as HDD (Hard Disk Drive) are used. The operation programs stored in the storage unit 40 can be updated and expanded by downloading from each server device on the network.

[0015] The sensor unit 50 is composed of a GPS (Global Positioning System) sensor 51, a geomagnetic sensor 52, a distance sensor 53, an acceleration sensor 54, a gyro sensor 55, an altitude sensor 56, etc., in order to detect various states of the HMD 1. These sensors detect the position, inclination, direction, movement, altitude, etc. of the HMD 1. In addition to these, other sensors such as an illuminance sensor and a proximity sensor may also be provided.

[0016] The communication processing unit 60 is composed of a LAN (Local Area Network) communication unit 61 and a telephone network communication unit 62. The LAN communication unit 61 is connected to a network such as the Internet via an access point or the like, and transmits and receives data to and from each server device on the network. The connection to the access point or the like may be made by a wireless connection such as Wi-Fi (registered trademark).

[0017] The telephone network communication unit 62 performs telephone communication (calls) and data transmission and reception through wireless communication with base stations of a mobile telephone communication network. The communication with base stations may be performed by W-CDMA (Wideband Code Division Multiple Access) (registered trademark), GSM (Global System for Mobile communications) (registered trademark), LTE (Long Term Evolution), or other communication methods. The LAN communication unit 61 and the telephone network communication unit 62 each include an encoding circuit, a decoding circuit, an antenna, and the like. The communication processing unit 60 may further include other communication units such as a Bluetooth (registered trademark) communication unit or an infrared communication unit.

[0018] The image processing unit 70 is composed of an imaging unit 71 and a display unit 72. The imaging unit 71 is a camera unit that inputs image data of a subject in the outside scene by converting light input from a lens into an electrical signal using an electronic device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) sensor. The display unit 72 is a display device such as a transmissive display using a laser projector and a half mirror, and provides image data to the user of the HMD1.

[0019] The audio processing unit 80 is composed of an audio input / output unit 81, a voice recognition unit 82, and an audio decoding unit 83. The audio input of the audio input / output unit 81 is a microphone, and the user's voice, etc. is converted into audio data and input. The audio output of the audio input / output unit 81 is a speaker, and audio information required by the user is output. The voice recognition unit 82 analyzes the input audio information and extracts instruction commands, etc. The audio decoding unit 83 performs decoding processing of the encoded audio signal and voice synthesis processing.

[0020] The operation input unit 90 is an instruction input unit that inputs operation instructions to the HMD 1. The operation input unit 90 is configured with operation keys such as an array of button switches, but may further include other operation devices. For example, the HMD 1 may be operated using a separate mobile terminal device connected by wired or wireless communication using the communication processing unit 60. In addition, the HMD 1 may be operated by voice commands as operation instructions using the voice recognition unit 82 of the voice processing unit 80.

[0021] 2 includes many components that are not essential to this embodiment, but the effects of this embodiment are not diminished even if these components are not provided. Conversely, components not shown, such as a digital broadcast receiving function and an electronic money payment function, may be added.

[0022] 3 is a diagram showing an example of a connection configuration with an external device by the communication processing unit 60. A LAN communication unit 61 of the HMD 1 connects to a network 5 such as the Internet via a wireless router 4, which is an access point. A server 6 is connected to the network 5, and data is transmitted and received via the LAN communication unit 61 of the HMD 1.

[0023] FIG. 4 is a diagram showing the configuration of functional blocks of the HMD 1. The overall control process of the HMD 1 is mainly executed by the main control unit 10 using various programs 41 and various data 42 in the storage unit 40. The process in the HMD 1 is composed of a trajectory collection process (S100) that collects and stores information on the user's movement trajectory, and an avatar display process (S200) that displays an avatar, which is a virtual image representing the user, based on the stored trajectory information (indicated by a dashed line and a chain line, respectively).

[0024] In the trajectory collection process (S100), the various sensor information acquisition function 11 acquires information from the various sensors of the sensor unit 50, and the trajectory information processing function 12 converts the acquired information from the various sensors into trajectory information that is easy to process internally. The converted trajectory information is stored in the trajectory information storage function 13. If a user instructs to shoot during the trajectory information acquisition, shooting is performed using the imaging unit 71 of the video processing unit 70. In this case, the shooting process is performed by the shooting data acquisition function 14, and the acquired shooting data is associated with the trajectory information and stored in the shooting data storage function 15.

[0025] In the avatar display process (S200), the avatar information of the user stored in advance in the avatar information storage function 16 is read out, and the avatar generated by the avatar generation function 17 is displayed by the avatar display function 18. The display position of the avatar is determined based on the trajectory information stored in the trajectory information storage function 13. However, the field of view calculation function 19 determines whether the display position of the avatar is within the field of view of the HMD 1, and if not, the avatar is not displayed. In addition, the shooting data playback function 20 plays back the shooting data stored in the shooting data storage function 15 in accordance with the route trajectory.

[0026] The storage functions such as the trajectory information storage function 13, the shooting data storage function 15, and the avatar information storage function 16 can also store information in an external server 6 via a LAN communication unit 61 of the communication processing unit 60 as shown in FIG.

[0027] 5 is a schematic diagram showing an example of user trajectory collection. In this example, a user 2 wearing an HMD 1 moves through trajectory points 501, 502, 503, and 504 in that order (indicated by ● marks). At trajectory point 503 (a corner) where the direction of movement changes by 90 degrees, the image capturing unit 71 captures an image of a subject 510. This trajectory point 503 therefore becomes an image capturing point. The objects to be captured here are determined by the user's will, such as places where the user has stopped, corners, places where the user has taken a wrong turn, and plants, animals, and buildings seen at those points, and the captured images are played back when the user later reproduces the route trajectory.

[0028] 6 is a flowchart showing the trajectory collection process (S100) of the HMD 1, and uses as an example the trajectory schematic diagram of FIG 5. This processing procedure is stored in the various programs 41 of the storage unit 40. S110: Start the trajectory collection process (S100) at the instruction of the user 2. In this embodiment, the instruction to start trajectory collection is given by voice of the user 2. For example, when the voice command "Kiseki kaishi" is uttered, the voice recognition unit 82 determines that this is an instruction to start the trajectory collection process.

[0029] S111: Initialize the point numbers of the trajectory points. Here, 0 is set as the initial value of the point number p, so that the point number p at the start of the trajectory collection process is p=0. S112: A timer is started. The timer measures time using a clock built into the main control unit 10. The timer is used in order to collect data on the user's trajectory points at regular time intervals (unit time).

[0030] S113: The position of the HMD1 is detected from information from the sensor unit 50 (various sensor information acquisition function 11), converted into a format of stored data (trajectory information processing function 12), and stored in the various data 42 of the memory unit 40 (trajectory information storage function 13). The position of the HMD1 on a plane is detected by the GPS sensor 51 of the sensor unit 50. By receiving radio waves from multiple GPS satellites, the global position coordinates (longitude and latitude) of the HMD1 can be detected. In addition, the altitude can be detected by the height sensor 56 of the sensor unit 50. For example, the atmospheric pressure is measured to calculate the altitude.

[0031] The position information of the HMD1 can be obtained not only from GPS information from GPS satellites, but also from Wi-Fi (registered trademark), Bluetooth (registered trademark), information transmitted from mobile base stations, etc., so if GPS information cannot be obtained, the position information of the HMD1 is obtained from such information. Of course, other information may be combined with the GPS information. Hereinafter, information for obtaining global position information will be referred to as "GPS information, etc."

[0032] In addition, in an environment where the above global position information can always be acquired with high accuracy, the user's trajectory information can be arranged in a spatial coordinate system using the global position information. On the other hand, a unique spatial coordinate system (local coordinate system) with the trajectory acquisition start position as the origin may be generated, and a change in position may be calculated from the user's moving distance and moving direction detected by the acceleration sensor 54, the gyro sensor 55, etc. of the sensor unit 50, and arranged in the local coordinate system. Then, by acquiring the global position information at a point where the global position information can be acquired with high accuracy and associating it with the position information of the local coordinate system, it can be applied, for example, to cases where trajectories are not acquired continuously or when viewing other people's trajectory information.

[0033] S114: Determine whether or not a shooting instruction has been given by the user. If a shooting instruction has been given, proceed to S115. If not, proceed to S116. The shooting function of the imaging unit 71 provides two shooting modes: still image shooting and video shooting. To distinguish between these, for example, a voice command "SHOT" is used to instruct shooting of a still image, and a voice command "START SHOT" is used to instruct shooting of a video. Also, for video, a voice command "END SHOT" is used to instruct shooting to end. S115: An object in front of the HMD 1 is photographed by the imaging unit 71 of the image processing unit 5 (photographed data acquisition function 14), and the photographed data is stored as various data 42 in the storage unit 40 (photographed data storage function 15).

[0034] S116: Determine whether the user has instructed to end the trajectory information acquisition process. For example, it is determined that the voice command "Kiseki owari" is an instruction to end the trajectory collection process. If an instruction to end is given, proceed to S117 and end the trajectory information acquisition process. If an instruction to end the trajectory information acquisition process is not given, proceed to S118. S118: Determine whether or not the unit time has elapsed. Specifically, determine whether or not the timer has exceeded the unit time. If the unit time has elapsed, proceed to S119, and if the unit time has not elapsed, return to S114. S119: The point number p is incremented by 1, and the process returns to S112 to reset the timer and restart it.

[0035] The above trajectory collection process (S100) allows the user's trajectory points (position information) to be collected at regular intervals. For example, by setting the timer unit time to one second, the trajectory of the HMD1 can be saved every second. In parallel with this, the subject along the way can be photographed and saved according to the user's instructions.

[0036] Here, two methods will be described for the trajectory information data (data table) stored in the storage unit 40 by the above trajectory collection process (S100).

[0037] FIG. 7A shows a trajectory information storage table 710 that stores position information as measured values. The table consists of items such as point numbers 711 indicating the order of each trajectory point, and position coordinates 712 corresponding to the trajectory point. The position coordinates 712 use planar position information (X, Y) from GPS information or the like and altitude (Y) from the altitude sensor 56. Point number 0 is the position coordinates (X0, Y0, Z0) at the time when the trajectory collection process is started (S110), and point number 1 is the position coordinates (X1, Y1, Z1) after a unit time (1 second here). Point number k is the position coordinates (Xk, Yk, Zk) at the time when a shooting instruction (S114) is given by the user k seconds later. Point number n is the position coordinates (Xn, Yn, Zn) at the time when the trajectory collection process is ended (S117) n seconds after the start.

[0038] In addition, since the position coordinates 712 are fixed-length data in the data table, if they are recorded in chronological order, the data of the point number 711 can be omitted. In other words, the position coordinate data of the target point number p can be searched for using the fixed-length multiple value p.

[0039] Here, if position information cannot be acquired from GPS information or the like, relative value information (difference value information) from the acceleration sensor 54, gyro sensor 55, or the like of the sensor unit 50 is temporarily stored. Then, at the stage where position information (absolute value information) from GPS information or the like can be acquired at any trajectory point, the temporarily stored relative value is corrected to an absolute value.

[0040] 7B shows a differential trajectory information storage table 720 that stores position information as differential values. When there is little change in the measurement value between trajectory points, storing the difference value (amount of change) rather than the measurement value as is is more practical because it reduces the amount of data to be stored.

[0041] It consists of items such as a point number 721 and differential position coordinates 722 corresponding to that trajectory point. The differential position coordinates 722 describe the difference between the position coordinates of the previous trajectory point and the position coordinates of the current trajectory point. That is, the differential value (ΔX, ΔY) of the planar position coordinates is described as the longitude distance and latitude distance between two trajectory points. The height difference value (ΔZ) is also described as a distance.

[0042] Point number 1 indicates the differential position coordinates (ΔX1, ΔY1, ΔZ1) 1 second after the start of the trajectory collection process, point number k indicates the differential position coordinates (ΔXk, ΔYk, ΔZk) k seconds later, and point number n indicates the differential position coordinates (ΔXn, ΔYn, ΔZn) at the end of the trajectory collection process (n seconds after the start). The relationship with the value of the position coordinate 712 in FIG. 7A (trajectory information storage table 710) is as follows: ΔXp=Xp-Xp-1, ΔYp=Yp-Yp-1, ΔZp=Zp-Zp-1 It becomes.

[0043] Fig. 7C shows a start point / end point position coordinate table 730 that stores position information at the start / end of trajectory collection. This table is required when calculating the display position (absolute position) of the avatar using the difference trajectory information storage table 720 in Fig. 7B.

[0044] The items are composed of start point / end point distinction 731, which distinguishes between the start point and the end point of the trajectory collection, and position coordinates 732, which indicate the respective position information in absolute values. Naturally, the start point is equal to the position coordinates (X0, Y0, Z0) at point number 0 in Fig. 7A (trajectory information storage table 710), and the end point is equal to the position coordinates (Xn, Yn, Zn) at point number n.

[0045] In this way, by storing position information (absolute values) 732 of the start point and end point, it is possible to convert the differential trajectory information storage table 720 in Fig. 7B into the trajectory information storage table 710 in Fig. 7A. In particular, by storing the position coordinates (Xn, Yn, Zn) of the end point, it becomes possible to efficiently display the avatar. The reason for this will be explained below.

[0046] As will be described later, the avatar is displayed in reverse time from the end point of the trajectory (point number n). In this case, if the position coordinates of the end point where display begins are unknown, all of the differential position coordinates 722 in FIG. 7B must be added to the position coordinates (X0, Y0, Z0) of the start point in FIG. 7C in order to calculate them. If the position coordinates (Xn, Yn, Zn) of the end point are known, the avatar is displayed at that position coordinate first. The position where the avatar is to be displayed next can be easily calculated by subtracting the differential position coordinates (ΔXn, ΔYn, ΔZn) at point number n in FIG. 7C from the current position coordinates. The same process can be repeated thereafter.

[0047] These pieces of trajectory information (trajectory information storage table 710, differential trajectory information storage table 720, start point / end point position coordinate table 730) can be stored not only in the various data 42 in the storage unit 40, but also in an external server 6 via a network 5 for each trajectory point or when the trajectory information table is completed, as described in Fig. 3. By storing the data in the server 6, the amount of data stored in the HMD 1 can be reduced.

[0048] Next, we will explain how to save the captured image data. The captured image data can be both still images and videos. 8A shows a still image data storage table 810 for storing still images. The still image data storage table 810 is composed of a point number 811 indicating the trajectory point at the time when a shooting instruction was issued, an image capturing direction 812 indicating the image capturing direction, an image capturing data size 813, and image capturing data 814. In this example, the HMD 1 captures an image in the direction of Θk at the trajectory point k (point number k), and the image capturing data Dk of the image capturing data size Mk is stored in the various data 42 of the storage unit 40 (S115).

[0049] By linking and storing the above-mentioned trajectory information storage table 710 and the still image data storage table 810 of this example, it is possible to play back still image footage captured in synchronization with the movement position of the HMD1 per unit time (here, 1 second).

[0050] Fig. 8B shows a video data storage table 820 for saving videos. As with the still image in Fig. 8A, the video data storage table 820 is made up of a point number 821 indicating the trajectory point at the time when a shooting instruction was issued, a shooting direction 822 at the start of shooting, shooting data size 823, and shooting data 824. Furthermore, in the case of a video, shooting time (start and end) 825 is described. This is to accommodate cases where a video is shot across multiple trajectory points. If the video is played back every unit time (every second), it can be synchronized with the avatar display time.

[0051] In this example, the HMD 1 starts shooting at trajectory point k (point number k) in the direction Θk from start time Tks to end time Tke, and the shooting data Dk of data size Mk captured until end time Tke is stored in the various data 42 of the storage unit 40 (S115). The video data may be played back on the entire display screen of the HMD, but in this example, it is displayed on a reduced sub-screen and played back from the video shooting start time Tks to the video shooting end time Tke.

[0052] As described in FIG. 3, information on these shooting data (still image data storage table 810, video data storage table 820) can also be stored in an external server 6 via network 5, thereby reducing the amount of data to be stored in the HMD 1.

[0053] Next, the avatar display process (S200) of the HMD1 will be described. This process procedure is stored in the various programs 41 of the storage unit 40. For display of the avatar, avatar information is extracted from the avatar information storage function 16 (avatar generation function 17), and the avatar is displayed on the display unit 72 of the HMD1 (avatar display function 18). The shape of the avatar is life-size with respect to the user, and the height of the avatar is set by the user himself. For convenience, the height can be determined by adding 10 cm to the distance from the ground (the height of the HMD1 from the ground) measured by the distance sensor 53 of the sensor unit 50 in a standing state. The display position of the avatar is superimposed on the background image on the display unit 72 of the HMD according to the position information stored in the trajectory information storage table 710 or the like.

[0054] If the trajectory information and shooting data to be used are stored in the various data 42 in the HMD, the trajectory information and shooting data are used, but if they are stored in an external server 6 via the LAN communication unit 61 of the communication processing unit 60, they are obtained from the server 6 and used.

[0055] In this embodiment, the avatars are displayed in a chronologically reverse order from newer trajectory information to older trajectory information based on the user's trajectory points. By displaying the avatars in a chronologically reverse order like this, the user can return from the end point (current location) to the start point (departure point) along the same route that he or she actually took.

[0056] FIG. 9A is a flowchart showing the entire avatar display process (S200). S210: The avatar display process (S200) is started. In this example, when the user utters the voice command "AVATAR START", which means to start avatar display, it is determined that this is an instruction to start avatar display.

[0057] S211: The current position and direction of the HMD 1 worn by the user is detected. The detection method is the same as S113 in the trajectory collection process (S100), and is performed by the GPS sensor 51, the geomagnetic sensor 52, the height sensor 56, etc. of the sensor unit 50. S212: Refer to the trajectory information storage table 710 (or the differential trajectory information storage table 720) to search for the point number s of the position coordinates closest to the current position of the HMD 1. That is, if the current position is the end point of the trajectory collection, then s=n, but if the user subsequently moves from the end point, then s≠n may occur. S213: Set the point number s closest to the current position as the initial value of the point number m to be displayed. This allows the display to start from the trajectory point closest to the user's current position.

[0058] S214: A timer is started. The timer uses a clock built in the main control unit 10. The timer is used in order to display the user's trajectory according to the trajectory collected at a fixed time interval (unit time). However, the unit time used for displaying the avatar can be made different from the unit time for the trajectory collection process and multiplied by an arbitrary coefficient. For example, if the unit time used for displaying the avatar is set to 1 / 2 the time (0.5 seconds in this example), the displayed avatar can be moved at twice the speed. Conversely, if the unit time used for displaying the avatar is set to twice the time (2 seconds in this example), the displayed avatar can be moved at 1 / 2 the speed (slow). Also, the avatar can be displayed as a moving image (animation) by displaying the avatar by interpolating between trajectory points in sequence (for example, every 1 / 24 seconds).

[0059] S215: Calculate the position and direction to display the avatar. If the trajectory information is stored in the trajectory information storage table 710, read the value (Xm, Ym, Zm) of the position coordinates 712 corresponding to the point number m, and place the avatar at the planar position (Xm, Ym) and altitude (Zm). If the trajectory information is stored in the differential trajectory information storage table 720, the value (difference value) of the differential position coordinates 722 at point number (m+1) is read out and subtracted from the previous display position (point number=m+1). Xm=Xm+1-ΔXm+1,Ym=Ym+1-ΔYm+1,Zm=Zm+1-ΔZm+1 However, for the first time (m=s), the calculation is performed using the end point position coordinates (Xn, Yn, Zn) described in the start point / end point position coordinate table 730 and the value of the differential position coordinate 722 up to point number s. For the sake of convenience, the direction in which the avatar is facing is taken to be the direction connecting the previous trajectory point (m+1) and the current trajectory point (m).

[0060] S216: Detect the current position and direction of the HMD 1. This is the same process as S211, and can be omitted for the first time (m=s) since it has already been detected in S211. S217: The field of view calculation function 19 calculates the display area of ​​the HMD 1, i.e., the field of view of the user, from the current position and direction of the HMD 1, and judges whether the placement position of the avatar calculated in S215 is within the display area. If it is judged that the avatar can be placed within the display area of ​​the HMD 1, the process proceeds to S218, and if it is judged that the avatar cannot be placed, the rescue process of S300 is performed. The rescue process of S300 performs a guidance process such as having the user change the direction of the field of view or moving the user to a position where the avatar can be seen, and details will be described later with reference to FIG. 9B. By the judgment process of S217 and the rescue process of S300, the avatar can be placed within the field of view of the user, and the user can move (track) the avatar without losing sight of it or overtaking it.

[0061] S218: If it is determined that the avatar can be displayed within the display area of ​​the HMD 1, an avatar is generated by reading avatar information from the avatar information storage function 16. Then, the avatar is displayed on the display unit 72 of the HMD 1 at the avatar display position and orientation calculated in S215.

[0062] S219: It is determined whether the currently displayed point number m is a photographing point k by referring to the still image data storage table 810 or the video data storage table 820. If it is determined that the trajectory point is a photographing point, the process proceeds to S220. If it is determined that the trajectory point is not a photographing point, the process proceeds to S222. S220: Notify the user that the current time point is the shooting point. The shooting point is notified to the user by changing the color of the displayed avatar or by making the avatar blink. S221: In response to a user's instruction, the corresponding photographic data is read from the still image data storage table 810 or the video data storage table 820 and played. The playback of the photographic data is executed by the voice command "play". The photographic data may be played on the entire display screen of the HMD, but in this example, it is displayed on a reduced sub-screen, and the playback of the photographic data is terminated after the display is continued for a certain period of time.

[0063] S222: Determine whether the user has instructed to end the avatar display process. For example, it is determined that the voice command "end avatar" is an instruction to end the avatar display process. If an instruction to end the avatar display process has been given, proceed to S226 and end the avatar display process. S223: Determine whether or not the unit time has elapsed. If it is determined that the unit time has elapsed, proceed to S224. If it is determined that the unit time has not elapsed, return to S222. S224: Subtract 1 from point number m to obtain a new point number. This returns the point to the previous trajectory point in time.

[0064] S225: Determine whether the value of the new point number m is less than 0 (i.e., whether the starting point of the trajectory m=0 has been exceeded). If the value of the point number m is not less than 0, return to S214 and continue avatar display for the next trajectory point. If the value of the point number m is less than 0, avatar display has been completed at all trajectory points, so proceed to S226 and end the avatar display process.

[0065] Fig. 9B is a flow chart showing the rescue process (S300) in Fig. 9A. This process is performed if the determination in S217 is No, that is, if the avatar cannot be placed within the display area. After the process ends, the process returns to S211.

[0066] S301: The field of view calculation function 20 determines whether it is possible to position the avatar by changing the direction of the HMD 1. If it is possible, proceed to S302, and if it is not possible, proceed to S303. S302: This corresponds to the case where the HMD is not facing the direction of the avatar, so the user is guided by voice or display to change the direction of the HMD. For example, the voice decoding unit 83 of the voice processing unit 80 generates voice guidance such as "right" or "left." At that time, the avatar may be displayed on the screen and made to speak. After that, the process returns to S211.

[0067] Fig. 10A is a diagram showing an example of the direction change guide display in S302. An avatar 3 appears on the display screen 400, and a message 401 is displayed indicating that the avatar is on the right side. In response to this, when the user turns to the right, the new direction of the HMD is detected in S216 in Fig. 9A. It is determined in S217 that the avatar is present in the new display area, and the avatar can be displayed in S218.

[0068] S303: If it is not possible to position the avatar by changing the direction of the HMD 1 in S301, it is assumed that the current position is far from any of the points from which the trajectory has been collected. In this case, it is determined whether map information can be acquired to guide the user to the nearest point. For example, if a navigation application is available, the process proceeds to S304. If map information cannot be acquired, the process proceeds to S306.

[0069] S304: Using the navigation application, a route analysis is performed to the closest trajectory point s found in S212. S305: Route information (new trajectory point) up to trajectory point s is added to the trajectory information storage table 710. After that, the process returns to S211, and the added trajectory point is included in the display target position of the avatar. As a result, in S217, the avatar is placed within the display area of ​​the HMD, and the avatar can be displayed in S218.

[0070] S306: If map information cannot be acquired in S303, the direction of travel to the nearest trajectory point s is presented to the user. The presentation method is audio or AR display. Then, the process returns to S211. If the user moves in the presented direction, the user can approach the nearest trajectory point s. As a result, the avatar is placed in the display area in S217, and the avatar can be displayed in S218.

[0071] 10B is a diagram showing an example of the movement direction guide display in S306. The display screen 400 informs the user of the direction in which to move by AR display 402. The AR display 402 only needs to show the direction to move, and uses characters, icons (arrows), etc.

[0072] As described above, if it is determined that the avatar cannot be placed in the HMD display area, the rescue process of S300 is performed to guide the user to a position and direction where the avatar can be seen, allowing the user to move (track) the avatar without losing sight of it.

[0073] 11A and 11B are diagrams showing examples of avatar display, which correspond to the schematic diagram of trajectory collection shown in FIG. On the display screen 400 in FIG. 11A, the avatar 3 is displayed by moving each trajectory point of the HMD whose trajectory was collected in reverse time. That is, the display position of the avatar 3 is displayed by moving in the direction of the arrow in the order of trajectory points 504, 503, 502, and 501 (indicated by ● marks). Among these, trajectory point 503 is the photographing point k, and is therefore indicated by a ◎ mark to distinguish it from the other positions. Also, in order to reproduce the direction in which the user 2 (HMD) was facing during the trajectory collection process, the orientation of the body when moving the avatar 3 is set to the opposite direction (backwards) to the traveling direction of the avatar 3, and the state of returning along the original route is shown.

[0074] Display screen 400 in Fig. 11B shows the display state of avatar 3 at trajectory point 503. Because trajectory point 503 is a corner, avatar 3 changes direction. In addition, because trajectory point 503 is a shooting point, avatar 3 is displayed in a different color. When the user gives an instruction to play back the shooting data, image 510' of subject 510 captured at the shooting point is displayed.

[0075] In this example, to inform the user of the return route, trajectory points (● marks) and a travel direction (arrow) are displayed on the display screen 400, but these can also be hidden. In addition, the avatar's trajectory points are displayed every unit time (1 second), but they can also be displayed at other time intervals. For example, if they are displayed every 1 / 24 seconds, the avatar will be displayed as a movie-like smooth animation.

[0076] Next, a description will be given of a method for a user to operate the HMD 1. As described above, in this embodiment, voice commands are used as the user's operations. 12 is a table showing examples of voice commands used to operate the HMD. The items in the voice command table 600 are composed of command classification 601, voice command 602 issued by the user, command name 603, and processing content 604.

[0077] The voice command classification 601 is divided into trajectory collection-related and avatar display-related. The voice commands 602 related to trajectory collection include "Kiseki Start" and "Kiseki Start" to start trajectory collection processing, "Kiseki End" and "Kiseki End" to end trajectory collection processing, "Shoot" and "Record" to take still images, "Shoot" and "Record" to start video recording, "Shoot" and "Record Start" to start video recording, and "Shoot" and "Record End" to end video recording.

[0078] The voice commands 602 related to avatar display include "AVAILABILITY START" and "AVAILABILITY START" for starting the avatar display process, "AVAILABILITY END" and "AVAILABILITY END" for ending the avatar display process, "PLAY" and "PLAYBACK DATA" for playing still images or videos, etc. Note that the voice commands listed here are merely examples, and needless to say, they can be set as appropriate according to the user's preferences.

[0079] On the other hand, the HMD 1 can use the voice synthesis function of the voice decoding unit 83 of the voice processing unit 80 to make a voice response corresponding to the voice command issued by the user. As a result, for example, it is possible to respond with the same voice (echo) as the user's voice command, or to make a voice response to confirm the user's voice command. For example, in response to a query from the HMD "Is it XX?", if the user responds with "Yes", the HMD responds with a confirming voice "I will execute XX", and if the user responds with "No", the HMD responds with a confirming voice "I will execute XX". Note that the voice commands "Yes" and "No" used by the user to respond are not shown in FIG. 12, but it goes without saying that they are also included in the voice command table 600. Furthermore, by using the voice synthesis function of the voice decoding unit 83, it is possible to generate synthetic voices such as "right" to encourage a right turn and "left" to encourage a left turn. This can be used in the step S306 in FIG. 9B.

[0080] Operation instructions for the HMD 1 can be realized by methods other than voice commands. For example, an operation menu can be displayed on the display screen of the HMD 1, and the user can select with a gesture (movement of hand or finger). In this case, the image capturing unit 71 of the image processing unit 70 captures an image of the user's finger, and the finger movement can be recognized from the captured image and used for operation.

[0081] 13 is a diagram for explaining an operation instruction by a user's finger. For example, a "Photo" menu 411 and a "Video" menu 412 are displayed as selection menus on an operation screen 410 of the HMD 1. The menu display is preferably semi-transparent, but is not limited to this.

[0082] When the user approaches the display screen 410 from outside the HMD 1 with a finger 415 to select, for example, the "Video" menu 412, the imaging unit 71 detects the finger image 415 that gradually grows larger toward the display screen 410. It also recognizes that the finger image 415 is pointing toward the "Video" menu 412, and determines that "Video" has been selected. As for other operations, when starting trajectory collection or avatar display, only the "Start" menu is displayed, and when ending, only the "End" menu is displayed. Then, the imaging unit 71 detects the approach of the user's finger image 415 and determines that an instruction to start or end has been given. Alternatively, a touch sensor may be provided on the operation input unit 90 of the HMD 1 to detect when the HMD 1 is touched by the user's hand or finger.

[0083] According to the first embodiment, it is possible to provide easy-to-understand guidance on the return route to a user who has forgotten where he or she stopped along the way or the return route. EXAMPLES

[0084] In the first embodiment, the position coordinates of the HMD are stored in three dimensions (X, Y, Z), but in the second embodiment, a case where the position coordinates are stored in two dimensions (X, Y) will be described. The basic configuration of the HMD in the second embodiment is the same as that in the first embodiment, and the differences from the first embodiment will be described.

[0085] Since there is little change in the height direction when moving on a normal flat surface, the avatar can be displayed based on the horizontal position coordinates even if the height data is omitted. The trajectory collection process is performed according to the same flowchart (S100) as in the first embodiment (FIG. 6). The collected data is then stored in the tables shown in FIGS. 14A and 14B.

[0086] Fig. 14A shows a two-dimensional trajectory information storage table 740, where position information is stored as a differential value. The table consists of items such as point number 741 indicating the number of each trajectory point, and differential position coordinate 742 indicating the planar position of the trajectory point. As in Fig. 7B above, differential position coordinate 742 is not an absolute value of the position coordinate, but a differential value (amount of change) between adjacent trajectory points, thereby reducing the amount of data to be stored.

[0087] In this example, the unit time is set to 1 second, so point number 1 indicates the differential position coordinates (ΔX1, ΔY1) 1 second after the trajectory collection process is started, point number k indicates the differential position coordinates (ΔXk, ΔYk) k seconds later, and point number n indicates the differential position coordinates (ΔXn, ΔYn) at the time when the trajectory collection process ends (n seconds after the start). The method of calculating the differential position coordinates is the same as in Example 1 (FIG. 7B).

[0088] Fig. 14B shows a two-dimensional start point / end point position coordinate table 750 that stores position information at the start / end of trajectory collection. This table is required when calculating the display position (absolute position) of the avatar using the two-dimensional difference trajectory information storage table 740 in Fig. 14A.

[0089] The items are composed of start / end point distinction 751, which distinguishes between the start point and end point of the trajectory collection, and position coordinates 752, which indicate the respective position information in absolute values, as in Fig. 7C above. However, in the position coordinates 752, only the position coordinates (X0, Y0, Z0) of the start point are saved as the position coordinate in the height direction (Z0). This is to handle the case where the amount of change in altitude exceeds a predetermined value (threshold) while acquiring trajectory information.

[0090] In the second embodiment, only two-dimensional position coordinates are handled, but when the altitude changes, the following processing is performed. When going up (down) a slope, the change in the HMD's planar position coordinates (ΔX, ΔY) is linear, and the change in the HMD's altitude (ΔZ) is also linear. When going up (down) stairs, the change in the HMD's planar position coordinates (ΔX, ΔY) is the width of the stairs, and the change in the HMD's altitude (ΔZ) is the difference in the height of the stairs.

[0091] In the case of an escalator, the change in the HMD's planar position coordinates (ΔX, ΔY) is linear, and the change in the HMD's altitude (ΔZ) is also linear. An escalator can be distinguished from a slope because the change in acceleration is smaller for an escalator. In the case of an elevator, the HMD's planar position coordinates barely change (ΔX=0, ΔY=0), and only the altitude of the HMD changes (ΔZ≠0).

[0092] In this way, when moving on slopes, stairs, escalators, elevators, etc., characteristic changes (ΔX, ΔY) and (ΔZ) occur in the detected values ​​of planar position coordinates and altitude, respectively, so when displaying the avatar, these changes are captured and route information (slope, stairs, escalators, elevators, etc.) is displayed. In addition, by displaying the avatar superimposed on images of slopes, stairs, escalators, elevators, etc., a realistic avatar can be displayed. Furthermore, great effects can be obtained by linking it to map information such as an underground mall map.

[0093] If the amount of change in altitude exceeds a predetermined value (threshold), the trajectory collection process using the two-dimensional trajectory information storage tables 740, 750 is terminated with trajectory A. Then, the process moves to a new trajectory collection process using the three-dimensional trajectory information storage tables 710 to 730 of the first embodiment, and trajectory B is selected. After that, if the amount of change in altitude falls within the threshold, the trajectory collection process using the two-dimensional trajectory information storage tables 740, 750 is resumed and trajectory C is selected. This method can minimize the amount of data to be stored. Of course, it is also possible to convert the two-dimensional trajectory information storage tables 740 and 750 of the second embodiment into the three-dimensional trajectory information storage tables 710 to 730 of the first embodiment.

[0094] In the above-described first and second embodiments, it is assumed that the GPS information (location information) is relatively easily acquired by the GPS sensor 51 of the sensor unit 50 outdoors, but there are cases where it is difficult to acquire GPS information, such as underground malls and indoors. When the GPS information cannot be acquired, it is supplemented by information from the acceleration sensor 54 and the gyro sensor 55 of the sensor unit 50.

[0095] If no position information can be obtained from GPS information or the like from the start point to the end point, the position coordinates of the start point are stored as X0 = 0, Y0 = 0. In that case, the height position coordinate is also stored as Z0 = 0. Furthermore, the position coordinate of the end point is calculated by accumulating all the differential position coordinates from the start point to the end point. EXAMPLES

[0096] In the first embodiment, the avatar is displayed facing backwards, with its orientation reversed from the direction of travel. In contrast, in the third embodiment, the avatar is displayed facing the direction of travel. 15 is a schematic diagram of an avatar display in Example 3. On the display screen 400, the avatar 3 is displayed facing the moving direction (facing forward). That is, the orientation of the avatar 3 is set to the vector direction from the current trajectory point 503 to the next trajectory point 502. This makes it possible to eliminate the sense of discomfort felt by the user due to the orientation of the avatar as shown in Example 1 (FIG. 11B) (the moving direction of the avatar and the orientation of the avatar do not match).

[0097] In addition, whether the avatar faces forward (Example 3) or backward (Example 1) also depends on the user's preference. Therefore, it is more preferable that the user can select and set the display form in which the avatar faces. EXAMPLES

[0098] In the fourth embodiment, when a building or the like is present in the field of view of the HMD and the avatar is located in its shadow, the avatar is hidden. 16 is a schematic diagram of avatar display in Example 4. Here, avatar 3 is moving backwards. Buildings 521 and 522 are present near the moving path, and avatar 3 is hidden by building 521. In such a case, avatar 3 is controlled so as not to display the part hidden by building 521. Whether avatar 3 is hidden or not can be determined by comparing the distance to avatar 3 with buildings 521 and 522, and the distance to buildings 521 and 522 can be measured by distance sensor 53.

[0099] This allows for a natural and realistic display when the avatar is superimposed on the real space visible through the HMD. However, if the entire avatar is hidden and completely hidden, it may become impossible to track it after that. Therefore, it is possible to make the hidden avatar part semi-transparent so that it can be tracked.

[0100] Images of the buildings 521 and 522 can be displayed using, for example, Street View (Google Inc.), an Internet service that provides panoramic photos of roadside scenery. In addition, depending on the position and direction of the HMD 1, names of buildings and the like associated with the buildings and the like can also be used. EXAMPLES

[0101] In the fifth embodiment, the image capturing is performed automatically, not by a user. The imaging unit 71 of the HMD starts capturing images from the starting point, and constantly captures and temporarily stores images for a certain period of time from the current time (for example, twice the length of the unit time between trajectory points used in the trajectory collection process). This automatically captures video or still images from a certain period of time (for example, 2 seconds) back from the current time. In the case of still images, the function can be fully realized by temporarily storing one or two still images between one trajectory point and the next trajectory point. Then, images captured at unique points on the movement path, for example, at points where the movement direction is changed, are saved.

[0102] A specific operation of the automatic shooting will be described with reference to the trajectory collection schematic diagram of Fig. 5. In Fig. 5, at the time of trajectory point 504, the video (twice the length of the unit time) from trajectory point 502 to trajectory point 504 is temporarily stored. However, at trajectory point 503, the movement direction (movement vector between points) changes by 90 degrees. Therefore, the video shot from trajectory point 502 to trajectory point 503 is automatically saved as the video just before the direction of the movement vector changes significantly. The video data of trajectory point 503 to be saved may be a video (video frames shot up to trajectory point 503) or a still image (shooting data of trajectory point 503).

[0103] According to the shooting method of the fifth embodiment, when the user changes the direction of travel, an image of the direction of travel just before the change in direction of travel is saved, so that an image optimal for evoking the user's memory can be displayed. EXAMPLES

[0104] In the sixth embodiment, information such as the user's posture is also collected in the trajectory collection process, and is reflected when the avatar is displayed. For example, the distance sensor 53 of the sensor unit 50 detects the distance from the ground or floor surface to the HMD 1 (i.e., the height of the user's head) and collects it for each trajectory point. By comparing this detection value with the user's height (known), the posture (standing, sitting, crouching, sleeping, etc.) of the user wearing the HMD 1 can be estimated. This posture information is reflected in the displayed avatar, and the avatar is displayed as a standing avatar, a sitting avatar, etc.

[0105] Furthermore, the geomagnetic sensor 52 of the sensor unit 50 can detect the facial direction of the user wearing the HMD 1. This facial direction information is reflected in the displayed avatar, and the facial direction of the avatar is changed for each trajectory point. The body direction of the avatar may be estimated and changed based on the facial direction, or may be adjusted to match the vector information (direction of travel) between trajectory points. These displays make it possible to display a more realistic avatar. EXAMPLES

[0106] In the seventh embodiment, the time at which each trajectory point is passed is stored in the trajectory collection process, and the passing time is displayed on the display screen when the avatar is displayed. Fig. 17A shows an example of a trajectory information storage table 710' in the embodiment 7. In addition to the trajectory information storage table 710 in the embodiment 1 (Fig. 7A), passing times 713 at which each trajectory point was passed are added.

[0107] Fig. 17B is a diagram showing an example of an avatar display. Display screen 400 displays avatar 3 along with passage time 420. This time can be obtained by referring to passage time 713 corresponding to point number 711 at which the avatar is currently displayed in trajectory information storage table 710' in Fig. 17A.

[0108] In this way, according to the seventh embodiment, the time when the user passed the currently displayed trajectory point is displayed, which is effective information for recalling the user's memory. EXAMPLES

[0109] In Example 8, a case will be described in which a smartphone is used as a portable image display device instead of an HMD. The smartphone has almost the same hardware and software configurations as the HMD, and can achieve the same functions.

[0110] FIG. 18 is a diagram showing the appearance of a smartphone 9, where (a) is the front side that displays images, and (b) is the rear side that has a camera. As shown in (b), a camera 901 is provided on the rear side to capture an external scene. The smartphone 9 has a sensor inside for detecting the three-dimensional position and the direction of the camera 901, and can collect trajectory information in the same way as an HMD.

[0111] As shown in (a), the front side has a display screen 902 with a built-in touch panel, and displays an image captured by a camera 901 directed at an external scene. The display screen 902 can display an avatar 3 superimposed on the displayed external scene image based on the collected trajectory information, as in the first embodiment (FIG. 11A).

[0112] Compared to the HMD, in the case of the smartphone 9, the photographing process for collecting trajectory information requires the camera 901 to be directed toward the subject each time, and the photographing operation becomes cumbersome when the number of photographing points increases. In addition, automatic photographing has a drawback in that it is not realistic for the user to maintain the photographing direction of the camera 901 constant. However, since trajectory information can be collected with an existing smartphone, convenience is improved. However, since it is difficult for the user to move while photographing, it is more realistic to perform the process only at the main points alone.

[0113] Although a smartphone is used as an example of a portable image display device, the operation of this embodiment can be realized with an equivalent or similar hardware configuration or software configuration. For example, the present invention can also be applied to a notebook PC, a tablet PC, etc. EXAMPLES

[0114] In the ninth embodiment, a video display system used in cooperation with a plurality of users wearing HMDs will be described. 19 is a diagram showing the configuration of a video display system to which multiple HMDs are connected. Here, the system shows the case of two users 2a and 2b, who are wearing HMDs 1a and 1b, respectively. The same applies to cases other than two users.

[0115] The HMD 1a worn by the user 2a is connected to an external server 6 via a wireless router 4a and a network 5. Moreover, the HMD 1b worn by the user 2b is connected to the same server 6 via a wireless router 4b and the network 5. When the HMD 1a (user 2a) and the HMD 1b (user 2b) are in close proximity, the wireless routers 4a and 4b can be shared.

[0116] In this case, the HMDs 1a and 1b collect the trajectories of the users 2a and 2b, respectively. Then, the trajectory information of the user 2a wearing the HMD 1a and the trajectory information of the user 2b wearing the HMD 1b are stored in a common server 6. After that, the trajectory information of the two people is read from the server 6 and displayed as two avatars, which are virtual images of the two people, on the HMDs 1a and 1b, respectively. This allows each trajectory to be mutually referenced by the avatar.

[0117] 20 is a schematic diagram showing a display example of a plurality of avatars. For example, on the display screen 400a of the HMD 1a, in addition to the avatar 3a of the user 2a wearing the HMD 1a, an avatar 3b of the user 2b wearing the HMD 1b is displayed. Then, the trajectories 520a and 520b of the actions of the two people are displayed as respective trajectory points (● and ▲ marks), and the positional relationship of the two people can be displayed in reverse chronological order. In this example, it can be seen that the two people met at the intersection of trajectory point 521. The same display is also shown on the display screen 400b of the HMD 1b.

[0118] In this way, according to the ninth embodiment, a plurality of users can mutually confirm each other's trajectories. As a result, as an additional effect, when another user goes out of sight during trajectory collection, the position of the other user can be immediately known.

[0119] As a practical embodiment of this embodiment, it is possible to use short-range wireless communication such as Bluetooth (registered trademark) to communicate between portable information terminals, without using the wireless router 4 or the server 6 via the network 5. For example, if a parent and child are traveling and the child is given a smartphone, and goes missing, the parent can immediately search for the child using the HMD or smartphone.

[0120] Although the embodiments of the present invention have been described above using Examples 1 to 9, it goes without saying that the configurations for realizing the technology of the present invention are not limited to the above-mentioned examples, and various modified examples are possible. For example, it is possible to replace a part of the configuration of one example with the configuration of another example, and it is also possible to add the configuration of another example to the configuration of one example. All of these belong to the scope of the present invention. In addition, the numerical values ​​and messages that appear in the text and figures are merely examples, and the effect of the present invention is not impaired even if different ones are used.

[0121] The functions of the present invention described above may be realized in part or in whole by hardware, for example by designing them as an integrated circuit. They may also be realized by software, with a microprocessor unit or the like interpreting and executing a program that realizes each function. Hardware and software may be used in combination. The software may be stored in the various programs 41 of the HMD 1 beforehand at the time of product shipment. The software may be obtained from various server devices on the Internet after product shipment. The software may also be obtained by being provided on a memory card, optical disk, or the like.

[0122] In addition, the control lines and information lines shown in the diagram are those considered necessary for explanation, and do not necessarily show all the control lines and information lines on the product. In reality, it can be considered that almost all components are connected to each other. [Explanation of symbols]

[0123] 1...HMD (image display device), 2...user, 3...avatar, 4...wireless router, 5...network, 6...server, 9...smartphone (image display device) 10...main control unit, 40...memory unit, 50...sensor unit, 51...GPS sensor, 52...geomagnetic sensor, 53...distance sensor, 56...altitude sensor, 60...communication processing unit, 70...image processing unit, 71...imaging unit, 72...display unit, 80...audio processing unit, 90...operation input unit.

Claims

1. In a video display device carried by a user and displaying a moving trajectory of the user as a video, A plurality of users use the image display device by wearing it on their heads, The image display devices each include a sensor for detecting the position, direction, and head height of each user wearing the image display device; a trajectory information acquisition unit that acquires information on a movement trajectory of each of the users from a detection result of the sensor; a communication processing unit that transmits and receives trajectory information of the user and other users and avatar information, which is a virtual image representing the user and the other users, between a video display device carried by the user and a video display device carried by another user; a storage unit that stores the user's trajectory information acquired by the trajectory information acquisition unit, the other users' trajectory information acquired by the communication processing unit, and avatar information of the user and the other users; a display unit that displays the movement trajectories of the user and other users using avatars of the user and other users; a control unit that controls the trajectory information acquisition unit, the communication processing unit, the storage unit, and the display unit, The control unit is storing information on the height and direction of the user's head detected by the sensor as the trajectory information in the storage unit; generating avatars of the user and other users from information on the avatars of the user and other users stored in the storage unit, and determining a current visual field of the user by the sensor; based on the user's trajectory information stored in the storage unit and the other users' trajectory information received by the communication processing unit, aligning an avatar of the user on the movement trajectory of the user in accordance with a current field of view of the user, and arranging avatars of other users on the movement trajectories of the other users, and displaying the avatars on the display unit; A video display device characterized in that, when moving the avatar of the user along the movement trajectory of the user and moving the avatar of another user along the movement trajectory of the other user, the avatars are moved from newer trajectory information to older trajectory information in time, and when generating and displaying avatars of the user and other users, the video display device determines the posture and facial direction of the avatar of the user by comparing the information on the height and direction of the user's head stored in the memory unit with the height information of the user.

2. 2. The image display device according to claim 1, A video display device characterized in that, when generating and displaying an avatar of another user, the device determines the posture and facial direction of the avatar of the other user by comparing the information on the height and direction of the head of the other user stored in the memory unit with the height information of the other user.

3. 2. The image display device according to claim 1, The control unit of the image display device is characterized in that, when the user's avatar cannot be placed within the user's current field of view, the control unit performs a guidance process to have the user change their field of view or move to a position where the user's avatar can be seen.

4. 2. The image display device according to claim 1, the trajectory information acquisition unit acquires information on a movement trajectory of each of the users at predetermined unit times and stores position coordinates of each trajectory point in the storage unit; The control unit reads out the position coordinates of each trajectory point from the memory unit for each unit time or for each time multiplied by an arbitrary coefficient of the unit time, and places the user's avatar and the other user's avatars at the position coordinates.

5. 5. The image display device according to claim 4, The video display device is characterized in that the trajectory information acquisition unit stores in the memory unit the position coordinates of each trajectory point as a difference value from the position coordinates of the trajectory point immediately before in time, and stores in the memory unit the position coordinates of at least the end trajectory point as an absolute value.

6. 5. The image display device according to claim 4, Further, an imaging unit is provided for capturing an image of the outside scene, The photographing data photographed by the imaging unit is stored in the storage unit in association with a photographing point which is a photographing position; The image display device is characterized in that the control unit displays an image of the shooting data stored in the memory unit on the display unit when the position of the user's avatar is at the shooting point.

7. 7. The image display device according to claim 6, The control unit stores, in the storage unit, an image captured at a capturing point where a direction of a movement trajectory of each of the users changes, among the captured image data captured by the imaging unit; The control unit displays an image of the shooting data stored in the memory unit on the display unit when the position of the user's avatar is at a shooting point where the direction of the user's movement trajectory has changed.

8. 5. The image display device according to claim 4, Further, an imaging unit is provided for capturing an image of the outside scene, The display unit displays a current image captured by the imaging unit, The image display device is characterized in that the control unit superimposes and displays an avatar of the user based on the movement trajectory of the user, in accordance with the image of the external scene displayed on the display unit.

9. In a video display device carried by a user and displaying a moving trajectory of the user as a video, A plurality of users use the image display device by wearing it on their heads, The image display devices each include a sensor for detecting the position, direction, and head height of each user wearing the image display device; a trajectory information acquisition unit that acquires information on a movement trajectory of each of the users from a detection result of the sensor; a communication processing unit that transmits and receives trajectory information of the user and other users and avatar information, which is a virtual image representing the user and the other users, between a video display device carried by the user and a video display device carried by another user; a storage unit that stores the user's trajectory information acquired by the trajectory information acquisition unit, the other users' trajectory information acquired by the communication processing unit, and avatar information of the user and the other users; a display unit that displays the movement trajectories of the user and other users using avatars of the user and other users; a control unit that controls the trajectory information acquisition unit, the communication processing unit, the storage unit, and the display unit, The control unit is storing information on the height and direction of the user's head detected by the sensor as the trajectory information in the storage unit; generating avatars of the user and other users from information on the avatars of the user and other users stored in the storage unit, and determining a current visual field of the user by the sensor; based on the user's trajectory information stored in the storage unit and the other users' trajectory information received by the communication processing unit, aligning an avatar of the user on the movement trajectory of the user in accordance with a current field of view of the user, and arranging avatars of other users on the movement trajectories of the other users, and displaying the avatars on the display unit; A video display device characterized in that, when moving the user's avatar along the user's movement trajectory and moving another user's avatar along the other user's movement trajectory, the avatars are moved from newer trajectory information to older trajectory information in time, and the body orientation of the user's avatar and the other user's avatar when moving is displayed in a direction opposite to the direction of travel of the user's avatar and the other user's avatar.

10. A video display method for displaying a moving trajectory of a user as an image using a video display device carried by the user, comprising: A plurality of users use the image display device by wearing it on their heads, detecting the position, direction, and head height of each user wearing the image display device; acquiring information on a movement trajectory of each of the users from a detection result of the sensor; transmitting and receiving, between a video display device carried by the user and a video display device carried by another user, trajectory information of the user and the other user and avatar information, which is a virtual image representing the user and the other user; storing the trajectory information of the user acquired by the trajectory information acquisition unit, the trajectory information of the other users acquired by the communication processing unit, and avatar information of the user and the other users; displaying the movement trajectories of the user and other users using avatars of the user and other users; storing information on the height and direction of the user's head detected by the sensor in the storage unit as the trajectory information; generating avatars of the user and other users from information of the avatars of the user and other users stored in the storage unit, and determining a current visual field area of ​​the user by the sensor; a step of arranging an avatar of the user on the movement trajectory of the user and arranging avatars of other users on the movement trajectories of the other users based on the trajectory information of the user stored in the storage unit and the trajectory information of the other users received by the communication processing unit, in accordance with a current visual field of the user, and displaying the avatars of the other users on the movement trajectories of the other users; a step of moving the avatar of the user along the movement trajectory of the user and the avatar of another user along the movement trajectory of the other user, from newer trajectory information to older trajectory information in time, and when generating and displaying avatars of the user and the other users, determining a posture and a face direction of the avatar of the user based on information on the height and direction of the head of the user stored in the storage unit and comparing it with height information of the user; A video display method comprising:

11. 11. The image display method according to claim 10, A video display method comprising a step of, when generating and displaying an avatar of another user, determining the posture and face direction of the avatar of the other user by comparing the head height and direction information of the other user stored in the memory unit with the height information of the other user.

Images