Video display device and video display method

Abstract

To provide a video display device and a video display method which show a backtracking route along a route which a user has actually moved along.SOLUTION: A head mounted display (HMD) 1 being a portable video display device comprises a sensor which detects a position and direction of a user, a track information acquisition unit which acquires information of a movement track of the user, a storage unit in which track information of the user and information of an avatar are stored, a display unit 72 which displays the movement track of the user with the avatar, and a main control unit. The main control unit creates the avatar and obtains user's current visible area by the sensor and displays the avatar on the movement track of the user in the display unit in accordance with the user's current visible area on the basis of the track information of the user stored in the storage unit. When being moved along the movement track of the user, the avatar is moved from the latest piece toward the oldest piece of the track information with respect to time.SELECTED DRAWING: Figure 2

Description

Technical Field

[0004] , , , , , , ,

[0001] The present invention relates to a portable video display device and a video display method.

Background Art

[0002] In recent years, portable video display devices represented by smartphones have become widespread. Among them, a head-mounted display (hereinafter, HMD: Head Mounted Display) worn on the user's head superimposes and displays a video of the real space and a video of augmented reality (AR: Augmented Reality) created by a computer or the like on a spectacle-type display screen. Further, by attaching a sensor to the HMD, information acquired by the sensor can be displayed on the display screen as an AR video. For example, in Patent Document 1, as a portable terminal that depicts a user's behavior history, there are provided a terminal information acquisition means for acquiring position information of its own terminal and the like, a camera means for generating a camera image obtained by imaging the surroundings, a behavior history calculation means for calculating behavior history description information (avatar) to be displayed based on a previously acquired behavior history and the imaging range of the camera means, an image synthesis means for generating a composite image in which the behavior history description information is depicted on the camera image, and a display means for displaying the composite image.

Prior Art Documents

Patent Documents

[0003] <OOO0016>

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Elderly users often forget places they stopped at along the way or their return route while traveling along complex paths. In such cases, there are methods like navigation systems that guide users back to a set starting point, but these do not necessarily guide them along the exact route the user actually took, and it cannot be expected that they will accurately reproduce the intermediate steps.

[0005] In the aforementioned Patent Document 1, when a user instructs to record their activity history, it is registered in the activity history server. Then, the activity history of the desired user is retrieved from the activity history server and an avatar is displayed overlaid on the camera image according to the user's location information. However, Patent Document 1 assumes the retrieval and display of the activity history of any other user, and does not specifically consider displaying the user's own activity history. Furthermore, when displaying the activity history, the avatar is displayed in the order in which the user moved to each location, so it is not suitable as a tool to guide a user who has forgotten their return route back from their current location.

[0006] The objective of the present invention is to provide a video display device that guides the user back along the route they actually traveled. [Means for solving the problem]

[0007] To solve the aforementioned problems, the present invention provides a video display device comprising: a sensor that detects the position and direction of a user carrying the video display device; a trajectory information acquisition unit that acquires information on the user's movement trajectory from the sensor's detection results; a storage unit that stores the user's trajectory information acquired by the trajectory information acquisition unit and information on an avatar, which is a virtual image representing the user; a display unit that displays the user's movement trajectory using the avatar; and a control unit that controls the trajectory information acquisition unit and the display unit. The control unit generates an avatar from the avatar information stored in the storage unit, determines the current user's field of view using the sensor, and, based on the user's trajectory information stored in the storage unit, positions the avatar along the user's movement trajectory to match the current user's field of view and displays it on the display unit. When the control unit moves the avatar along the user's movement trajectory, it moves it from the most recent trajectory information to the oldest trajectory information in time. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide clear directions to a user who has forgotten a place they stopped at along the way or their return route. [Brief explanation of the drawing]

[0009] [Figure 1] External view of the HMD according to Example 1. [Figure 2] A block diagram showing the internal structure of the HMD. [Figure 3] A diagram showing an example of a connection configuration between the communication processing unit and external devices. [Figure 4] A diagram showing the configuration of the functional blocks of the HMD. [Figure 5] A schematic diagram illustrating an example of user trajectory collection. [Figure 6] A flowchart illustrating the trajectory collection process. [Figure 7A] A table for saving location information and trajectory information. [Figure 7B] A differential trajectory information storage table that stores location information as differential values. [Figure 7C] A start / end point coordinate table that stores the position information at the beginning and end of the session. [Figure 8A] A still image data storage table for storing still images. [Figure 8B] A video data storage table for storing videos. [Figure 9A] A flowchart showing the overall avatar display process. [Figure 9B] A flowchart showing the relief process in FIG. 9A. [Figure 10A] A diagram showing an example of the direction change guide display. [Figure 10B] A diagram showing an example of the movement direction guide display. [Figure 11A] A diagram showing an example of the avatar display. [Figure 11B] A diagram showing an example of the avatar's changed orientation display. [Figure 12] A table showing examples of voice commands used for HMD operations. [Figure 13] A diagram explaining the operation instructions by the user's finger. [Figure 14A] A two-dimensional trajectory information storage table for storing the position information of Example 2. [Figure 14B] A two-dimensional start / end point position coordinate table at the start / end. [Figure 15] A schematic diagram of the avatar display in Example 3. [Figure 16] A schematic diagram of the avatar display in Example 4. [Figure 17A] A trajectory information storage table in Example 7. [Figure 17B] A diagram showing an example of the avatar display. [Figure 18] A diagram showing the appearance of the smartphone in Example 8. [Figure 19] A diagram showing the configuration of a video display system with multiple connected HMDs in Example 9. [Figure 20] A schematic diagram showing examples of the display of multiple avatars.

BEST MODE FOR CARRYING OUT THE INVENTION

[0010] The embodiments of the present invention will be described below, focusing primarily on the example of a head-mounted display (HMD), which is a head-worn image display device. [Examples]

[0011] Figure 1 shows an external view of the HMD according to Embodiment 1. The HMD1 is glasses-shaped, with a display unit 72 for the user to view images and an imaging unit 71 for capturing the surrounding scenery located on the front, and various processing units, which will be described later, housed in the temples (arms) 91. Alternatively, some of the processing units in the temples 91 may be housed in a separate housing and connected to the HMD body by a cable.

[0012] Figure 2 is a block diagram showing the internal configuration of the HMD1. The HMD1 consists of a main control unit 10, a system bus 30, a storage unit 40, a sensor unit 50, a communication processing unit 60, a video processing unit 70, an audio processing unit 80, and an operation input unit 90.

[0013] The main control unit 10 is a microprocessor unit that controls the entire HMD1 according to a predetermined operating program. The system bus 30 is a data communication path for sending and receiving various commands and data between the main control unit 10 and each component block within the HMD1.

[0014] The storage unit 40 stores various programs 41 for controlling the operation of the HMD1, various data 42 including operation setting values ​​and objects such as detected values ​​and content from the sensor unit described later, and has a work area 43 used for various program operations. The storage unit 40 can store operation programs downloaded from the network, various data created by the operation programs, and downloaded content such as videos, still images, and audio. It can also store data such as videos and still images captured by the imaging unit 71. The storage unit 40 needs to retain the stored information even when the HMD1 is not supplied with external power. Therefore, devices such as semiconductor memory such as flash ROM or SSD (Solid State Drive), or magnetic disk drives such as HDD (Hard Disc Drive) are used. The operation programs stored in the storage unit 40 can be updated and their functions expanded by download processing from various server devices on the network.

[0015] The sensor unit 50 consists of a GPS (Global Positioning System) sensor 51, a geomagnetic sensor 52, a distance sensor 53, an acceleration sensor 54, a gyroscope sensor 55, an altitude sensor 56, etc., to detect various states of the HMD1. These sensors detect the position, tilt, direction, movement, altitude, etc. of the HMD1. In addition to these, other sensors such as an illuminance sensor and a proximity sensor may also be included.

[0016] The communication processing unit 60 consists of a LAN (Local Area Network) communication unit 61 and a telephone network communication unit 62. The LAN communication unit 61 is connected to a network such as the Internet via an access point, and transmits and receives data with various server devices on the network. The connection to the access point may be made using a wireless connection such as Wi-Fi (registered trademark).

[0017] The telephone network communication unit 62 performs telephone communication (calls) and data transmission / reception via wireless communication with base stations, etc., of a mobile telephone communication network. Communication with base stations, etc., may be performed using W-CDMA (Wideband Code Division Multiple Access) (registered trademark), GSM (registered trademark) (Global System for Mobile communications), LTE (Long Term Evolution), or other communication methods. The LAN communication unit 61 and the telephone network communication unit 62 are each equipped with encoding circuits, decoding circuits, antennas, etc. Furthermore, the communication processing unit 60 may also be equipped with other communication units such as a Bluetooth (registered trademark) communication unit or an infrared communication unit.

[0018] The image processing unit 70 consists of an imaging unit 71 and a display unit 72. The imaging unit 71 is a camera unit that inputs image data of an external subject by converting light input from a lens into an electrical signal using an electronic device such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) sensor. The display unit 72 is a display device for a transmissive display using, for example, a laser projector and a half mirror, and provides the image data to the user of the HMD1.

[0019] The audio processing unit 80 consists of an audio input / output unit 81, an audio recognition unit 82, and an audio decoding unit 83. The audio input of the audio input / output unit 81 is a microphone, which converts the user's voice and other audio data into audio data for input. The audio output of the audio input / output unit 81 is a speaker, which outputs necessary audio information to the user. The audio recognition unit 82 analyzes the input audio information and extracts instruction commands, etc. The audio decoding unit 83 performs decoding processing of encoded audio signals and speech synthesis processing.

[0020] The operation input unit 90 is an instruction input unit that inputs operation instructions to the HMD1. The operation input unit 90 consists of operation keys arranged in a row of button switches, etc., but may also be equipped with other operation devices. For example, the HMD1 may be operated using a separate portable terminal device connected via wired or wireless communication using the communication processing unit 60. Alternatively, the HMD1 may be operated using voice commands for operation instructions using the voice recognition unit 82 of the voice processing unit 80.

[0021] Note that the HMD1 configuration example shown in Figure 2 includes many components that are not essential to this embodiment, but the effectiveness of this embodiment will not be impaired even if these components are not included. Conversely, additional components not shown, such as digital broadcasting reception functions and electronic money payment functions, may also be added.

[0022] Figure 3 shows an example of a connection configuration with external devices by the communication processing unit 60 described above. The LAN communication unit 61 of the HMD1 connects to a network 5 such as the Internet via a wireless router 4, which is an access point. A server 6 is connected to network 5 and transmits and receives data via the LAN communication unit 61 of the HMD1.

[0023] Figure 4 shows the configuration of the functional blocks of the HMD1. The overall control processing of the HMD1 is mainly performed by the main control unit 10 using various programs 41 and various data 42 in the storage unit 40. The processing in the HMD1 consists of a trajectory collection process (S100) that collects and stores information on the user's movement trajectory, and an avatar display process (S200) that displays an avatar, which is a virtual image representing the user, based on the stored trajectory information (shown by dashed and dashed lines, respectively).

[0024] In the trajectory acquisition process (S100), the various sensor information acquisition function 11 acquires information from various sensors in the sensor unit 50, and the trajectory information processing function 12 converts the acquired information from the various sensors into trajectory information that is easy to process internally. The converted trajectory information is saved by the trajectory information saving function 13. In addition, if the user gives a shooting instruction during the trajectory information acquisition process, shooting is performed using the imaging unit 71 of the video processing unit 70. In this case, the shooting process is performed by the shooting data acquisition function 14, and the acquired shooting data is associated with the trajectory information and saved by the shooting data saving function 15.

[0025] In the avatar display process (S200), the user's avatar information, which has been previously stored in the avatar information storage function 16, is read, and the avatar generated by the avatar generation function 17 is displayed by the avatar display function 18. The display position of the avatar is determined based on the trajectory information stored in the trajectory information storage function 13. However, the field of view calculation function 19 determines whether the avatar's display position is within the field of view of the HMD1, and if it is not, the avatar is not displayed. In addition, the shooting data playback function 20 plays back the shooting data stored in the shooting data storage function 15 according to the path trajectory.

[0026] As shown in Figure 3, the storage functions such as the trajectory information storage function 13, the shooting data storage function 15, and the avatar information storage function 16 can also be saved to an external server 6 via the LAN communication unit 61 of the communication processing unit 60.

[0027] Figure 5 is a schematic diagram illustrating an example of user trajectory collection. In this example, user 2, wearing HMD1, moves in the order of trajectory points 501, 502, 503, and 504 (indicated by ● marks). At trajectory point 503 (a corner) where the direction of movement changes by 90 degrees, the imaging unit 71 captures a photograph of the subject 510. Therefore, this trajectory point 503 becomes a shooting point. The objects to be photographed here are determined by the user, such as places visited, corners, places where the user took a wrong turn, or plants, animals, or buildings seen at those locations, but the captured images will be played back when the trajectory is reconstructed later.

[0028] Figure 6 is a flowchart showing the trajectory acquisition process (S100) of HMD1, using the trajectory schematic diagram in Figure 5 as an example. This processing procedure is stored in various programs 41 of the memory unit 40. S110: The trajectory collection process (S100) is started at the instruction of User 2. In this embodiment, the instruction to start trajectory collection is given by User 2's voice. For example, when the voice command "Kiseki Kaishi" (start trajectory collection) is spoken, the voice recognition unit 82 determines that this is an instruction to start the trajectory collection process.

[0029] S111: Initialize the point numbers of the track points. Here, the initial value of point number p is set to 0, so that point number p=0 at the start of the track collection process. S112: Start the timer. The timer uses a clock built into the main control unit 10 to measure time. The timer is used to collect data of the user's trajectory points at regular time intervals (unit time).

[0030] S113: The position of the HMD1 is detected using information from the sensor unit 50 (various sensor information acquisition function 11), converted into a storage data format (trajectory information processing function 12), and stored in the various data 42 of the storage unit 40 (trajectory information storage function 13). The position of the HMD1 on a plane is detected by the GPS sensor 51 of the sensor unit 50. By receiving radio waves from multiple GPS satellites, the global position coordinates (longitude and latitude) of the HMD1 can be detected. The altitude can also be detected by the height sensor 56 of the sensor unit 50. For example, the altitude can be calculated by measuring atmospheric pressure.

[0031] In addition to GPS information from GPS satellites, the HMD1 can also obtain global location information using Wi-Fi®, Bluetooth®, and information transmitted from cellular base stations. Therefore, if GPS information cannot be obtained, the HMD1's location information will be obtained using these other sources. Of course, other information can also be combined with GPS information. Hereinafter, the information used to obtain global location information will be referred to as "GPS information, etc."

[0032] Furthermore, if the above global location information can be acquired with high accuracy at all times, the user's trajectory information can be placed in a spatial coordinate system using the global location information. Alternatively, a unique spatial coordinate system (local coordinate system) can be generated with the trajectory acquisition start position as the origin, and the change in position can be calculated from the user's movement distance and direction detected by the acceleration sensor 54 and gyro sensor 55 of the sensor unit 50, and placed in the local coordinate system. Then, by acquiring global location information at a point where global location information can be acquired with high accuracy and associating it with the position information in the local coordinate system, it can be applied, for example, when the trajectory is not acquired continuously or when viewing the trajectory information of others.

[0033] S114: Determine whether the user has given a shooting instruction. If there is a shooting instruction, proceed to S115; otherwise, proceed to S116. The imaging unit 71 has two shooting modes: still image shooting and video shooting. To distinguish between these, for example, the voice command "SATSUEI" is used to instruct still image shooting, and the voice command "SATSUEI KAISHI" is used to instruct video shooting. In addition, for video, the voice command "SATSUEI OWRI" is used to instruct the end of shooting. S115: The imaging unit 71 of the video processing unit 5 captures a subject in front of the HMD1 (capture data acquisition function 14), and the captured data is saved as various data 42 in the storage unit 40 (capture data storage function 15).

[0034] S116: Determine whether the user has instructed the system to terminate the track information acquisition process. For example, the voice command "Kiseki Owari" (end of miracle) is used to determine if the system has instructed the system to terminate the track collection process. If termination is instructed, proceed to S117 to terminate the track information acquisition process. If termination of the track information acquisition process is not instructed, proceed to S118. S118: Determine whether the unit time has elapsed. Specifically, this is determined by whether the timer has exceeded the unit time. If the unit time has elapsed, proceed to S119; otherwise, return to S114. S119: Add 1 to point number p, return to S112, reset the timer, and restart.

[0035] The trajectory collection process (S100) described above allows the user's trajectory points (location information) to be collected at regular intervals. For example, by setting the timer unit time to 1 second, the HMD1's trajectory can be saved every second. In parallel with this, subjects along the trajectory can be photographed and saved according to the user's instructions.

[0036] Here, we will explain two methods for storing trajectory information data (data table) in the storage unit 40 by the trajectory collection process (S100) described above.

[0037] Figure 7A shows the trajectory information storage table 710, which stores location information as measured values. The table consists of a point number 711 indicating the order of each trajectory point and a position coordinate 712 corresponding to that trajectory point. The position coordinate 712 uses planar position information (X,Y) from GPS information, etc., and the altitude (Y) value from the altitude sensor 56. Point number 0 is the position coordinate (X0,Y0,Z0) at the time the trajectory collection process starts (S110), and point number 1 is the position coordinate (X1,Y1,Z1) after a unit of time (here, 1 second). Point number k is the position coordinate (Xk,Yk,Zk) at the time when the user gives a shooting instruction (S114) after k seconds. Point number n is the position coordinate (Xn,Yn,Zn) at the time the trajectory collection process ends (S117) n seconds after the start.

[0038] In the data table, the position coordinates 712 are fixed-length data, so if recorded chronologically, the data for point number 711 can be omitted. In other words, the position coordinate data for the target point number p can be searched using a multiple of the fixed-length value p.

[0039] If location information cannot be obtained from GPS or other sources, relative value information (difference value information) from sensors such as the accelerometer 54 and gyroscope 55 of the sensor unit 50 is temporarily stored. Then, when location information (absolute value information) can be obtained from GPS or other sources at any point along the trajectory, the temporarily stored relative value is corrected to an absolute value.

[0040] Figure 7B shows the differential trajectory information storage table 720, which stores location information as differential values. When the change in measured values ​​between trajectory points is small, it is more practical to store the differential values ​​(amount of change) rather than the measured values ​​themselves, as this reduces the amount of data to be stored.

[0041] Each item consists of point number 721 and the corresponding difference position coordinate 722. The difference position coordinate 722 describes the difference between the position coordinate of the previous trajectory point and the position coordinate of the current trajectory point. That is, the difference value of the planar position coordinate (ΔX, ΔY) is described by the longitude distance and latitude distance between the two trajectory points. The difference value in the height direction (ΔZ) is also described by distance.

[0042] Point number 1 represents the differential position coordinates (ΔX1, ΔY1, ΔZ1) 1 second after the start of the track collection process, point number k represents the differential position coordinates (ΔXk, ΔYk, ΔZk) k seconds later, and point number n represents the differential position coordinates (ΔXn, ΔYn, ΔZn) at the end of the track collection process (n seconds after the start). The relationship with the value of position coordinate 712 in Figure 7A (track information storage table 710) is as follows: ΔXp=Xp-Xp-1, ΔYp=Yp-Yp-1, ΔZp=Zp-Zp-1 This is the result.

[0043] Figure 7C shows the start and end point coordinate table 730, which stores the position information at the start and end of trajectory collection. This table is necessary when calculating the avatar's display position (absolute position) using the difference trajectory information storage table 720 in Figure 7B.

[0044] Each item consists of a start / end point distinction 731, which distinguishes whether it is the start or end point of track collection, and a position coordinate 732, which shows the position information of each point in absolute value. Naturally, the start point is equal to the position coordinate (X0, Y0, Z0) of point number 0 in Figure 7A (track information storage table 710), and the end point is equal to the position coordinate (Xn, Yn, Zn) of point number n.

[0045] In this way, by saving the position information (absolute values) of the start and end points 732, the difference trajectory information storage table 720 in Figure 7B can be converted to the trajectory information storage table 710 in Figure 7A. In particular, saving the position coordinates (Xn, Yn, Zn) of the end point allows for more efficient display of the avatar. The reason for this will be explained below.

[0046] As described later, the avatar is displayed by reversing time from the endpoint of the trajectory (point number n). In this case, if the position coordinates of the endpoint where the display begins are unknown, it is necessary to add all the difference position coordinates 722 in Figure 7B to the starting point position coordinates (X0, Y0, Z0) in Figure 7C in order to calculate it. If the position coordinates of the endpoint (Xn, Yn, Zn) are known, the avatar is first displayed at that position coordinate. The position where the avatar will be displayed next can be easily calculated by subtracting the difference position coordinates (ΔXn, ΔYn, ΔZn) at point number n in Figure 7C from the current position coordinate. This process can then be repeated.

[0047] These trajectory information (trajectory information storage table 710, differential trajectory information storage table 720, start / end point position coordinate table 730) can be stored not only in the various data 42 of the storage unit 40, but also, as explained in Figure 3, on an external server 6 via the network 5, either for each trajectory point or when the trajectory information table is completed. By storing the data on the server 6, the amount of data stored on the HMD1 can be reduced.

[0048] Next, I will explain how to save the captured data. The system supports both still images and videos. Figure 8A shows the still image data storage table 810 for saving still images. The still image data storage table 810 consists of a point number 811 indicating the trajectory point at the time of the shooting instruction, a shooting direction 812 indicating the shooting direction, a shooting data size 813, and a shooting data 814. In this example, at the point of trajectory point k (point number k), the HMD1 takes a picture in the direction Θk, and the shooting data Dk with a shooting data size Mk is saved in the various data 42 of the storage unit 40 (S115).

[0049] By linking and saving the aforementioned trajectory information storage table 710 with the still image data storage table 810 in this example, it is possible to play back still images captured in synchronization with the movement position of the HMD1 at each unit of time (in this case, 1 second).

[0050] Figure 8B shows the video data storage table 820 for saving videos. Similar to the case of still images in Figure 8A, the video data storage table 820 consists of a point number 821 indicating the trajectory point at the time of the shooting instruction, the shooting direction at the start of shooting 822, the shooting data size 823, and the shooting data 824. In addition, for videos, the shooting time (start and end) 825 is also recorded. This is to accommodate cases where video shooting spans multiple trajectory points. Playing the video at unit time intervals (every second) allows it to be synchronized with the time displayed on the avatar.

[0051] In this example, at the trajectory point k (point number k), HMD1 starts capturing images in the direction Θk from start time Tks until end time Tke. The captured data Dk, with a data size Mk, is saved to the various data 42 in the memory unit 40 (S115). The video data can be played back on the entire HMD display screen, but in this example, it is displayed in a reduced window and played back from the video recording start time Tks to the video recording end time Tke.

[0052] As explained in Figure 3, the amount of data that needs to be stored in the HMD1 can be reduced by saving this shooting data information (still image data storage table 810, video data storage table 820) to an external server 6 via the network 5.

[0053] Next, the avatar display process (S200) of the HMD1 will be explained. This processing procedure is stored in various programs 41 of the memory unit 40. To display the avatar, avatar information is extracted from the avatar information storage function 16 (avatar generation function 17), and the avatar is displayed on the display unit 72 of the HMD1 (avatar display function 18). The shape of the avatar is life-size to the user, and the user sets the height of the avatar. For convenience, the height can also be set to the distance from the ground (height of the HMD1 from the ground) plus 10 cm using the distance sensor 53 of the sensor unit 50 while standing. The display position of the avatar is displayed superimposed on the background image on the display unit 72 of the HMD according to the position information stored in the trajectory information storage table 710, etc.

[0054] If the trajectory information and shooting data to be used are stored in the various data 42 within the HMD, that trajectory information and shooting data will be used. However, if they are stored on an external server 6 via the LAN communication unit 61 of the communication processing unit 60, they will be retrieved from the server 6 and used.

[0055] In this embodiment, the avatar is displayed by working backward in time from the most recent trajectory information to the oldest trajectory information, based on the user's trajectory points. By displaying the avatar in this reverse chronological order, the user can return to the starting point (departure point) along the same path they actually traveled.

[0056] Figure 9A is a flowchart showing the overall avatar display process (S200). S210: Start the avatar display process (S200). In this example, the user utters the voice command "Avatar Start" which is interpreted as an instruction to start displaying the avatar.

[0057] S211: The current position and orientation of the HMD1 worn by the user are detected. The detection method is the same as in S113 in the trajectory collection process (S100), using the GPS sensor 51, geomagnetic sensor 52, height sensor 56, etc. of the sensor unit 50. S212: Refer to track information storage table 710 (or differential track information storage table 720) and search for the point number s with the position coordinates closest to the current position of HMD1. That is, if the current position is the endpoint of track collection, s=n, but if the user has moved from the endpoint since then, s≠n may occur. S213: The initial value of the point number m to be displayed is set to the point number s closest to the current location. This allows the display to start from the trajectory point closest to the user's current location.

[0058] S214: Start the timer. The timer uses the clock built into the main control unit 10. The timer is used because the user's trajectory is collected at regular time intervals (unit time), and the display is synchronized with this. However, the unit time used for avatar display can be different from the unit time of the trajectory collection process, and can be set to a time multiplied by an arbitrary coefficient. For example, if the unit time used for avatar display is set to 1 / 2 the time (0.5 seconds in this example), the displayed avatar can be moved at twice the speed. Conversely, if the unit time used for avatar display is set to twice the time (2 seconds in this example), the displayed avatar can be moved at half speed (slow). In addition, the avatar can be displayed as a video (animation) by sequentially interpolating the avatar between trajectory points (for example, every 1 / 24th of a second).

[0059] S215: Calculate the position and direction to display the avatar. If trajectory information is stored in the trajectory information storage table 710, read the position coordinate values ​​(Xm, Ym, Zm) of 712 corresponding to point number m and place the avatar at the planar position (Xm, Ym) and altitude (Zm). If the track information is stored in the differential track information storage table 720, read the value of the differential position coordinate 722 at point number (m+1) (difference value) and subtract it from the previous displayed position (point number = m+1). That is, Xm=Xm+1-ΔXm+1,Ym=Ym+1-ΔYm+1,Zm=Zm+1-ΔZm+1 This is the result. However, for the first time (m=s), the calculation is performed using the position coordinates of the endpoint (Xn, Yn, Zn) described in the start / end point position coordinate table 730, and the difference position coordinates 722 up to point number s. For convenience, the direction the avatar is facing is defined as the direction connecting the previous trajectory point (m+1) and the current trajectory point (m).

[0060] S216: Detect the current position and orientation of HMD1. This is the same process as S211, and the initial position (m=s) can be omitted as it was already detected in S211. S217: The field of view calculation function 19 determines the display area of ​​the HMD1, i.e., the user's field of view, from the current position and orientation of the HMD1, and determines whether the avatar placement position calculated in S215 is within the display area. If it is determined that the avatar can be placed within the display area of ​​the HMD1, the process proceeds to S218; if it is determined that the avatar cannot be placed, the rescue process in S300 is performed. The rescue process in S300 involves guidance processes such as having the user change their field of view direction or moving the user to a position where the avatar is visible, and details will be described later in Figure 9B. Through this determination process in S217 and the rescue process in S300, the avatar can be placed within the user's field of view, and the user can move (track) the avatar without losing sight of it or overtaking it.

[0061] S218: If it is determined that the avatar can be displayed within the display area of ​​the HMD1, the avatar information is read from the avatar information storage function 16 and the avatar is generated. Then, the avatar is displayed on the display unit 72 of the HMD1 so that it matches the display position and orientation of the avatar calculated in S215.

[0062] S219: Refer to the still image data storage table 810 or the video data storage table 820 to determine whether the currently displayed point number m is a shooting point k. If it is determined that the track point is a shooting point, proceed to S220. If it is determined that it is not a shooting point, proceed to S222. S220: Notifies the user that the current location is a shooting point. The method of notifying the user of a shooting point is to change the color of the displayed avatar or to make the avatar blink (flash). S221: Upon user instruction, the system reads the corresponding captured data from the still image data storage table 810 or the video data storage table 820 and plays it back. Playback of the captured data is performed by the voice command "Playback". Playback of the captured data can be performed on the entire HMD display screen, but in this example, it is displayed in a reduced window and the playback of the captured data is terminated after a certain period of time.

[0063] S222: Determine whether the user has instructed the avatar display process to end. For example, the voice command "Avatar End" is used to determine if the avatar display process has been terminated. If the user has instructed the avatar display process to end, proceed to S226 and terminate the avatar display process. S223: Determine whether the unit time has elapsed. If it is determined that the unit time has elapsed, proceed to S224. If it is determined that the unit time has not elapsed, return to S222. S224: Subtract 1 from point number m to obtain a new point number. This returns to the previous trajectory point in the timeline.

[0064] S225: Determine whether the value of the new point number m has become less than 0 (i.e., whether it has passed the starting point m=0 of the trajectory). If the value of point number m does not become less than 0, return to S214 and continue displaying the avatar for the next trajectory point. If the value of point number m becomes less than 0, the avatar display has been completed for all trajectory points, so proceed to S226 and end the avatar display process.

[0065] Figure 9B is a flowchart of the rescue process (S300) in Figure 9A. This process is performed when the determination in S217 is No, that is, when the avatar cannot be placed within the display area. After the process is completed, the process returns to S211.

[0066] S301: The field of view calculation function 20 determines whether the avatar can be positioned by changing the direction of HMD1. If possible, proceed to S302; otherwise, proceed to S303. S302: This applies when the HMD is not facing the direction of the avatar, so the user is guided by voice or display to change the direction of the HMD. For example, the voice decoding unit 83 of the voice processing unit 80 generates voice guidance such as "right" or "left". At this time, the avatar may be displayed on the screen and spoken. After that, return to S211.

[0067] Here, Figure 10A shows an example of the direction change guide display in S302. Avatar 3 appears on the display screen 400, and message 401 is displayed indicating that the avatar is on the right side. In response, when the user turns to the right, the HMD's new direction is detected in S216 in Figure 9A. The determination in S217 indicates that the avatar is within the new display area, and the avatar can be displayed in S218.

[0068] S303: If the avatar cannot be positioned by changing the direction of HMD1 in S301, it is likely that the current location is far from any of the points from which the trajectory was collected. Therefore, to guide the user to a nearby point, it is determined whether map information can be obtained. For example, if a navigation app is available, proceed to S304. If map information cannot be obtained, proceed to S306.

[0069] S304: Using the navigation app, perform route analysis to the nearest track point s found in S212. S305: Route information (new track point) up to track point s is added to track information storage table 710. Then, the process returns to S211, and the added track point is included in the avatar's display area. As a result, in S217, the avatar is positioned within the HMD's display area, and in S218, the avatar can be displayed.

[0070] If map information cannot be obtained in S306:S303, the user is shown the direction to move to the nearest trajectory point s. This is done via voice or AR display. Then, the process returns to S211. When the user moves in the indicated direction, they can approach the nearest trajectory point s. As a result, the avatar is placed within the display area in S217, and the avatar can be displayed in S218.

[0071] Figure 10B shows an example of the movement direction guide display in S306. The display screen 400 informs the user of the direction they should move in using the AR display 402. The AR display 402 only needs to show the direction to move, and uses text, icons (arrows), etc.

[0072] As described above, if it is determined that the avatar cannot be placed within the HMD display area, the recovery process of S300 can be performed to guide the user to a position and direction where the avatar is visible. This allows the user to move (track) the avatar without losing sight of it.

[0073] Figures 11A and 11B show examples of avatar display. These examples correspond to the schematic diagram of trajectory collection shown in Figure 5. In the display screen 400 of Figure 11A, Avatar 3 is displayed by reversing time each trajectory point of the HMD that has collected the trajectory. That is, the display position of Avatar 3 is moved in the direction of the arrow in the order of trajectory points 504, 503, 502, and 501 (indicated by ●). Among these, trajectory point 503 is the shooting point k, so it is indicated with ◎ to distinguish it from the other positions. In addition, in order to reproduce the direction that User 2 (HMD) was facing during the trajectory collection process, the body orientation when moving Avatar 3 is set to face the opposite direction of Avatar 3's movement (backward), showing the avatar returning along the path it came from.

[0074] The display screen 400 in Figure 11B shows the display state of Avatar 3 at trajectory point 503. Since trajectory point 503 is a corner, the orientation of Avatar 3 has been changed. Also, since trajectory point 503 is a shooting point, the color of Avatar 3 has been changed for display. When the user instructs playback of the shooting data at this point, the image 510' of the subject 510 that was photographed at the shooting point is displayed.

[0075] In this example, to inform the user of the return route, trajectory points (● marks) and the direction of travel (arrows) are displayed on the display screen 400, but these can also be hidden. Furthermore, while the avatar's trajectory points are displayed at unit time (1 second), they can be displayed at other time intervals as well. For example, if the points are interpolated and displayed every 1 / 24th of a second, the avatar will appear as a smooth, movie-like video.

[0076] Next, we will explain how the user interacts with the HMD1. As mentioned above, in this embodiment, voice commands are used as the user's controls. Figure 12 is a table showing examples of voice commands used to operate the HMD. The voice command table 600 consists of items: command classification 601, voice command 602 spoken by the user, command name 603, and processing content 604.

[0077] Voice command classification 601 is divided into track collection-related and avatar display-related. Track collection-related voice commands 602 include "Kiseki Kaishi" and "Kiseki Start" to start the track collection process, "Kiseki Owari" and "Kiseki End" to end the track collection process, "Satsuei" and "Kiroku" to take a still image, "Satsuei Kaishi" and "Kiroku Start" to start recording a video, and "Satsuei Owari" and "Kiroku End" to end recording a video.

[0078] Voice commands 602 related to avatar display include "Avatar Start" and "Avatar Begin" to start the avatar display process, "Avatar End" and "Avatar End" to end the avatar display process, and "Play" and "Screenplay Data" to play a still image or video. It goes without saying that the voice commands listed here are just examples, and can be configured as needed according to the user's preferences.

[0079] On the other hand, the HMD1 can use the speech synthesis function of the speech decoding unit 83 of the speech processing unit 80 to provide voice responses corresponding to voice commands uttered by the user. As a result, it becomes possible to respond, for example, with the same voice as the user's voice command (parroting), or to provide voice responses confirming the user's voice command. For example, in response to the inquiry "Is it XX?" from the HMD, if the user replies "Yes", the HMD will respond with the confirmation voice "I will perform XX", and if the user replies "No", it will respond with the confirmation voice "I will cancel XX". It goes without saying that the voice commands "Yes" and "No" used by the user in their replies are also included in the voice command table 600, although they are not shown in Figure 12. Furthermore, the speech synthesis function of the speech decoding unit 83 can be used to generate synthesized speech such as "right" to encourage rotation to the right and "left" to encourage rotation to the left. This can be used in step S306 in Figure 9B.

[0080] Operation instructions for the HMD1 can be provided by methods other than voice commands. For example, an operation menu can be displayed on the HMD1's screen, and the user can select options using gestures (hand and finger movements). In this case, the imaging unit 71 of the video processing unit 70 captures the user's fingers, and the finger movements can be recognized from the captured video and used for operation.

[0081] Figure 13 illustrates the user's finger gestures for operation. The HMD1's operation screen 410 displays selection menus, such as the "Photo" menu 411 and the "Video" menu 412. While it is preferable for the menu display to be semi-transparent, it is not limited to this.

[0082] When a user brings their finger 415 closer to the display screen 410 from outside the HMD1 to select, for example, the "Video" menu 412, the imaging unit 71 detects the finger image 415, which is gradually enlarging towards the display screen 410. It also recognizes that the finger image 415 is moving towards the "Video" menu 412 and determines that "Video" has been selected. For other operations, when starting trajectory collection or avatar display, only the "Start" menu is displayed, and when ending, only the "End" menu is displayed. The imaging unit 71 then detects the approach of the user's finger image 415 and determines that a start or end has been instructed. Alternatively, a touch sensor could be provided on the HMD1's input unit 90 to detect when the user's hand or fingers touch the HMD1.

[0083] According to Example 1, it is possible to provide clear directions back to a user who has forgotten a place they stopped at along the way or their return route. [Examples]

[0084] In Example 1, the HMD's position coordinates were saved in 3D (X, Y, Z), but in Example 2, we will describe the case where the position coordinates are saved in 2D (X, Y). The basic configuration of the HMD in Example 2 is the same as in Example 1, and the differences from Example 1 will be explained.

[0085] Under normal movement conditions on flat ground, there is little change in height, so even if height data is omitted, the avatar can be displayed based on the planar position coordinates. The trajectory collection process is performed using the same flowchart (S100) as in Example 1 (Figure 6). The collected data is then saved in the tables shown in Figures 14A and 14B.

[0086] Figure 14A shows the 2D trajectory information storage table 740, where position information is stored as difference values. The table consists of a point number 741 indicating the number of each trajectory point and a difference position coordinate 742 indicating the planar position of that trajectory point. As shown in Figure 7B, the difference position coordinate 742 is not the absolute value of the position coordinate, but rather the difference value (amount of change) between adjacent trajectory points, thereby reducing the amount of data to be stored.

[0087] In this example, the unit time is set to 1 second, so point number 1 represents the differential position coordinates (ΔX1, ΔY1) 1 second after the start of the trajectory collection process, point number k represents the differential position coordinates (ΔXk, ΔYk) k seconds later, and point number n represents the differential position coordinates (ΔXn, ΔYn) at the end of the trajectory collection process (n seconds after the start). The method for determining the differential position coordinates is the same as in Example 1 (Figure 7B).

[0088] Figure 14B shows the 2D start / end point coordinate table 750, which stores the position information at the start and end of trajectory collection. This table is necessary when calculating the avatar's display position (absolute position) using the 2D differential trajectory information storage table 740 in Figure 14A.

[0089] The items consist of a start / end point distinction 751 that distinguishes between the start and end points of track collection, and position coordinates 752 that show the position information of each point in absolute value, similar to Figure 7C above. However, in position coordinates 752, only the position coordinates (X0, Y0, Z0) of the start point store the position coordinates (Z0) in the height direction. This is to handle cases where the amount of altitude change exceeds a predetermined value (threshold) during track information acquisition.

[0090] While Example 2 only deals with two-dimensional position coordinates, the case where altitude changes is handled as follows. When going up (down) a slope, the change in the HMD's planar position coordinates (ΔX, ΔY) is linear, and the change in the HMD's altitude (ΔZ) is also linear. When going up (down) stairs, the change in the HMD's planar position coordinates (ΔX, ΔY) is equal to the width of the stairs, and the change in the HMD's altitude (ΔZ) is equal to the difference in the number of steps.

[0091] In the case of an escalator, the change in the planar position coordinates of the HMD (ΔX, ΔY) is linear, and the change in the altitude of the HMD (ΔZ) is also linear. It can be distinguished from a slope by the fact that the change in acceleration is smaller on an escalator. In the case of an elevator, the planar position coordinates of the HMD hardly change (ΔX=0, ΔY=0), and only the altitude of the HMD changes (ΔZ≠0).

[0092] As described above, when moving on slopes, stairs, escalators, elevators, etc., characteristic changes (ΔX, ΔY) and (ΔZ) occur in the detected planar position coordinates and altitude values, respectively. When displaying the avatar, these changes are captured and route information (slope, stairs, escalator, elevator, etc.) is displayed. Furthermore, by overlaying this information with images of slopes, stairs, escalators, elevators, etc., a highly immersive avatar can be displayed. In addition, significant effects can be obtained by linking it with map information such as underground shopping area maps.

[0093] If the altitude change exceeds a predetermined value (threshold), the trajectory collection process using the two-dimensional trajectory information storage tables 740 and 750 is terminated as trajectory A. Then, the process moves to the three-dimensional trajectory information storage tables 710 to 730 of Example 1, and trajectory B is created. Subsequently, when the altitude change falls within the threshold, the trajectory collection process using the two-dimensional trajectory information storage tables 740 and 750 is resumed, and trajectory C is created. This method minimizes the amount of data that needs to be stored. Of course, it is also possible to convert from the 2D trajectory information storage tables 740 and 750 of Example 2 to the 3D trajectory information storage tables 710 to 730 of Example 1.

[0094] In the above-described embodiments 1 and 2, we assumed an outdoor environment where it is relatively easy to acquire GPS information (location information) using the GPS sensor 51 of the sensor unit 50. However, there are cases where it is difficult to acquire GPS information, such as in underground shopping malls or indoors. If GPS information cannot be acquired, it is supplemented by information from the acceleration sensor 54, gyro sensor 55, etc., of the sensor unit 50.

[0095] If no position information can be obtained from GPS or other sources from the starting point to the ending point, the starting point's position coordinates will be stored as X0=0,Y0=0. In this case, the height position coordinates will also be stored as Z0=0. Furthermore, the ending point's position coordinates will be calculated by summing all the difference position coordinates from the starting point to the ending point. [Examples]

[0096] In Example 1, when displaying the avatar, the avatar's orientation was reversed from the direction of travel, causing the avatar to move backward. In contrast, in Example 3, the avatar is displayed facing the direction of travel. Figure 15 is a schematic diagram of the avatar display in Example 3. On the display screen 400, avatar 3 is displayed facing the direction of travel (forward). That is, the orientation of avatar 3 is set in the vector direction from the current trajectory point 503 to the next trajectory point 502. This eliminates the user discomfort caused by the avatar's orientation as shown in Example 1 (Figure 11B) (the avatar's direction of travel and the avatar's orientation do not match).

[0097] Furthermore, whether the avatar faces forward (Example 3) or backward (Example 1) depends on the user's preference. Therefore, a configuration that allows the user to select and set which display mode the avatar is facing is more preferable. [Examples]

[0098] In Example 4, if a building or other object is present in the HMD's field of view and the avatar is positioned behind it, the avatar is hidden. Figure 16 is a schematic diagram of the avatar display in Embodiment 4. Here, avatar 3 is moving backward. Buildings 521 and 522 are located near the movement path, and the diagram shows the case where avatar 3 is hidden by building 521. In such cases, avatar 3 is controlled so that the part hidden by building 521 is not displayed. Whether or not avatar 3 is hidden can be determined by comparing the distance to buildings 521 and 522 and avatar 3, and the distance to buildings 521 and 522 can be measured by distance sensor 53.

[0099] This allows for a realistic and seamless display when overlaying an avatar onto the real world visible through the HMD. However, if the entire avatar is hidden and completely invisible, there is a risk that it will become impossible to track it. Therefore, a configuration that displays the hidden portion of the avatar semi-transparently to allow tracking is also acceptable.

[0100] Images of buildings 521 and 522 can be displayed using services such as Street View (Google), an internet service that provides panoramic photos of roadside scenery. Furthermore, depending on the position and orientation of the HMD1, the names of buildings and other structures associated with them can also be used. [Examples]

[0101] In Example 5, the system is configured to automatically take photos rather than requiring user input. The HMD's imaging unit 71 starts capturing images from the starting point and continuously loops and temporarily saves video for a certain period of time from the current point (for example, twice the unit time between trajectory points used in the trajectory collection process). As a result, the automatically captured video is either a video or a still image from a certain period of time (for example, 2 seconds) prior to the current point. In the case of still images, it is sufficient to temporarily save one or two still images between trajectory points to achieve the desired functionality. Furthermore, video captured at specific points along the movement path, such as when the direction of movement changes, is saved.

[0102] The specific operation of automatic shooting will be explained using the schematic diagram of trajectory collection shown in Figure 5. In Figure 5, at trajectory point 504, the video from trajectory point 502 to trajectory point 504 (for a length of twice the unit time) is temporarily stored. However, at trajectory point 503, the direction of movement (movement vector between points) changes by 90 degrees. Therefore, the video captured from trajectory point 502 to trajectory point 503 is automatically saved as the video just before the direction of the movement vector changes significantly. The video data of trajectory point 503 to be saved can be either a video (video frames captured up to trajectory point 503) or a still image (capture data of trajectory point 503).

[0103] According to the shooting method of Example 5, when the user changes direction, the video of the direction of travel immediately before the change is saved, so that the video best suited to recalling the user's memory can be displayed. [Examples]

[0104] In Example 6, information such as the user's posture was also collected during the trajectory collection process and reflected when displaying the avatar. For example, the distance sensor 53 of the sensor unit 50 detects the distance from the ground or floor surface to the HMD1 (i.e., the user's head height) and collects this data at each trajectory point. By comparing this detected value with the user's height (known), the posture of the user wearing the HMD1 (standing, sitting, crouching, lying down, etc.) can be estimated. This posture information is reflected in the avatar display to show standing avatars, sitting avatars, etc.

[0105] Furthermore, the geomagnetic sensor 52 of the sensor unit 50 can determine the orientation of the user's face while wearing the HMD1. This face orientation information is reflected in the displayed avatar, and the avatar's face orientation is changed at each trajectory point. The orientation of the avatar's body may be estimated and changed based on the face orientation, or it may be adjusted to match the vector information (direction of travel) between trajectory points. These displays make it possible to display a more realistic avatar. [Examples]

[0106] In Example 7, the time at which each trajectory point is passed is stored during the trajectory collection process, and the time of passage is displayed on the display screen when the avatar is displayed. Figure 17A shows an example of the trajectory information storage table 710' in Example 7. Compared to the trajectory information storage table 710 in Example 1 (Figure 7A), the time of passage 713 for each trajectory point has been added and recorded.

[0107] Figure 17B shows an example of avatar display. The display screen 400 shows avatar 3 along with the time of passage 420. This time can be obtained by referring to the time of passage 713 in the track information storage table 710' in Figure 17A, which corresponds to the point number 711 where the avatar is currently displayed.

[0108] Thus, according to Example 7, the time when the user passed through the currently displayed trajectory point is shown, which is useful information for recalling the user's memory. [Examples]

[0109] Example 8 describes the use of a smartphone instead of an HMD as a portable video display device. Smartphones have almost the same hardware and software configuration as HMDs and can achieve equivalent functionality.

[0110] Figure 18 shows the external appearance of the smartphone 9, where (a) is the front side that displays the image and (b) is the back side that has the camera. As shown in (b), the rear side has a camera 901 that captures the surrounding scenery. Inside the smartphone 9, there is a sensor that detects the 3D position and the direction of the camera 901, and it can collect trajectory information in the same way as an HMD.

[0111] As shown in (a), the front side has a display screen 902 with a built-in touch panel, which displays images captured by pointing the camera 901 at the outside scenery. The display screen 902 can then superimpose the avatar 3 onto the displayed image of the outside scenery, based on the collected trajectory information, similar to Embodiment 1 (Figure 11A).

[0112] Compared to an HMD, using a smartphone 9 presents challenges in trajectory information collection. The camera 901 must be pointed at the subject each time, making the process cumbersome as the number of shooting points increases. Furthermore, automatic shooting presents drawbacks, such as the impracticality of users maintaining a consistent shooting direction for the camera 901. However, the ability to collect trajectory information with existing smartphones improves convenience. That said, it's difficult for users to shoot while moving, so it's more practical to focus on key points individually.

[0113] Here, a smartphone was used as an example of a portable video display device, but the operation of this embodiment can be realized with equivalent or similar hardware and software configurations. For example, it can be applied to notebook PCs, tablet PCs, etc. [Examples]

[0114] Example 9 describes a video display system used collaboratively by multiple users wearing HMDs. Figure 19 shows the configuration of a video display system with multiple HMDs connected. Here, we show the case of two users 2a and 2b, each wearing HMD1a and HMD1b, respectively. The same applies to cases with more than two users.

[0115] The HMD1a worn by user 2a is connected to an external server 6 via wireless router 4a and network 5. Similarly, the HMD1b worn by user 2b is connected to the same server 6 via wireless router 4b and network 5. When HMD1a (user 2a) and HMD1b (user 2b) are in close proximity, wireless routers 4a and 4b can be shared.

[0116] In this case, each HMD1a and 1b collects the trajectory data of their respective users, 2a and 2b. The trajectory information of user 2a wearing HMD1a and the trajectory information of user 2b wearing HMD1b are then stored on a common server 6. Subsequently, the trajectory information of both users is read from the server 6 and displayed on each HMD1a and 1b as two avatars, representing the two virtual images of the two users. This allows the avatars to mutually reference each other's trajectories.

[0117] Figure 20 is a schematic diagram showing an example of displaying multiple avatars. For example, the display screen 400a of HMD1a shows the avatar 3a of user 2a wearing HMD1a, as well as the avatar 3b of user 2b wearing HMD1b. The trajectories 520a and 520b of the two users are displayed at their respective trajectory points (● and ▲ marks), allowing the relative positions of the two users to be displayed in reverse time. In this example, it can be seen that the two users met at the intersection of trajectory point 521. The same display is shown on the display screen 400b of HMD1b.

[0118] As described in Example 9, multiple users can mutually confirm each other's trajectories. As a side effect of this, if another user moves out of sight while trajectory collection is in progress, their location can be immediately determined.

[0119] As a practical form of this embodiment, it is also possible to use short-range wireless communication such as Bluetooth® to communicate between portable information terminals, thereby eliminating the need for a server 6 via a wireless router 4 or network 5. For example, if a child goes missing while traveling with their parents, the parent can immediately locate them using their HMD or smartphone.

[0120] Although examples of embodiments of the present invention have been described above using Examples 1 to 9, it goes without saying that the configurations for realizing the technology of the present invention are not limited to the above examples, and various modifications are conceivable. For example, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. All of these fall within the scope of the present invention. Furthermore, the numbers and messages that appear in the text and figures are merely examples, and using different ones will not impair the effects of the present invention.

[0121] The functions of the present invention described above may be implemented in hardware, in whole or in part, for example, by designing them using an integrated circuit. Alternatively, they may be implemented in software by having a microprocessor unit or the like interpret and execute a program that implements each of these functions. Hardware and software may be used in combination. The software may be pre-stored in the various programs 41 of the HMD1 at the time of product shipment. It may also be obtained from various server devices on the internet after product shipment. Alternatively, the software may be obtained by providing it on a memory card, optical disc, or the like.

[0122] Furthermore, the control lines and information lines shown in the diagram are those deemed necessary for explanation and do not necessarily represent all control lines and information lines on the product. In reality, it is safe to assume that almost all components are interconnected. [Explanation of symbols]

[0123] 1...HMD (Hand-Mounted Display), 2...User, 3...Avatar, 4...Wireless Router, 5...Network, 6...Server, 9...Smartphone (Hand-Mounted Display), 10...Main Control Unit, 40...Storage Unit, 50...Sensor Unit, 51...GPS Sensor, 52...Geomagnetic Sensor, 53...Distance Sensor, 56...Altitude Sensor, 60...Communication Processing Unit, 70...Video Processing Unit, 71...Imaging Unit, 72...Display Unit, 80...Audio Processing Unit, 90...Operation Input Unit.

Claims

1. In a video display device carried by a user that displays the user's movement trajectory as video, A sensor that detects the position and orientation of the user carrying the video display device, A trajectory information acquisition unit acquires information on the user's movement trajectory from the detection results of the aforementioned sensor, A storage unit that stores the user's trajectory information acquired by the trajectory information acquisition unit and information about an avatar, which is a virtual image representing the user. A display unit that displays the user's movement trajectory using the avatar, The system comprises a trajectory information acquisition unit and a control unit that controls the display unit, The control unit, The system generates the avatar from the avatar information stored in the memory unit, and uses the sensor to determine the current user's field of view. Based on the user's trajectory information stored in the memory unit, the generated avatar is positioned on the user's movement trajectory to match the user's current field of view and displayed on the display unit. A video display device characterized in that, when moving the avatar along the user's movement trajectory, the orientation of the avatar's body when it is moving is displayed in the opposite direction to the avatar's direction of movement, and the avatar is moved from newer trajectory information to older trajectory information in time.

2. In the video display device according to claim 1, The trajectory information acquisition unit acquires information on the user's movement trajectory at predetermined unit time intervals and stores the position coordinates of each trajectory point in the storage unit. The control unit reads the position coordinates of each trajectory point from the storage unit at intervals of the unit time, or at intervals obtained by multiplying the unit time by an arbitrary coefficient, and places the avatar at those position coordinates.

3. In the video display device according to claim 2, The trajectory information acquisition unit stores the position coordinates of each trajectory point in the storage unit as the difference between the position coordinates of the trajectory point immediately preceding it in time, and stores the position coordinates of at least the endpoint trajectory point in the storage unit as absolute values.

4. In the video display device according to claim 1, The image display device is characterized in that, when the control unit cannot place the avatar within the user's current field of view, it performs guidance processing to cause the user to change their field of view or move to a position where the avatar is visible.

5. In the video display device according to claim 2, Furthermore, it is equipped with an imaging unit that captures the exterior scenery. The imaging data captured by the imaging unit is stored in the storage unit in association with the shooting point, which is the shooting location. The control unit is characterized in that when the position of the avatar is at the shooting point, it displays the image of the shooting data stored in the storage unit on the display unit.

6. In the video display device according to claim 5, The control unit stores in the storage unit the video footage captured at the point where the direction of the user's movement trajectory changed, from the captured data captured by the imaging unit. The control unit is characterized in that when the position of the avatar is at a shooting point where the direction of the user's movement trajectory changes, it displays the image of the shooting data stored in the storage unit on the display unit.

7. In the video display device according to claim 2, Furthermore, it is equipped with an imaging unit that captures the exterior scenery. The display unit displays the current video being captured by the imaging unit, The control unit adjusts the image of the exterior view displayed on the display unit accordingly. A video display device characterized by superimposing and displaying the avatar based on the user's movement trajectory.

8. In a video display system that connects multiple video display devices carried by a user and displays the user's movement trajectory as a video on the video display devices, The aforementioned video display devices are, A sensor that detects the position and orientation of the user carrying the video display device, A trajectory information acquisition unit acquires information on the user's movement trajectory from the detection results of the aforementioned sensor, A storage unit that stores the user's trajectory information acquired by the aforementioned trajectory information acquisition unit and the information of the avatar, which is a virtual image representing the user. A display unit that shows the user's movement trajectory using an avatar, The system comprises a trajectory information acquisition unit and a control unit for controlling the display unit, The control unit, An avatar is generated from the avatar information stored in the memory unit, and the current field of view of the user is determined by the sensor. The user's trajectory information stored in the memory unit and the generated avatar are placed on the user's movement trajectory, aligned with the user's current field of view, and displayed on the display unit. A video display system characterized in that, when moving the avatar along the user's movement trajectory, the orientation of the avatar's body when it is moving is displayed in the opposite direction to the avatar's direction of movement, and the avatar is moved from newer trajectory information to older trajectory information in time.

9. In a video display system that connects multiple video display devices carried by multiple users and displays the movement trajectories of the multiple users as video on the multiple video display devices, Each of the aforementioned multiple video display devices is: A sensor that detects the position and orientation of the user carrying the video display device, A trajectory information acquisition unit acquires information on the user's movement trajectory from the detection results of the aforementioned sensor, A storage unit that stores the user's trajectory information acquired by the aforementioned trajectory information acquisition unit, and the information of each avatar, which is a virtual image representing the user and other users. A communication processing unit that sends and receives trajectory information of each user with other video display devices carried by other users, A display unit that shows the movement trajectory of each user with their respective avatar, The system comprises the trajectory information acquisition unit, the communication processing unit, and the control unit for controlling the display unit, The control unit, The system generates avatars for each user from the avatar information stored in the memory unit, and uses the sensor to determine the current field of view of the user. Based on the user's trajectory information stored in the memory unit and the trajectory information of other users received by the communication processing unit, the generated avatars are placed on the movement trajectories of the user and other users, aligned with the current field of view of the user, and displayed on the display unit. A video display system characterized in that, when moving the avatar along the user's movement trajectory, the orientation of the avatar's body when it is moving is displayed in the opposite direction to the avatar's direction of movement, and the avatar is moved from newer trajectory information to older trajectory information in time.

10. In a video display method that shows the user's movement trajectory in video, The steps include detecting the user's position and direction and obtaining information on the user's movement trajectory, The steps include storing the acquired user trajectory information in a storage unit, The steps include generating an avatar, which is a virtual image representing the user, The steps include determining the current field of view of the user, Based on the user's trajectory information stored in the memory unit, the generated avatar is positioned on the user's movement trajectory to match the user's current field of view and displayed on the display unit. A video display method characterized by the step of moving the avatar along the user's movement trajectory, displaying the avatar's body orientation in the opposite direction to the avatar's direction of movement, and moving from newer trajectory information to older trajectory information in time.

Images