Video display system and method

Abstract

To provide a technique capable of improving the operability and usability for a user in relation to a generated image such as augmented reality (AR), with respect to a video display device.SOLUTION: An HMD serving as a video display device displays an AR image 30 on a display screen 2, configures an ID image 40 including identification information (ID) for supporting an operation of a user to the AR image 30, detects a sight line direction of the user to set a sight line region 20, and displays the ID image 40 so as to be associated with the AR image 30 when the AR image 30 falls within a range of the sight line region 20. The HMD receives an operation to select the ID image 40 from the user, selects the AR image 30 associated with the ID image 40 when receiving the operation to select the ID image 40, and executes predetermined processing associated with the AR image 30.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to technology for image display devices, and more particularly to technology for displaying a computer-generated image (sometimes referred to as a virtual image) in contrast to an image in real space (sometimes referred to as a real image). [Background technology]

[0002] Technologies for displaying generated images such as Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR) images on real objects are being developed. Generated images include still images and videos. Examples of image display devices capable of displaying generated images include head mounted displays (HMDs) and head up displays (HUDs). For example, an HMD with an AR function displays an AR image (sometimes referred to as an AR object) that is a generated image superimposed on a real image on a transparent or non-transparent display surface. A user selects a desired AR image from multiple AR images on the display surface. The HMD executes a predetermined process associated with the selected AR image.

[0003] An example of the prior art related to the above-mentioned image display device is JP 2016-218868 A (Patent Document 1). Patent Document 1 describes that the selectability of displayed object data is improved in a display control method or the like. Patent Document 1 describes that the HMD generates identification information in a number corresponding to the number of selectable object data included in the display range of the display, and displays each of the generated identification information in association with each of the object data included in the display range. That is, Patent Document 1 describes that, as shown in FIG. 2 and the like, an image (sometimes referred to as an ID image) of identification information (sometimes referred to as an ID image) is added to the AR image and displayed. This ID image can also be considered as a type of AR image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2016-218868 A Summary of the Invention [Problem to be solved by the invention]

[0005] In the image display device such as HMD of the conventional technology, for example, when there are many objects and many AR images on the display screen, it is difficult for the user to select the desired AR image. In addition, the method of selecting the AR image in the conventional technology is, for example, a method of detecting a gesture of fingers using a camera. For example, this gesture includes an operation of virtually touching the desired AR image with fingers or an operation of pinching. However, this method may make it difficult to detect the selection operation of the AR image.

[0006] The image display device of the conventional art has room for improvement in terms of user operability and usability of generated images such as AR. The object of the present invention is to provide a technology that can improve the user operability and usability of generated images such as AR with respect to the technology of the image display device. Problems, configurations, effects, etc. other than those described above will be described in the embodiment of the invention. [Means for solving the problem]

[0007] A representative embodiment of the present invention has the following configuration: An image display device in one embodiment is an image display device that displays a generated image superimposed on a real image, displays the generated image on a display surface, configures an ID image including identification information for assisting a user in performing operations on the generated image, detects the user's line of sight and sets a line of sight area according to the line of sight direction, and if the generated image falls within the line of sight area, displays the generated image in association with the ID image, accepts a selection operation by the user on the ID image, and when the selection operation on the ID image is received, selects the generated image associated with the ID image and executes a predetermined process associated with the generated image. Effect of the Invention

[0008] According to a representative embodiment of the present invention, in relation to video display device technology, it is possible to improve the user operability and ease of use regarding generated images such as AR. [Brief description of the drawings]

[0009] [Figure 1] 1 is a diagram showing the external appearance of an HMD which is an image display device according to a first embodiment of the present invention. [Diagram 2] 1 is a diagram showing a functional block configuration of a video display device according to a first embodiment. [Diagram 3] FIG. 2 is a diagram showing a main process flow in the video display device according to the first embodiment. [Figure 4] FIG. 11 is a diagram showing a process flow relating to an ID image in the video display device of the first embodiment. [Diagram 5] FIG. 2 is a diagram illustrating an example of a software configuration of the video display device according to the first embodiment. [Figure 6] FIG. 2 is a diagram showing the basics of ID image display in the video display device of the first embodiment. [Figure 7] 3 is a diagram showing a configuration example of a line-of-sight area in the video display device according to the first embodiment. FIG. [Figure 8] FIG. 2 is a diagram showing an example of a configuration of a management table in the video display device according to the first embodiment. [Figure 9]FIG. 4 is a diagram showing a command table for voice input in the video display device of the first embodiment. [Figure 10] 3A to 3C are diagrams showing a first example of a real image, an AR image, and an ID image in the video display device of the first embodiment. [Figure 11] 5A to 5C are diagrams showing a second example of a real image, an AR image, and an ID image in the video display device of the first embodiment. [Figure 12] FIG. 2 is a diagram showing a first example of an ID image in the video display device of the first embodiment. [Figure 13] FIG. 11 is a diagram showing a second example of an ID image in the video display device of the first embodiment. [Figure 14] FIG. 2 is a diagram showing a first state of an image in the image display device of the first embodiment. [Figure 15] FIG. 11 is a diagram showing a second state of the image in the image display device of the first embodiment. [Figure 16] FIG. 2 is a diagram showing an example of an image displayed by the image display device of the first embodiment. [Figure 17] FIG. 11 is a diagram showing an example of when an ID image is selected in the video display device of the first embodiment. [Figure 18] FIG. 2 is a diagram showing a first example of enlarging / reducing a line-of-sight area in the video display device of the first embodiment. [Figure 19] FIG. 11 is a diagram showing a second example of enlarging / reducing the line-of-sight area in the video display device of the first embodiment. [Figure 20] FIG. 2 is a diagram showing an example of fixing a gaze area in the video display device according to the first embodiment. [Figure 21] 4A to 4C are diagrams showing examples of deviation of the line-of-sight area in the image display device of the first embodiment. [Figure 22] FIG. 11 is a diagram showing an example of an image displayed on the image display device according to the second embodiment of the present invention. [Figure 23] FIG. 11 is a diagram showing a first example of a selection region in a video display device according to a second embodiment. [Figure 24] FIG. 11 is a diagram showing a second example of a selection region in the video display device according to the second embodiment. [Diagram 25] FIG. 11 is a diagram showing a third example of a selection region in the video display device according to the second embodiment. [Figure 26]FIG. 11 is a diagram showing a fourth example of a selection region in the video display device according to the second embodiment. [Figure 27] FIG. 11 is a diagram showing a process flow in a video display device according to a third embodiment of the present invention. [Figure 28] FIG. 13 is a diagram showing an example of an image displayed on a video display device according to a fourth embodiment of the present invention. [Figure 29] FIG. 13 is a diagram showing the line-of-sight depth in a video display device according to a fifth embodiment of the present invention. [Diagram 30] FIG. 23 is a diagram showing an example of an image displayed on a video display device according to a sixth embodiment of the present invention. [Diagram 31] FIG. 13 is a diagram showing an example of an image displayed on a video display device according to a seventh embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In addition, in all the drawings for explaining the embodiment, the same parts are basically given the same reference numerals, and the repeated explanation will be omitted.

[0011] (Embodiment 1) 1 to 21, the image display device according to the first embodiment of the present invention will be described. The image display device according to the first embodiment will be described as being applied to an HMD. Although the image display device according to the first embodiment will be described as being a transmissive HMD, the image display device can be similarly applied to other types of HMD such as a non-transmissive HMD.

[0012] [HMD(1)] FIG. 1 shows the appearance of the HMD1, which is an image display device according to the first embodiment. The HMD1 is provided with a display surface 2, a speaker 3, a gaze detection unit 4, a camera 5, and a microphone (not shown) in a head-mounted housing. The housing of the HMD1 includes a control unit and a display unit. The control unit is composed of an integrated circuit or the like, and controls the entire HMD1 including the display unit. The display unit includes the display surface 2 and displays an image on the display surface 2. The display surface 2 is an area in which the user can view the image. The display surface 2 is, for example, a display element in the shape of glasses arranged in front of both eyes of the user, and is, for example, a transmissive display surface. That is, the HMD1 is a transmissive HMD. The control unit controls the display of an image including an AR image on the display surface 2. The display unit is, for example, composed of a projection display device, and includes a light source, an optical system such as a lens, a driving circuit, and the like. In the case of a transmissive HMD, a real image in front is transmitted through the display surface 2 seen from the user's eyes, and a generated image such as an AR image is superimposed on the real image. When a non-transmissive HMD is used in another embodiment, the display unit including the display surface 2 is configured, for example, with a liquid crystal display element. In this case, an image obtained by synthesizing a generated image with an image of the front captured by the camera 5 is displayed on the display surface 2. The image display method on the display surface 2 can be, for example, a projection display method, but is not particularly limited.

[0013] The speaker 3 emits sound according to control from the control unit. The microphone inputs the user's voice and sends it to the control unit. The control unit performs voice recognition processing from the voice input from the microphone to detect predetermined commands and the like. The control unit executes predetermined processing according to the detected command. This allows various functions, including AR control, to be realized. The memory of the HMD 1 is provided with a dictionary of voice commands (a command table described below) in which various commands are registered in advance. The HMD 1 of the first embodiment allows the selection of an ID image of an AR image and the operation of the gaze area using voice commands.

[0014] The gaze detection unit 4 detects the state of the gaze of both eyes of the user, at least the gaze direction. The gaze detection unit 4 may also detect other conditions such as blinking of the eyes. The control unit constantly grasps the gaze direction of the user in cooperation with the gaze detection unit 4. The camera 5 captures an image in front of the HMD 1. The camera 5 may be a stereo camera or the like. The control unit processes the image from the camera 5 to, for example, detect objects.

[0015] The housing of the HMD 1 is equipped with sensors such as a GPS receiver, an electronic compass, a gyro sensor, an acceleration sensor, etc. The HMD 1 uses these sensors to detect the position, orientation, acceleration, etc. of the HMD 1.

[0016] An operation device (in other words, a remote control) may be connected to the HMD1 in FIG. 1, in other words, the main body of the image display device, through a wireless communication interface or a wired communication interface. The control unit cooperates with the operation device and accepts operation input from the operation device. In this case, the user can issue commands to the main body device by operating buttons on the operation device with his / her fingers. The operation device is not limited to a dedicated device, and may be a general-purpose mobile information terminal device such as the user's smartphone. In this case, the user can operate the HMD1 from the smartphone or the like. For example, the user's smartphone may have an AR app or data. In that case, the control unit of the HMD1 cooperates with the AR app or data in the smartphone through communication.

[0017] [HMD(2)] 2 shows a functional block configuration of the HMD 1. The HMD 1 has a main control unit 100, a storage unit 110, a video processing unit 120, an audio processing unit 130, a communication processing unit 140, a sensor unit 150, an operation input unit 160, a battery 170, a system bus 180, and the like. Each unit transmits and receives commands and data via the system bus 180. The main control unit 2 is a processor constituted by an MPU or the like, and controls the entire HMD 1 according to a program 111 in the storage unit 120. The processor reads the data of the program in the storage device onto a processing memory, and executes processing according to the program, thereby realizing predetermined functions, etc.

[0018] The storage unit 120 is configured with a non-volatile storage device such as a flash ROM or SSD. The storage unit 120 stores data and information such as a program 111 for operation control, a DB 112, and setting information 113. The program 111 includes a program for implementing the OS of the HMD 1, a program for implementing the AR function, a program for implementing the assistance function, and the like.

[0019] The DB 112 is a database that stores various data such as content data such as AR images, data such as ID images for assistance, and management information, and includes data with data structures such as files, tables, or lists. The setting information 113 includes system setting information and user setting information. The storage unit 120 includes a work area for processing. The storage unit 120 also stores various information, such as AR programs, content data such as video and audio, acquired from a communication network or an external device. The various information includes, for example, search information related to objects on a map. The storage unit 120 also stores video data captured using the camera 5. Not limited to this, various data and programs may be stored in an external server device linked to the HMD 1.

[0020] The sensor unit 150 includes various sensors for detecting the state of the HMD 1 and its vicinity. The sensor unit 150 includes, for example, a GPS receiver, a geomagnetic sensor (electronic compass), an acceleration sensor, a gyro sensor, a distance sensor, and the like. Using these sensors, the HMD 1 can detect the position (e.g., latitude, longitude, altitude), orientation (e.g., azimuth angle and elevation angle), inclination, movements such as acceleration, distance to an object, and the like of the HMD 1. The sensor unit 150 may include other sensors such as an illuminance sensor, a proximity sensor, and an air pressure sensor.

[0021] The communication processing unit 140 is composed of an IC chip or the like, and includes a communication processing circuit, an antenna, and the like. The communication processing unit 140 includes a LAN communication unit 141, a telephone network communication unit 142, and the like. The LAN communication unit 141 performs data communication processing with a LAN via an access point using a method such as Wi-Fi (registered trademark). The telephone network communication unit 142 enables telephone and data communication by performing wireless communication processing with a base station of a mobile telephone communication network using a method such as W-CDMA (registered trademark), GSM, or LTE. The communication processing unit 140 may also include a communication unit using a method such as BlueTooth (registered trademark), infrared, or the like.

[0022] The image processing unit 120 is composed of an IC chip or the like, and includes an image capturing unit 121, a gaze detection unit 4, a display unit 123, and the like. The image capturing unit 121 includes a camera 5. The display unit 123 includes a display surface 2. The camera 5 of the image capturing unit 121 obtains image data by converting light input from a lens into an electric signal using an element such as a CCD or CMOS. A known method can be applied as the gaze detection method of the gaze detection unit 4. For example, the method includes a method of detecting the gaze by capturing an image of the inner corner of the eye and the iris with a visible camera, and a method of detecting the gaze using a corneal reflex method with an infrared LED and an infrared camera. The display unit 123 includes an image processing circuit, a video RAM, and the like, and displays an AR image superimposed on the display surface 2 based on the image data input to the video RAM. Note that a touch panel or the like may be further applied to the display unit 123.

[0023] The voice processing unit 130 is composed of an IC chip or the like, and includes a voice input unit 131, a voice recognition unit 132, and a voice output unit 133. The voice input unit 131 includes a microphone and converts input voice into voice data. The voice output unit 133 includes a speaker 3 and converts voice data into voice and outputs it. The voice recognition unit 132 analyzes the input voice data and detects commands, etc. The voice recognition unit 132 may be composed of a dedicated IC, or may be composed of a program processed by the main control unit 100.

[0024] The operation input unit 160 is a part that accepts basic operation inputs (e.g., power on / off, volume adjustment, etc.) for the HMD 1, and includes, for example, hardware buttons, touch sensors, etc. The HMD 1 uses power from a battery 180 to operate even when no power is supplied from an external source.

[0025] [Processing flow (1)] Fig. 3 shows a flow of main processing of the HMD 1. The flow of Fig. 3 has steps S1 to S9. The steps will be explained below in order. The processor is responsible for the main processing.

[0026] In step S1, the HMD1 detects the position and orientation of the HMD1. The position of the HMD1 is, for example, a position in a geographical three-dimensional space, and is obtained as coordinates (latitude, longitude, altitude) using a GPS receiver or the like. The orientation of the HMD1 corresponds to the direction (Z direction) in which the user looks forward on the display surface 2 in FIG. 1, and is obtained as an azimuth angle and an elevation angle using an electronic compass. In addition, the HMD1 detects the current state using the sensor unit 150.

[0027] In step S2, the HMD1 (e.g., an AR program) generates and displays an AR image in a predetermined manner. For example, the HMD1 detects a predetermined object to be the target of AR from within the real image on the display surface 2. For example, the HMD1 detects the three-dimensional position of the object and the distance from the HMD1 using the camera 5 or the like, and detects the two-dimensional position and image area of ​​the object on the display surface 2. The HMD1 acquires object information regarding the detected object. For example, the HMD1 may search for object information corresponding to the position of the object from map data or the Internet. Alternatively, the HMD1 may acquire information of objects existing near the current position from map data based on the current position of the HMD1. The HMD1 may acquire information of objects such as stores on a map using a service such as Google Street View.

[0028] The HMD 1 generates an AR image based on the acquired object information, and displays it on the display surface 2. For example, the AR program of the HMD 1 displays the AR image in association with the position and image area of ​​the object on the display surface 2.

[0029] The HMD 1 detects an AR image generated and displayed by the AR program and acquires information about the AR image. The HMD 1 may refer to information about AR objects stored in the DB 112 (a management table described later) or may acquire information about AR objects from the Internet.

[0030] In step S3, the HMD1 detects the user's gaze direction, such as the gaze direction ED in FIG. 1, in real time using the gaze detection unit 4. The HMD1 sets a gaze area (FIG. 6 described later) on the display surface 2 according to the detected gaze direction. The gaze area is set, for example, as a circular area having a predetermined radius with the point where the gaze direction intersects with the display surface 2 as the center point. The HMD1 of the first embodiment is configured not to display the gaze area on the display surface 2 in the basic setting, but may be configured to display the gaze area on the display surface 2 in the basic setting. The HMD1 of the first embodiment can also switch between displaying and not displaying the gaze area in response to a user's operation (a command described later). The position of the gaze area, etc. is updated at each control timing. If the movement of the gaze is small, the same gaze area as the previous time is used.

[0031] In addition, in the process of step S3, the HMD 1 also determines whether the line of sight area has deviated from the display surface 2. When the line of sight area deviates, the HMD 1 executes a predetermined process. This process may include, for example, a process of reconstructing an ID image in the line of sight area. When the HMD 1 starts to be used, the initial setting line of sight area is used.

[0032] In step S4, the HMD 1 performs a corresponding determination within the display surface 2 using the AR image in step S2 and the line-of-sight area in step S3. That is, the HMD 1 determines whether the position or area of ​​the AR image is within the line-of-sight area at that time, whether they overlap, etc. For example, the HMD 1 determines whether the "display position" of the "AR image" in a management table (FIG. 8) described later is within the range of the line-of-sight area at that time.

[0033] In step S5, the HMD 1 generates an ID image to be associated with the corresponding AR image in the line of sight area, and displays the ID image at a position associated with the AR image on the display surface 2. This ID image indicates that the corresponding AR image can be selected. At this time, if there is one or more AR images in the line of sight area, the HMD 1 assigns ID numbers to them in order, for example, starting from 1. Then, the HMD 1 generates an ID image with that ID, and displays it in a position near the AR image.

[0034] In this way, the HMD1 of the first embodiment displays an ID image in a limited manner for an AR image that is within the line of sight area, and does not display an ID image for an AR image that is outside the line of sight area. Note that the HMD1 may prepare an ID and an ID image for an AR image that is outside the line of sight area by internal processing, but does not display them. An AR image that can be selected is an AR image to which an ID image that is within the line of sight area is attached. As described above, the number of ID images displayed on the display surface 2 can be reduced, and the amount of information can be reduced, making it easier for the user to select an ID image (corresponding AR image).

[0035] In step S6, the HMD 1 accepts an operation input from the user. In particular, the HMD 1 accepts an input of a selection operation related to an ID image. The HMD 1 of the first embodiment accepts at least an ID image selection operation by a command input of voice recognition.

[0036] In step S7, the HMD 1 checks whether or not there has been an input for selecting an ID image in step S6. If there has been an input for selecting an ID image (Y), the HMD 1 proceeds to step S8, and if there has not been an input for selecting an ID image (N), the HMD 1 proceeds to step S9.

[0037] In step S8, the HMD 1 selects the AR image indicated by the ID of the selected ID image (in other words, puts the AR image into a selected state), and causes a predetermined process associated with the AR image to be executed by the corresponding AR program.

[0038] In step S9, the HMD1 checks whether a specified termination operation related to the AR function and the assistance function has been performed, and if the termination operation has been performed (Y), the flow ends, and if the termination operation has not been performed (N), the HMD1 returns to the initial step S1 and repeats the same processing.

[0039] In addition, in the repetition of the loop of the above flow, in step S1, the HMD1 may determine whether the change in the position, orientation, etc. of the HMD1 is equal to or greater than a predetermined degree, and if it is equal to or greater than the predetermined degree, may execute the subsequent update process (for example, update of the gaze area or ID image). Similarly, in step S3, the HMD1 may determine whether the change in the gaze direction is equal to or greater than a predetermined degree, and if it is equal to or greater than the predetermined degree, may execute the subsequent update process. In addition, the HMD1 may use a set threshold value when determining the change in the gaze area. For example, when the angle representing the gaze direction is equal to or greater than a set angle threshold value (for example, 10 degrees), the HMD1 may determine that the gaze direction has changed, and execute the update process. Alternatively, the HMD1 may determine that the gaze direction has changed, when the angle representing the gaze direction has changed continuously for a period of equal to or greater than a predetermined time threshold value (for example, 1 second). In addition, for example, when the difference in the position of the HMD1 between time points is equal to or greater than a set distance threshold value (for example, 0.1 m), the HMD1 may determine that the position of the HMD1 has changed, and execute the update process. If there is no change or there is only a small change in the line of sight area, etc., the HMD 1 continues to use the line of sight area, etc. that was previously set.

[0040] [Processing flow (2)] FIG. 4 shows a detailed process example flow relating to the processing of the ID image in steps S6 to S8. FIG. 4 includes steps S21 to S27. In step S21, the HMD 1 detects a predetermined command based on voice recognition. The HMD 1 performs processing according to a command table (FIG. 9) described below. In step S22, the HMD 1 executes processing of a predetermined function corresponding to the detected command. The commands include commands relating to the operation of the line of sight area, commands relating to the operation of the ID image, etc., and corresponding processing is defined for each command.

[0041] In step S23, the HMD 1 checks whether an ID image has been selected, and if not (N), ends the flow, and if selected (Y), proceeds to step S24. In step S24, the HMD 1 differentiates the display state of the selected ID image on the display surface 2. That is, the HMD 1 displays the ID image on the display surface 2 with a predetermined display effect (e.g., blinking display, high brightness display, specific color display, etc.). This notifies the user which ID image has been selected, etc.

[0042] In step S25, the HMD1 selects an AR image associated with the selected ID image. For example, the HMD1 transmits a selection notification to an AR program associated with the selected AR image (see FIG. 5 described later). As a result, the AR program that receives the selection notification selects the AR image and executes a predetermined process corresponding to the selection of the AR image. As an example, in the case of an AR application that provides information on stores on a map, the AR application obtains detailed information on the store corresponding to the selected AR image, and generates and displays an AR image that displays the detailed information. As another example, in the case of an AR application that allows a user to arbitrarily place an AR image in a real image, the AR application selects the AR image, moves or erases the AR image, or issues a command to the AR image in response to a user's operation.

[0043] In the first embodiment, a gesture method can be used in addition to the operation input method. The flow in FIG. 4 shows a flow in the case where this method is used in addition, and includes corresponding steps S26 and S27. In step S26, the HMD 1 obtains an image of the front using the camera 5, and determines and detects a predetermined gesture from the image by image recognition processing. The gesture to be detected is, for example, a touch gesture in which the user places the fingers of the hand on the position of the ID image in the area corresponding to the display surface 2 and keeps it still for a predetermined period of time or more. Alternatively, another gesture to be detected is a gesture in which the user virtually pinches the ID image with the fingers of the hand.

[0044] In step S27, the HMD 1 executes a process of a predetermined function that is predefined in response to the detected gesture. This process includes at least an ID image selection operation process in response to the predetermined gesture. After step S27, the process proceeds to step S23, where it is similarly confirmed whether an ID image has been selected.

[0045] [Software configuration example] FIG. 5 shows an example of the software configuration of the HMD1 of the first embodiment. FIG. 5 shows an example in which an assistance function is implemented at the level of the OS or middleware of the HMD1. The HMD1 has an OS500, an AR program 501, an AR application 502, and a program 510. The OS500 includes middleware and the like. The AR program 501 is a program included in the OS500 and has a function of generating and displaying an AR image. The AR program 501 is a program that displays, for example, information such as the time, battery state, and communication state of the HMD1, icons of various functions, and the like as an AR image on the display surface 2. In this example, a case is shown in which the AR program 501 generates an AR image Ax and displays it on the display surface 2.

[0046] The AR application 502 is one of various application programs that run on the OS 500 and have the function of generating and displaying AR images. The AR application 502 includes, for example, AR applications P1 to Pm as a plurality of AR applications. In this example, the AR applications 502 (P1 to Pm) generate AR images A1 to Am and display them on the display surface 2.

[0047] As in this example, the HMD 1 is assumed to display an AR image on the display surface 2 by an AR program 501 and each AR application 502 of an OS 500. The AR program 501 and each AR application 502 each have a function of executing a predetermined process associated with the AR image.

[0048] The HMD 1 of the first embodiment has a program 510 in which an assistance function is implemented. The program 510 may be implemented as a part of the OS 500, or may be implemented at the level of one application. The main control unit 100 executes processing according to the program 510 on the OS 500 to realize the assistance function. This assistance function is an AR image operation assistance function. This assistance function is a function that uses an ID image to assist in making selection operations and the like easier for a plurality of AR images by the AR program 501 and each AR application 502. This assistance function is a function that assigns and displays an ID image to an AR image according to a gaze area. This assistance function is a function that enables selection operations of an AR image by selection operations of an ID image.

[0049] The program 510 cooperates with the AR program 501 and each AR application 502. The program 510 generates an ID image 504 for an AR image 504 displayed on the display surface 2, and displays the ID image 504 in association with the AR image 504. For example, the following shows a case in which ID images B1 to Bm and an ID image Bx are displayed in association with AR images A1 to Am and AR image Ax, respectively.

[0050] The program 510 accepts an input of a selection operation of the ID image 504 by the user (for example, a voice command specifying an ID). The program 510 interprets the input of the selection operation as a selection of an ID image and an AR image. When the AR image 503 associated with the selected ID image 504 is selected, the program 510 transmits a selection notification to the corresponding program (the AR program 501 or the AR application 502). The program that receives the selection notification selects the target AR image and executes a predetermined process associated with the AR image.

[0051] In another embodiment, the support function of the program 510 may be implemented only in a specific application. In this case, the specific application provides support related to the selection operation by displaying an ID image only for a specific AR image that the specific application controls.

[0052] [ID image display] FIG. 6 shows the basics of assigning and displaying an ID image to a video including an AR image on the display surface 2. In FIG. 6, a rectangular frame shows the display surface 2 in a schematic manner. The two-dimensional coordinate system of the display surface 2 is indicated by (x, y). The x direction is the horizontal direction in the plane, and the y direction is the vertical direction in the plane. In the display surface 2, one or more AR images 30 may be displayed depending on an object or geographical position not shown, or independently of the object. In this example, the AR image 30 has an AR image a1 with a heart-shaped mark and an AR image a2 with a star-shaped mark. The AR image a1 is displayed at a position p2 (x2, y2) in the display surface 2.

[0053] Meanwhile, on the display surface 2, a gaze area 20 is set according to the user's gaze direction. The gaze area 20 is dynamically set according to changes in the gaze direction. The point where the user's gaze direction and the display surface 2 intersect is assumed to be position p1 (x1, y1), for example. In this case, a circular area having a predetermined radius r1 with position p1 as its center is set as the gaze area 20. The gaze area 20 is not displayed in the basic settings, but is illustrated by a dashed line.

[0054] For example, an AR image 30 (a1) is included in the line-of-sight region 20. In this case, the HMD 1 generates an ID image 40 (b1) for the AR image 30 (a1) and displays it in association with the AR image 30 (a1). In this example, the ID image 40 has a rectangular ID image b1 with the ID number "1". The ID image b1 is, for example, an image in which the number "1" is arranged within a rectangular frame. The inside of the frame of the ID image b1 is, for example, a transparent region, but may be a non-transparent region. In this example, the ID image b1 is displayed at a position p3 (x3, y3) close to the position p2 of the AR image a1, for example, at a position above the AR image a1. Since the AR image a2 is not included in the line-of-sight region 20, the ID image 40 is not displayed. The position at which the ID image 40 is displayed relative to the position of the AR image 30 is not limited to the upper position, and a suitable vacant position is selected depending on the display state of the AR image 30 and the ID image 40 in the display surface 2 at that time, for example.

[0055] [View area] FIG. 7 shows an example of the configuration of the gaze area 20. In FIG. 7, gaze areas 20a, 20b, and 20c are shown as examples of the gaze area 20. The gaze area 20a is an example having a circular shape, similar to FIG. 6. The shape of the gaze area 20 is not limited to this, and various shapes can be set. The gaze area 20b is an example having a horizontally long elliptical shape. The gaze area 20c is an example having a horizontally long rectangle. In addition, the size of the gaze area 20, such as the radius, may be a fixed size set in advance, or may be changeable by user setting. For example, the vertical and horizontal widths of the gaze area 20 may be set by the user. In addition, as will be described later, when using the HMD 1, the size of the gaze area 20, etc., can be changed in response to a predetermined operation input by the user.

[0056] Also, the case is shown where gaze area 20a is set to position pc(xc, yc) at the center of display surface 2. In the first embodiment, the position of gaze area 20 varies depending on the gaze direction of the user. For example, when the position at the end of the gaze direction changes from position pc to position pd(xd, yd), gaze area 20a moves to gaze area 20d.

[0057] In a modified example of HMD1, the gaze area 20 may be fixedly set at a fixed position, for example, at the central position pc(xc, yc), in the display surface 2 that is set in advance, without depending on the user's gaze direction. The vertical and horizontal dashed lines passing through the position pc are the bisectors of the vertical and horizontal widths of the display surface 2. This modification corresponds to a form in which the gaze area 20 is controlled by regarding the user's gaze direction as approximately the same as the orientation of the HMD1. In this modification, the hardware and software configurations can be further simplified. The user changes the orientation of the HMD1 by moving his head. This causes the gaze area to change in space.

[0058] As will be described later, it is possible to fix or release the position of the gaze area 20 on the display surface 2 in response to a predetermined operational input by the user. This makes it possible to fix the display of an ID image in the gaze area 20 while the user is focusing on a certain gaze direction.

[0059] As will be described later, an image representing the gaze area 20 (gaze area boundary line) can be displayed as a type of AR image on the display surface 2, or can be switched to non-display in response to a predetermined operation input by the user. The image representing the gaze area 20 is, for example, a boundary line, and may be an image of a dashed line as shown in the figure. For example, when the gaze area 20a is displayed, a dashed circular line as shown in the figure is displayed as the boundary line.

[0060] [Management table] FIG. 8 shows an example of the configuration of a management table for control of AR images and ID images. This management table has a list of all AR images and ID images. The HMD1 reads and writes information of all AR images and ID images from this management table. The processor of the HMD1 creates and stores the management table in memory as part of the DB 112. The HMD1 can grasp the state of the AR image or ID image at that time by referring to this management table.

[0061] The management table in FIG. 8 has the following items: management ID, AR object, display position, base position, and identification information. The "management ID" is identification information for management, and is an address or a line number. The "AR object" is an AR image, and has a corresponding image data file, etc. The "display position" is a two-dimensional coordinate (x, y) that is a reference position when the AR image is displayed on the display surface 2. The "base position" is a three-dimensional position coordinate (X, Y, Z) that is a base for generating the AR image. The "base position" corresponds to a position on a map or in real space, for example. Note that, in the case of an AR image generated independently of a real object, there is a case where the "base position" does not exist. The "identification information" includes an ID and an ID image. The ID is information such as a number for identifying and selecting an AR image. The ID image is an image for display that includes the ID, and has a corresponding image data file. The ID number is generated in order, for example, 1, 2, 3, ...... Note that, in the case where an ID image is not assigned to an AR image, a value (0) indicating no ID image is set in the "identification information" item. As another data item, a flag indicating whether or not the ID image can be selected may be provided.

[0062] [Command Table] FIG. 9 shows an example of the configuration of a command table corresponding to the voice recognition method in the first embodiment. In this command table, commands that can be used are set in advance. This command table has the following items: command ID, target (type), command, voice, and remarks. A command is set for each row of the table. The HMD1 detects a "command" in the command table from the input voice. "Command ID" indicates the identifier of the "command". "Target (type)" indicates the target and type of the "command". "Voice" indicates an example of the voice when the "command" is input. "Voice" is not limited to the example voices, and can be set in various ways as long as it is a voice that can uniquely distinguish each command. "Remarks" indicates an explanation of the processing content, etc.

[0063] Command IDs C01 to C06 have a "target (type)" of "gaze" and are a group of commands relating to control of the gaze area. Command IDs C11 to C42 have a "target (type)" of "ID" and are a group of commands relating to control of the ID image.

[0064] The "gaze area display on" command with command ID=C01 is a command to switch to a state in which an image (e.g., a boundary line) representing the gaze area 20 in FIG. 6 or the like is displayed on the display surface 2, and the sound is, for example, "Shisen on." The "gaze area display off" command with command ID=C02 is a command to switch to a state in which an image representing the gaze area 20 is not displayed on the display surface 2, and the sound is, for example, "Shisen off."

[0065] The "fix gaze area" command with command ID=C03 is a command to switch to a state in which the position of gaze area 20 on display surface 2 is fixed, and the sound is, for example, "fix gaze area" or "freeze." The "unfix gaze area" command with command ID=C04 is a command to release the fixation of the position of gaze area 20 and switch to a free state, and the sound is, for example, "unfix gaze area."

[0066] The "expand gaze area" command with command ID=C05 is a command to expand the size of gaze area 20 on display surface 2, and the voice is, for example, "large gaze area", "bigger", etc. The "shrink gaze area" command with command ID=C06 is a command to reduce the size of gaze area 20, and the voice is, for example, "reduced gaze area", "reduced", etc.

[0067] The "ID display on" command with command ID=C11 is a command to switch to a state where an ID image is displayed on the display surface 2, and the voice is, for example, "ID display on", "ID on", etc. The "ID display off" command with command ID=C12 is a command to switch to a state where an ID image is not displayed on the display surface 2, and the voice is, for example, "ID display off", "ID off", etc.

[0068] Commands from the "Select ID1" command with command ID=C21 to the "Select ID9" command with command ID=C29 are commands for individually selecting each of the nine IDs, for example, ID "1" to ID "9". For example, the "Select ID1" command is a command for selecting ID image "1" having ID number "1" (for example, ID image b1 in FIG. 6), and the voice is, for example, "Ichiban", "Bangoichi", "IDiichi", etc. Note that, although this example uses nine IDs, "1" to "9", ten or more IDs may be used, and commands corresponding to numbers "10" and above may be prepared.

[0069] For example, if the voice detected by voice recognition from the user's input voice corresponds to "ichiban," it is interpreted as a "select ID1" command and as a selection operation of the ID image "1." The selection operation of this ID image "1" is then interpreted as a selection operation of the AR image associated with the ID image "1" at that time. This causes the AR image to be selected, and a predetermined process associated with the AR image is executed.

[0070] The "select all IDs" command with command ID=C30 is a command to select all ID images within the gaze area, and the voice is, for example, "all." The "select one ID" command with command ID=C31 is a command to select any one of the ID images within the gaze area according to a predetermined rule, and the voice is, for example, "select." This command is a command to select, for example, one ID image closest to the center point of the gaze area (i.e., the gaze direction) especially when there are multiple ID images within the gaze area.

[0071] The "deselect all IDs" command with command ID=C32 is a command that cancels the selection state of all ID images in the line of sight if there is an ID image in the selected state and returns it to an unselected state. The corresponding sound is, for example, "zenbukaijo." Also, the "deselect 1 ID" command with command ID=C33 is a command that cancels the selection state of any one ID image in the line of sight and returns it to an unselected state. The commands C32 and C33 can be applied when there is a distinction between the selected state and the unselected state for the ID image and the corresponding AR image in terms of control.

[0072] The "ID execution" command with command ID=C34 is a command to execute the processing of the AR image associated with a selected ID image. The voice input is, for example, "execute." The C34 command can be applied when selection and execution are separate in terms of control for an AR image. When selection and execution are not separate in terms of control for an AR image, the selection operation directly corresponds to an instruction to execute processing.

[0073] The "AR information display" command of command ID=C41 is a command that displays information about an ID image in the line of sight area or an AR image associated with a selected ID image. The voice is, for example, "INFORMATION ON". The "AR information non-display" command of C42 is a command that switches the AR information by the command of C41 to a non-display state, and the voice is, for example, "INFORMATION OFF". The information about the AR image is, for example, information such as the name and function of the AR program that generates the AR image. By looking at this information, the user can confirm the function and meaning of the AR image. Note that this command is different from the data output as an example of the processing execution of the AR image by the AR program. An AR program performs data output as an example of a predetermined process when an AR image is selected. For example, in the case of an AR app that provides information about stores on a map, this data output is to display the store information as an AR image.

[0074] In addition to the above example, the HMD 1 may be provided with voice commands for operating various functions by the OS, etc. In addition, in a form in which the AR application and the support function are implemented integrally, an example of the voice command may be a command corresponding to the execution of AR image processing by the AR application. For example, a "data on" command that displays data corresponding to an AR image may be provided. For example, an AR application that provides store information displays an AR image including store information with respect to the selected AR image when the "data on" command is input.

[0075] The voice processing unit 130 in Fig. 2 may have a voice synthesis function. The HMD 1 may output an appropriate voice response to the user using the voice synthesis function. For example, when the user inputs a predetermined voice command, the HMD 1 may output a corresponding voice response. Examples of the voice response include a voice confirming the input command, a voice asking a question, etc.

[0076] [AR and ID images] 10 and 11 show examples of display of an AR image and an ID image on the display surface 2. The example of FIG. 10 shows a schematic example of an image when the HMD 1 is used outdoors, including a real image of a building such as a store near a road. The example of FIG. 11 shows a schematic example of an image when the HMD 1 is used indoors, including a real image of a table, a wall, etc. First, a display example of an AR image that can be displayed by an existing AR program is as follows. The method of generating and displaying an AR image can be applied to known techniques and is not particularly limited. Various AR methods may be mixed. For example, the AR methods of each AR application 503 in FIG. 5 may be different.

[0077] In the image of FIG. 10, AR images A1 to A13 are shown as examples of AR images (=AR objects). AR image A1 and AR image A2 are examples of AR images that a user arbitrarily places in a real image for healing or the like. The AR images are, for example, AR images of animals such as birds and cats or virtual characters, and are composed of, for example, animation images. The user places the selected desired AR image in the display surface 2 by a predetermined operation. The AR image is displayed, for example, at a specified fixed position in the display surface 2. Alternatively, the AR image may move autonomously in the display surface 2 according to a program. For example, the AR image A1 of a bird moves as if flying in the display surface 2. In addition, the AR image may be displayed constantly in the display surface 2 regardless of the positions of the user and the HMD 1.

[0078] The AR images A3 and A4 are examples of AR images that display store information, and are images in which, for example, characters such as the name of a store are displayed in a rectangular area. For example, an AR application uses an AR image to guide information about a store, etc. The AR application searches for nearby stores, etc. from map data according to the current position and orientation of the HMD1. The AR application uses the searched information to generate an AR image A3, etc., and displays it at a position on the display surface 2 that corresponds to the geographical position of the store, etc. Alternatively, an AR application detects an object such as a building or signboard of a store, etc. from an image of the camera 5, obtains information about the object, and generates and displays an AR image using the information. Any information that has been set, such as a name, type, address, or state, can be applied as information about the object.

[0079] AR images A5 and A6 are examples of images that display information such as arrows and speech bubbles for navigation of the user's movement. For example, an AR application has a navigation function including route search, and generates an AR image for navigation of the route from the current position to the destination according to the destination input by the user. For example, AR image A5 is an image that indicates that the user should turn right at the next intersection and that the destination is 30 meters ahead. AR image A6 is an image that indicates that the station is in the direction of the arrow.

[0080] The AR image A7 is an example of displaying a frame image around the face of a detected person. For example, in an AR application, information on people such as acquaintances of a user is registered in advance. The AR application detects the face of the acquaintance based on an image from the camera 5. The AR application displays the detected face area with a frame image and information about the person as the AR image A7. This allows the user to find acquaintances and check information about them.

[0081] AR images A8, A81, A82, and A83 show an example of a group of multiple AR images that display information about today's recommended dishes offered by a certain store shown in AR image A4 when the store is a restaurant. AR image A8 is a text image of "Today's Recommendation." AR images A81, A82, and A83 are images of multiple dishes associated with AR image A8. For example, an AR app automatically displays such detailed information as push information as store information.

[0082] The image in FIG. 11 shows AR images A21 to A28. The AR image A21 is an AR image of a plurality of birds, etc., arranged by the user, and moves autonomously. The AR image A22 is an AR image of a cat, etc., arranged by the user, and is stationary, for example. The AR images A23 and A24 are AR images of a vase, food, etc., arranged by the user on a table in a real image. The AR images A25, A26, and A27 are AR images of a calendar, a painting, a clock, etc., arranged by the user on a wall in a real image. For example, an AR application is an application that can arrange various AR objects at desired positions in a room. The AR image A28 is an AR image of icons of a plurality of documents, etc., arranged by the user at a predetermined position on the display surface 2. When the AR image A28 is selected, the corresponding program displays the contents of the documents, etc. on the display surface 2.

[0083] As in the above example, the display mode of the AR image can be various modes including images such as icons, marks, characters, figures, animations, and three-dimensional models, and is not particularly limited. Also, as in the above example, there are AR images of a type that are displayed in association with real objects, and there are also AR images of a type that are displayed independently of objects. Also, the AR image may be accompanied by audio output (for example, sound effects, animal cries, etc.).

[0084] Another example of the latter type is when a program such as the OS of the HMD 1 displays an AR image for control. Examples of this AR image include AR images of icons of various functions and applications, AR images showing the date and time, battery, communication status, etc., AR images showing email notifications, etc. This type of AR image is displayed on the display surface 2 for a certain period of time, regardless of the position of a real object, for example.

[0085] The method of AR etc. is not limited to the above example, and various known methods can be applied. As an example, an AR marker method may be applied. In this method, the HMD 1 detects a predetermined marker (e.g., a two-dimensional code) from within a real image by recognizing an image from the camera 5, and generates and displays a predetermined AR image in association with the position of the marker. As another method, a method may be used in which an object or point in the line of sight of the user is detected using the line of sight, and an AR image is generated and displayed in association with the object or point.

[0086] An example of displaying ID images for the above AR images is as follows. In FIG. 10, as an example of the gaze area 20, the gaze area 20 is located slightly to the upper right in the display surface 2. A total of seven AR images, AR image A1, AR image A4, AR image A6, AR image A8, A81, A82, and A83, are included in this gaze area 20. The HMD 1 assigns ID numbers to the multiple AR images in the gaze area 20 in order of proximity to the center point and displays the ID images. In this example, seven ID images, ID images "1" to "7", are generated and displayed in order of proximity to the center point of the gaze area 20. For example, the ID image "1" is assigned to the AR image A81. The ID image "2" is assigned to the AR image A82. The ID image "3" is assigned to the AR image A83. The ID image "4" is assigned to the AR image A8. The ID image "5" is assigned to the AR image A1. The ID image "6" is assigned to the AR image A4. The AR image A6 is given an ID image “7.” This example is an example in which ID images are given to all AR images within the line of sight area 20 and are displayed.

[0087] In FIG. 11, as an example of the gaze area 20, the gaze area 20 is located at a slightly upward position within the display surface 2. A total of seven AR images, namely, three AR images A21, A23, A24, A25, and A26, are contained within this gaze area 20. In this example, seven ID images, ID images "1" to "7", are generated and displayed in order of proximity to the center point of the gaze area 20. For example, the three AR images A21 are assigned ID images "1" to "3". The AR image A23 is assigned ID image "4". The AR image A25 is assigned ID image "5". The AR image A24 is assigned ID image "6". The AR image A26 is assigned ID image "7".

[0088] [ID image] In the above example, numbers such as "1" and "2" are used as the IDs of the ID images. The IDs of the ID images may be set within a usable range. For example, in the default settings of the HMD 1, the ID range is set to "1" to "9." The number of ID images that can be simultaneously displayed on the display surface 2 or the number of ID images that can be simultaneously displayed within the line-of-sight area 20 can also be set. For example, in the default settings of the HMD 1, the maximum number of ID images within the line-of-sight area 20 is set to five. These restrictions can reduce the amount of information on the display surface 2, making it easier to see.

[0089] The ID number of the ID image is assigned according to a predetermined rule. For example, the numbers are assigned in order from "1" in the order of proximity to the center point in the line of sight area. The ID number is assigned dynamically according to the situation. A certain same AR image may be assigned a different ID according to the situation. In a first control example described later, the HMD1 controls the ID image to maintain the same ID as much as possible for a certain same AR image in a time series. For example, after a certain ID is assigned to a certain AR image, the same ID may be maintained for at least a certain period of time. In a second control example described later, the HMD1 reassigns an ID image to the AR image in the line of sight area according to a predetermined rule at each control point.

[0090] In addition, the HMD 1 enables the user to set the controlled objects, such as displaying an AR image of a specific AR application with an ID image added thereto, and displaying an AR image of another specific AR application with no ID image added thereto.

[0091] FIG. 12 shows examples of various display aspects of ID images, all of which are applicable. In the example (A), an ID image b1 having an ID "1" is displayed without a frame on the left side of the AR image a1. In the example (B), an ID image b1 having an ID "1" is displayed on the AR image a1 so as to be superimposed on the same position. In the example (C), an ID image b1 having an ID "A" is displayed in a circular frame on the upper side of the AR image a1. This is an example in which the English letter "A" is used as the ID. The ID is not limited to numbers, and various characters are applicable.

[0092] The example of (D) is an example in which an ID image b1 having an ID of "1" is placed at a position slightly away from the position of the AR image a1. In addition, the AR image a1 and the ID image b1 are connected by a connecting line to show the association. For example, when there is no space near the AR image a1, such a display may be applied.

[0093] The example of (E) has an AR image a101 for displaying information about store j1 through an association connection line at a position slightly away from the position of store j1 in the real image. The AR image a101 displays characters such as the name of store j1 within a rectangular area. An ID image b1 with an ID of "1" is displayed above the AR image a101.

[0094] In the example of (F), an ID image b1 having an ID of "1" is displayed relative to an AR image A1 that moves within the display surface 2. This ID image b1 is basically displayed at a position that follows the movement of the AR image A1. As a modified example, this ID image b1 is displayed at a fixed position, not at a position that follows the movement of the AR image A1. Alternatively, this ID image b1 may be displayed to move at a slower speed than the movement of the AR image A1.

[0095] The example of (G) shows a case where an ID image is not added to an AR image A5 such as an arrow, even if the image is within the line of sight area, according to user settings. If it is difficult to understand an ID image for a certain type of AR image, it can be prevented from being displayed in this way. Also, for example, in the case of an AR image that originally has characters or shapes such as numbers, it is possible to prevent the ID image from being displayed.

[0096] As another example of the ID, the ID may be identified by a difference in color or shape, for example, a circle, a triangle, a square, etc. Alternatively, the ID may be identified by a color, for example, red, blue, yellow, etc.

[0097] In the example of (A) in FIG. 13, a plurality of, for example, three AR images a1, a2, and a3 are arranged close to each other. Basically, the HMD 1 displays each ID image in a position close to each AR image. In this case, for example, when the ID image is displayed above each AR image, the state shown in (A) is obtained. Three ID images b1, b2, and b3 with IDs "1" to "3" are displayed close to each other. In this case, the information is somewhat difficult for the user to see. In contrast, an example of an improvement is shown in (B) and the like. In the example of (B), the ID images are displayed in different positions, for example, positions that spread outward, for the same three AR images a1 to a3. As a result, the ID images b1 to b3 do not overlap, making them easy to see. In the example of (C), the three ID images b1 to b3 are displayed in a balloon image for the same three AR images a1 to a3. First, the AR images (which may be reduced images) are displayed vertically in the balloon image. In the balloon image, each corresponding ID image is displayed next to each AR image.

[0098] Further, the example of (D) shows an example in which a single ID image b101 is assigned to and displayed as a single group of multiple AR images (for example, AR images a1 to a3). The ID image b101 is, for example, an image having an ID=group ID="A". For example, the ID image b101 of the group ID "A" is displayed at a position near the AR images a1 to a3 or at a position near the balloon image. In this case, the multiple AR images a1 to a3 of the group can be selected collectively in response to a selection operation of the ID image b101 in group units. As a modified example, the HMD1 first displays only the ID image b101 in group units at a position near the multiple AR images a1 to a3 that are densely packed together. When the user selects the ID image b101, the HMD1 displays information on the multiple AR images a1 to a3 and ID images b1 to b3 included in the group as balloon images or the like, allowing them to be individually selected.

[0099] (E) shows an example in which, when there are multiple AR images that are originally related as a group, such as AR images A8, A81 to A83 in Fig. 10, an ID image b101 is similarly assigned and displayed for each group. The HMD 1 surrounds the AR images A8, A81 to A83 that are to be grouped, for example, with a frame, and assigns and displays an ID image b101 that includes the frame. (F) shows an example in which an ID image b101 is similarly assigned and displayed for each group, for the three AR images A21 in Fig. 11.

[0100] [Display control example (1)] The following describes in more detail an example of ID image display control in embodiment 1. The HMD 1 performs control to add and display an ID image in response to a change in the AR image in accordance with a change in the line-of-sight area 20.

[0101] FIG. 14 and the like show a first control example. In the first control example, an ID image with the same ID is maintained as much as possible for a certain AR image. For example, when a certain first AR image enters the line of sight area, an ID image with ID "1" is assigned and displayed. When the certain first AR image then leaves the line of sight area, the ID "1" is maintained in terms of internal processing, and the ID image is hidden. When the certain first AR image then enters the line of sight area again, the ID image with the same ID "1" is displayed again. On the other hand, in the second control example, instead of maintaining an ID image with the same ID as much as possible for a certain AR image, an ID is reassigned each time depending on the state of the line of sight area at that time.

[0102] FIG. 14 shows the first state of the image on the display surface 2. In this example, the real image is near the road, and three store buildings are visible. Examples of AR images displayed in the first state include AR images a1, a2, a3, a4, a5, a11, a12, and a13. The AR images a1 to a5 are examples of marks of various shapes, and the AR images a11, a12, and a13 are examples of rectangular images that display store information. The gaze area 20 is the gaze area E1 in the first state, and has a central position p1 (x1, y1) and a radius r1. In the first state, the AR images a1, a2, and a3 are within the gaze area E1. The AR images a4, a5, a11, a12, and a13 are outside the gaze area E1. In this example, the determination of whether an AR image is within the gaze area is based on whether the "display position" (FIG. 8) of the AR image is within the gaze area, but is not limited to this. As another method, it may be determined whether or not a part of the area of ​​the AR image is within the line-of-sight area.

[0103] For example, the HMD 1 assigns IDs to AR images in order of proximity to the position p1 of the center of the line of sight area E1. In this example, the AR images are a1, a2, and a3 in order of proximity to the position p1. Therefore, the HMD 1 assigns and displays three ID images "1", "2", and "3" to the three AR images a1, a2, and a3.

[0104] The ID number assignment method is not limited to this example, and various methods can be applied. As an example of another method, there is a method of assigning numbers in the order in which AR images are placed in the gaze area. As another example of another method, there is a method of scanning the gaze area in a predetermined order (for example, line-sequential scanning from the upper left to the lower right) and assigning numbers in the corresponding order.

[0105] As in the above example, when there are multiple AR images on the display surface 2, the HMD 1 of the first embodiment displays an ID image limited to within the line of sight area. Therefore, the amount of information is suppressed for the user, making it easy to operate the AR image. The user can operate the AR image by selecting and operating the ID image for the AR image that is of interest near the line of sight. Note that the HMD 1 may erase the ID image after a certain period of time has elapsed after displaying the ID image in the AR image within the line of sight area. The ID image may be a semi-transparent image.

[0106] 15 shows a second state of the image on the display surface 2. Assume that the orientation of the HMD 1 and the line of sight move slightly to the left from the line of sight area E1 in the first state to become the line of sight area E2 in the second state. In the second state, the AR image a1 remains within the line of sight area E2, a new AR image a11 has entered, and the AR images a2 and a3 have moved out. The AR image a11 is an example of a rectangular image that displays store information ("Store AAA").

[0107] The HMD1 maintains the display of the ID image "1" for the AR image a1. The HMD1 hides the ID images "2" and "3" for the AR images a2 and a3. The HMD1 assigns and displays the ID image "4" for the AR image a11. In the first control example, even if the ID images "2" and "3" for the AR images a2 and a3 are hidden, the assigned numbers are maintained, and the number "4", the next number after "3", is assigned to the AR image a11. In the second state, the two ID images "1" and "4" are displayed within the line of sight area E2.

[0108] When the second control example is used, the following occurs. The HMD 1 cancels the assignment of the ID images "2" and "3" to the AR images a2 and a3 that are outside the line of sight E2. The HMD 1 assigns and displays the ID image "2", which has the next number after the ID image "1", to the newly entered AR image a11. In this state, the ID images "1" and "2" are displayed in the line of sight E2.

[0109] Also, for example, when returning from the line of sight area E2 in the second state to the line of sight area E1 in the first state, the first control example is as follows: The HMD1 again displays the same ID images "2" and "3" as before for the AR images a2 and a3 that have entered the line of sight area E1, and does not display the ID image "4" for the AR image a11 that has left the line of sight area E1.

[0110] [Display control example (2)] FIG. 16 shows an example of an image on the display surface 2. It is assumed that the user's gaze moves slightly to the right from the gaze area E1 in the first state in FIG. 14 to the gaze area E3 in FIG. 16. AR images a1, a2, and a3 remain in the gaze area E3, and AR images a4 and a5 are newly included. Here, it is assumed that the ID range is "1" to "9" as a state of user settings, and the maximum number of ID images that can be simultaneously displayed in the gaze area is three. Although five AR image positions are included in the gaze area E3, the HMD 1 displays ID images with a maximum number limited to three.

[0111] In the first control example, the HMD1 first assigns and displays an ID image "4" to the first incoming AR image a4, and hides the ID image "1" for the AR image a1. At this point, the maximum number of ID images "2", "3", and "4" are displayed. The HMD1 assigns and displays an ID image "5" to the next incoming AR image a5, and hides the ID image "2" for the AR image a2. At this point, the maximum number of ID images "3", "4", and "5" are displayed. In this way, the HMD1 displays an ID image for the newly incoming AR image, and sequentially hides older IDs (or IDs behind the movement of the gaze area) so as to keep the number to the maximum. In the same way, when the HMD1 has used up to the last ID "9" in the range, it returns to the first ID "1" in the range and uses them cyclically.

[0112] Regarding a similar situation, the second control example is as follows. Similarly, assume that the range of IDs is "1" to "3" and the maximum number is 3. The HMD1 first assigns and displays the ID image "1" to the first AR image a4, and cancels and erases the assignment of the ID image "1" to the AR image a1. At this point, the maximum number of ID images "1", "2", and "3" are displayed. The HMD1 assigns and displays the ID image "2" to the next AR image a5, and erases the ID image "2" from the AR image a2. At this point, the maximum number of ID images "1", "2", and "3" are displayed. In this way, the HMD1 assigns and displays the ID image to the newly entered AR image so as to always use the ID images "1", "2", and "3" in a predetermined range, and also cancels and erases the assignment of the ID image to any of the AR images.

[0113] As described above, the HMD1 of the first embodiment can set the maximum number of ID images that can be displayed in the line of sight area in advance, and can suppress the number of ID images to be displayed. In addition, for example, there are cases where multiple AR images are densely arranged in the line of sight area, exceeding the maximum number, and multiple ID images related to all of the AR images cannot be displayed. The HMD1 may automatically control the expansion or contraction of the line of sight area according to the density state of the multiple AR images in the line of sight area. For example, the HMD1 may contract the line of sight area to reduce the size when multiple AR images are densely arranged. For example, the HMD1 may expand the line of sight area to increase the size when multiple AR images are sparsely arranged. For example, the HMD1 may adjust the size of the line of sight area to match the maximum number of simultaneous displays. By setting the number of ID images displayed on the display surface 2 to a roughly constant number, the user can easily perform a selection operation. It is also possible to set the maximum number of simultaneous displays to one. In this case, only a maximum of one ID image is displayed in the display surface 2 and the line of sight area 20, making the selection operation easy.

[0114] [Select ID image and process AR image] FIG. 17 is an example of a real image similar to the example of the image in FIG. 15, and shows a display example in which a user selects an AR image a11 in the gaze area E2 with a corresponding ID image b11 (=ID image "4"). AR images a1 and a11 are contained in the gaze area E2, and ID images "1" and "4" are displayed. An AR application displays an AR image a11 having store information. In response to the selection operation of the AR image a11, the AR application further provides detailed information about the store as a further AR image as a predetermined process.

[0115] Assume that the user wants to see detailed information about the store shown in the AR image a11. In this case, the user basically selects the AR image a11. At that time, the user can use the ID image b11 displayed for support instead of the AR image a11. In this example, the user selects the ID image b11. When using a voice command, the user inputs the ID number "4" shown in the ID image b11 (corresponding "select ID4" command) by voice. For example, the user inputs the voice "yonban". The HMD1 recognizes the voice and detects it as the "select ID4" command. The HMD1 interprets the command as a selection operation of the ID image b11 having the ID "4" and as a selection operation of the AR image a11 associated with the ID image b11 at that time.

[0116] In response to the selection operation of the ID image b11, the HMD1 performs display differentiation of the ID image b11 (FIG. 4, step S24). Note that correspondingly, display differentiation may also be performed for the target AR image a11. Furthermore, the HMD1 notifies the corresponding AR application of the selection of the AR image a11 (FIG. 5). As a result, the AR application places the AR image a11 in a selected state, and performs a predetermined process defined in response to the selection operation, which in this example is displaying detailed store information.

[0117] The AR application creates an AR image a11b including detailed information about the store, and displays it at the position of the AR image a11 or a position nearby. In this example, the detailed information about the AR image a11b includes information such as the type of "Store AAA" and business hours. In the AR application, an AR image may be composed of multiple AR images (e.g., multiple hierarchies or multiple pages). The HMD1 may assign and display ID images to the multiple AR images. Data output by the AR application may be audio output. Also, for example, when the AR application receives a predetermined user operation (e.g., a second selection operation) on the AR image a11 or the AR image a11b, the AR application erases the AR image a11b.

[0118] [Delete ID image] An example of control for erasing an ID image will be described. The HMD1 erases the ID image in the gaze area, in other words, switches it to hidden, in response to a predetermined operation by the user. An example of the image in FIG. 14 will be used for the description. In the gaze area E1, three ID images "1" to "3" are displayed corresponding to the AR images a1 to a3. If the user wants to erase the ID images in the gaze area E1 in this state, the user inputs, for example, the "ID display off" command of C12 in FIG. 9. In response to this, the HMD1 switches the ID images "1" to "3" in the gaze area E1 to a hidden state. In the hidden state of the ID image, the HMD1 disables the selection operation of the ID image. In the erased state of the ID image, the corresponding AR image is displayed, but the selection operation cannot be performed. For example, even if the user inputs the "Select ID1" command in this state, the ID image "1" and the AR image a1 cannot be selected.

[0119] Furthermore, if the user wishes to display the ID image again, for example, the user inputs the "ID display on" command of C11 in Fig. 9. In response to this, the HMD 1 switches the display state of the ID images "1" to "3" in the line of sight area E1.

[0120] As a modified example, the ID image and the corresponding AR image may be erased at the same time in response to a predetermined command.

[0121] [Select multiple ID images] A control example in which one or more ID images are selected from a plurality of ID images in the line of sight area will be described. First, in the example of the image in FIG. 14, the user can specify and select individual AR images for ID images "1" to "3" in the line of sight area E1 using individual commands (FIG. 9, C21 to C29). The user can also select a plurality of AR images collectively by an operation of successively specifying a plurality of ID images within a certain period of time. For example, when using the voice method, the user utters a plurality of ID numbers continuously within a certain period of time. During the voice recognition process, the HMD1 detects a plurality of words (corresponding voice units) within a certain period of time and interprets them as a command based on a connection of a plurality of words. The input voice is, for example, "ichi, san, go". This allows the ID images "1", "3", and "5" to be selected collectively. For example, when the user wants to select a range of ID images "2", "3", "4", and "5" with consecutive numbers, the user utters the range. The input voice is, for example, "ni, kara, go" (meaning 2 to 5). This allows the range to be selected collectively.

[0122] Alternatively, if the user wishes to select all ID images "1" to "3" within the line of sight area E1, the user inputs the "select all IDs" command of C30 in Fig. 9. In response to this, the HMD1 selects all ID images "1" to "3" within the line of sight area E1. This allows all three corresponding AR images a1 to a3 to be selected.

[0123] Alternatively, the user may select one ID image that is closest to the center of the line of sight in the line of sight area E1. In this case, the user inputs, for example, the "select 1 ID" command in C31 of FIG. 9. In response to this, the HMD1 selects the ID image that is closest to the center of the line of sight area E1, for example, the ID image "1". This allows only one corresponding AR image a1 to be selected.

[0124] [Zoom in / out of viewing area] FIG. 18 shows an example of control in the case where the gaze area is enlarged or reduced in response to a predetermined operation by the user. The example of the image in FIG. 18 is the same as that in FIG. 14. Assume that the gaze area is initially in a state of E1. If the user wishes to enlarge the gaze area, he / she inputs, for example, the "enlarge gaze area" command in C05 of FIG. 9. In response to this, the HMD1 enlarges and displays the gaze area E1. The enlarged gaze area E11 is shown. The gaze area E11 has a radius r11 that is larger than the original (r11>r1). Furthermore, if the user wishes to reduce the gaze area, he / she inputs, for example, the "reduce gaze area" command in C06 of FIG. 9. In response to this, the HMD1 reduces and displays the gaze area E1. The reduced gaze area E12 is shown. The gaze area E12 has a radius r12 that is smaller than the original (r12 <r1)。

[0125] In the case of the gaze area E11 after enlargement, the AR images a4 and a11 are newly included. In response to this, for example, the HMD1 assigns and displays the ID images "4" and "5" to the AR images a4 and a11 within the condition of the maximum number (for example, 5). Note that, if the maximum number is, for example, 3, the ID images "4" and "5" are not assigned and displayed to the AR images a4 and a11. In the case of the gaze area E12 after reduction, the AR images a2 and a3 are outside, and only the AR image a1 remains. In response to this, the HMD1 switches the ID images "2" and "3" of the AR images a2 and a3 to non-display. In this way, the number of ID images displayed on the display surface 2 can be changed by the operation of enlarging / reducing the gaze area. Since the number of ID images can be changed depending on the state seen by the user, it is easy for the user to select an ID image.

[0126] The HMD 1 of the modified example may use the following as another control example regarding the enlargement / reduction of the gaze area. In this control example, the HMD 1 does not enlarge / reduce the gaze area in response to a user operation as in FIG. 18, but enlarges / reduces the gaze area based on an automatic determination. It is assumed that the maximum number of ID images that can be simultaneously displayed in the gaze area is set in advance, for example, to three, based on a user setting. Note that a command for changing the setting of this maximum number may be provided. An example of the sound is "maximum three," etc.

[0127] In the example of FIG. 19A, a plurality of AR images (for example, AR images a1, a2, a3) are arranged in a relatively low density and at wide intervals on the display surface 2. In this case, the HMD 1 detects up to a maximum of three AR images in order of distance from the position p1 of the center of the line of sight based on the position p1. For example, the AR images are a1, a2, a3 in order of distance. The HMD 1 configures the line of sight area 20 with a size according to the arrangement state of these three AR images. In this example, a radius r1a that can include the three AR images a1 to a3 is taken, and a line of sight area E1a having the radius r1a is configured. Then, the HMD 1 assigns and displays the ID images "1" to "3" to the AR images a1 to a3 in the line of sight area E1a.

[0128] In the example of (B) of FIG. 19, a plurality of AR images (for example, AR images a1, a2, a3) are arranged in a relatively high density and at narrow intervals on the display surface 2. For example, assume that the state of the AR images changes from state (A) to state (B). In this case, the HMD1 similarly detects up to a maximum number of 3 AR images in order of distance from the position p1 of the center of the line of sight. The HMD1 configures the line of sight area 20 with a size according to the arrangement state of these three AR images. In this example, a radius r1b that can include the three AR images a1 to a3 is taken, and a line of sight area E1b having the radius r1b is configured. Then, the HMD1 assigns and displays the ID images "1" to "3" to the AR images a1 to a3 in the line of sight area E1b.

[0129] In this manner, in this modified example, the line-of-sight area is automatically adjusted according to the arrangement of the AR image on the display surface 2. This modified example is particularly effective in cases where an image representing the line-of-sight area is displayed.

[0130] As another control example, the shape of the gaze area may be changed using a predetermined command. For example, commands such as vertical enlargement, vertical reduction, horizontal enlargement, and horizontal reduction may be provided. The user can select the size, shape, and the like of the gaze area in advance on a display corresponding to the user setting screen provided by the HMD1.

[0131] [Fix gaze area] Fig. 20 shows an example of control related to fixing the gaze area and fixing the ID image within the gaze area. The user can temporarily fix the gaze area or release the fixation in response to a predetermined operation. Fixing the gaze area fixes the display of the ID image within the gaze area, and the display of the ID image can be maintained even if the gaze moves.

[0132] In the example of the image in FIG. 20, assume that the state is initially the gaze area E21. AR images a1 and a11 are contained within the gaze area E21 centered on position p21, and the corresponding ID images "1" and "2" are displayed. If the user wishes to fix the state of the ID images "1" and "2" in this gaze area E21, the user inputs, for example, the "fix gaze area" command C03 in FIG. 9. When this operation is received, the HMD1 fixes the state including the position and size of the gaze area E21 at that time. Together with this fixation, the states of the ID images "1" and "2" displayed within the gaze area E21 are also fixed.

[0133] After the above fixation, for example, suppose that the user moves the line of sight to the right, and the line of sight moves from position p21 to position p22, and from position p22 to position p23. At this time, if there is no fixation, the line of sight area E21 changes to line of sight areas E22 and E23. A new AR image a13 enters the line of sight area E23, and an ID image is assigned and displayed. On the other hand, if there is fixation as in this control example, the line of sight area E21 remains unchanged. Therefore, the ID images "1" and "2" of the AR images a1 and a11 remain displayed, and no ID image is assigned or displayed for the AR image a13. Even when the user's line of sight is at position p23, the user can select and operate the ID images "1" and "2".

[0134] Furthermore, when the user wishes to release the fixation of the gaze area 20, the user inputs, as a predetermined operation, for example, the "release gaze area fixation" command of C04 in FIG. 9. In response to this, the HMD1 releases the fixation of the gaze area E21. The HMD1 resets a new gaze area according to the gaze position present at the time of release. For example, if the gaze is at position p23 at the time of release, the gaze area E23 is configured. As a result, for example, an ID image "1" is assigned and displayed for the AR image a13 within the gaze area E23, and the ID image "1" or the like is erased for the AR image a1 etc. outside the gaze area E23.

[0135] [A deviation in line of sight] FIG. 21 shows an example of control in the case of deviation of the gaze area. When setting the gaze area in step S3 of FIG. 3 described above, the HMD 1 judges whether the gaze area deviates from the display surface 2. If the HMD 1 judges that the gaze area deviates, it resets the gaze area accordingly. Assume that the state is initially as shown in FIG. 21(A). In this state, AR images a1 to a3 are included in the gaze area E1, and ID images "1" to "3" are displayed. In addition, an AR image a11 corresponding to a store or the like is displayed in the display surface 2. Note that there is also a building to the left of the building shown in the AR image a11, but since information could not be acquired, a corresponding AR image has not been generated. In this state, the gaze area E1 does not deviate from the display surface 2.

[0136] Next, for example, the orientation of the HMD1 changes due to the movement of the user's head, resulting in the state of (B). The gaze is assumed to be looking at the same position p1. In this state, about half of the gaze area E1 is outside the area of ​​the display surface 2, and about a semicircular portion remains within the display surface 2. For example, when the position p1 of the center of the gaze goes outside the area of ​​the display surface 2, the HMD1 judges this state as a "deviation" of the gaze area. Note that in (B), for ease of explanation, an AR image corresponding to (A) is also illustrated outside the display surface 2. In this example, the gaze area deviates due to a change in the orientation of the HMD1, but the gaze area may also deviate due to a movement of the user's gaze.

[0137] If position p1 is within the display surface 2, the gaze area remains valid even if it is, for example, a semicircular portion. For AR image a1 within that gaze area, the display of ID image "1" is maintained. On the other hand, for AR images a2 and a3 that were within a semicircular portion outside the display surface 2, the corresponding ID images "2" and "3" are not displayed, of course. If position p1 moves outside the display surface 2, the HMD1 determines that the gaze area E1 is in an out-of-state state, and resets the ID images that were within that gaze area. For example, the HMD1 cancels the assignment of ID images "2" and "3" to AR images a2 and a3.

[0138] Furthermore, when the user's gaze position returns to within the display surface 2, the HMD 1 sets a gaze area according to that position. For example, when the gaze point moves from position p1 to position p5 in (B), a gaze area E5 is configured corresponding to the position p5. In this case, in response to the new AR image a11 entering the gaze area E5, for example, an ID image "2" is assigned to the AR image a11 and displayed.

[0139] Alternatively, the HMD 1 may determine whether the entire image area of ​​the line of sight area 20 has completely moved outside the display surface 2 .

[0140] [Effects etc. (1)] As described above, according to the video display device of the first embodiment, a gaze area is set according to the gaze direction in which the user focuses, and an ID image is displayed within the gaze area, thereby improving the operability and usability of the user with respect to the AR image. In particular, according to the first embodiment, the ID image display is limited within the gaze area, so that the amount of information on the display surface 2 as seen by the user is reduced, and the user can easily select an AR image using the ID. The user only needs to select from a small number of IDs, making the operation easy.

[0141] In addition, in the first embodiment, after the AR image is detected (step S2), a gaze area is set (step S3) and an ID image is displayed (step S5) as shown in Fig. 3. The user can set the gaze area while viewing and checking the AR image on the display surface 2, and can select and operate the ID image in the gaze area. This reduces the chance of missing an AR image on the display surface 2.

[0142] [Comparative Example] A comparative example of an HMD, which is a video display device, compared to the first embodiment will be described using a configuration example described in Patent Document 1. The HMD of the comparative example acquires one AR image from a list of AR images (=AR objects) displayed on the display surface, and if the AR image is a selectable AR image, assigns an ID number to it. Then, the HMD displays the image with the assigned ID number near the AR image. In the comparative example, an ID image is assigned to and displayed on the AR image regardless of the user's line of sight. Therefore, there may be a large number of AR images and corresponding ID images (corresponding numbers) on the display surface, or multiple ID images may be crowded together. In that case, it may be difficult for the user to select an ID image.

[0143] On the other hand, the HMD 1 of the first embodiment basically displays the ID image within the line of sight area even when there are many AR images on the display surface 2. Moreover, the HMD 1 of the first embodiment displays multiple ID images so that they are as easy to see as possible. This makes it easy for the user to select an ID image associated with a desired AR image. Moreover, the HMD 1 of the first embodiment can also operate the line of sight area using voice, thereby improving the operability of the ID image.

[0144] [Variation (1)] The following modifications of the HMD 1 of the first embodiment are also possible. In the first embodiment, the selection of an ID image includes, as a basic concept, the selection of an AR image and the execution of a process associated with the AR image. That is, when a user selects an ID image, a predetermined process associated with the AR image is automatically executed by a corresponding program. In the HMD of the modified embodiment, the selection of an ID image may be a concept that separates the selection of an AR image from the execution of an AR image process. In this case, when a user selects an ID image, the AR image associated with the ID image is selected by a corresponding program. Then, when a user performs a predetermined operation on the AR image in the selected state, a predetermined process associated with the AR image is executed by a corresponding program. In addition, the above ID image selection concepts may be mixed, and may be set to perform different operations when an ID image is selected depending on an AR application or a specific AR image.

[0145] [Variation (2)] The HMD of the modified example has a function of selecting and operating a plurality of ID images as one group. For example, there are cases where a user wants to select a plurality of desired AR images in the display surface 2 as one group, in other words, wants to select a plurality of AR images collectively. First, a group selection operation is possible by an operation of specifying a group ID using the group ID shown in FIG. 13 described above. In addition, by using the "select all IDs" command described above, all ID images in the line of sight can be selected collectively.

[0146] As another method, the user may, for example, input a predetermined "group selection" command, and then sequentially specify individual ID images. For example, when selecting ID images "1", "3", and "5" in the line of sight area, the input voice may be, for example, "select, one, three, five". Also, when AR image selection and AR image processing execution are separate, the desired AR image may be selected, and then the AR image processing may be executed by inputting a predetermined "execute" command.

[0147] (Embodiment 2) A video display device according to a second embodiment of the present invention will be described with reference to Figs. 22 to 26. The basic configuration of the second embodiment etc. is the same as that of the first embodiment, and the following describes the components of the second embodiment etc. that are different from the first embodiment. The HMD of the second embodiment provides an area (sometimes referred to as a "selection area" or an "identification information area") for selecting and manipulating an ID image within the display surface. The user can select and operate not only ID images displayed in areas other than the selection area within the display surface, but also ID images displayed within the selection area.

[0148] In the second embodiment, in the flow of Fig. 4 described above, the confirmation in step S23 includes confirmation of whether or not an ID image in the selection area is selected. Also, the determination of the gesture in steps S26 and S27 includes determination of the gesture for the ID image in the selection area.

[0149] [Selection area (1)] 22 shows an example of an image on the display surface 2 of the HMD 1 of the second embodiment. A selection area 60 related to an ID image is provided within the display surface 2. In this example, this selection area 60 is provided as a horizontally long rectangular area near the bottom side of the area of ​​the display surface 2, and is, for example, a transparent area. This selection area 60 may be invisible to the user, or may be visible to the user by displaying a selection area boundary line, etc.

[0150] As described above (FIG. 11), seven AR images, three AR images A21, A23, A24, A25, and A26, are contained within the line-of-sight area 20. These AR images are arranged relatively close to each other, making it difficult to perform a selection operation as is. When the control of the first embodiment is used, ID images "1" to "7" are assigned to the seven AR images and displayed. These ID images "1" to "7" are also arranged relatively close to each other, making it possible that selection operation as is.

[0151] Therefore, the HMD 1 of the second embodiment displays ID images corresponding to the seven AR images in the line of sight area 20 in the selection area 60. The HMD 1 arranges the ID images in the selection area 60 in a predetermined order. In this example, the seven ID images "1" to "7" are arranged in the selection area 60 at equal intervals from left to right in ascending order of numbers. For example, the ID images c1, c2, c3, . . . , c7 are arranged in order from the left side of the selection area 60. In this example, the ID images are displayed in the selection area 60 in a size larger than the ID images in the line of sight area 20. The ID images function like ID selection buttons.

[0152] The user can select and operate each ID image in the selection area 60. In this case, the operation input method is not limited to the above-mentioned voice method, but can also use a gesture method. For example, the user wants to select ID image c3 = ID image "3". In the case of the voice method, the ID image c3 can be selected by inputting the above-mentioned "select ID3" command, and one corresponding AR image A21 can be selected. In the case of using the gesture method, the user performs a virtual touch gesture with his / her finger at the position of the ID image c3. This gesture is, for example, moving the fingertip to a position overlapping the rectangular area of ​​the ID image "3" as seen from the user and keeping the fingertip at that position for a predetermined period of time or more. During this gesture, the user's fingers are on the outside in front of the display surface 2 of the HMD 1, and are visible through the display surface 2 and the ID image. Therefore, the user and the camera 5 can grasp the state of the fingers.

[0153] The HMD1 detects the gesture based on the recognition of the image of the camera 5, and interprets it as a selection operation of the ID image c3. Detection of an ID image selection operation by this gesture is easier and more accurate than when a conventional HMD detects gestures for AR images and ID images at various positions on the display surface. In the second embodiment, the gesture is easily detected because it is sufficient to determine the gesture for the position of the ID image in a predetermined selection area 60. Even if the AR image or ID image moves on the display surface 2, the ID image is displayed at a fixed position in the selection area 60, so detection is easy.

[0154] In the control example of FIG. 22, the ID images are displayed in both the line-of-sight area 20 and the selection area 60. For example, for the AR image A21, the ID image "1" is displayed doubly in the line-of-sight area 20 and the selection area 60. As a modified example, the ID image may be displayed only in the selection area 60. In this case, however, it is necessary to make the correspondence between the AR image and the ID image clear. For example, the correspondence can be made clear by displaying a reduced image of the AR image together with the ID image, or by displaying the AR image and the ID image connected by a connecting line.

[0155] As another operation input method, the operation input unit 160 of the HMD 1 may be used. For example, the user can select an ID image in the selection area 60 by pressing a predetermined hardware button provided on the main body device or an operation device.

[0156] [Selection area (2)] As a modification of the second embodiment, in the selection area 60, together with the ID image, a reduced image of the corresponding AR image or the like may be displayed in association with the ID image as a set.

[0157] FIG. 23 shows a display example in a modified example. AR images a1, a2, and a11 are included in the line-of-sight area 20. The HMD 1 arranges the ID images "1", "2", and "3" associated with the AR images a1, a2, and a11 in the line-of-sight area 20 in the selection area 60 at positions (left, center, and right) that are distributed and aligned at equal intervals k1. Furthermore, the HMD 1 arranges the reduced images of each of the AR images a1 to a3 in pairs with each ID image. For example, in the selection area 60, the ID image "1" of the AR image a1 and a rectangular image including the reduced image of the AR image a1 are displayed as a pair at the left side position. For example, the ID image "2" of the AR image a11 and a rectangular image including the reduced image of the AR image a11 are displayed as a pair at the center position. The selection area 60 may display not only the reduced images of the AR images, but also partial images obtained by cutting out a part of the AR image, images in which some characters are enlarged, and the like.

[0158] Since the reduced image of an AR image is an image that makes it easy to understand its correspondence with the original AR image, the user can easily select an ID image in a manner similar to directly operating the original AR image.

[0159] The ID images in the selection area 60 can also be identified by their positions (e.g., left, center, right). The HMD 1 of the second embodiment can also select an ID image in the selection area 60 using a position-designated command. Examples of voice commands include a "left ID selection" command, a "right ID selection" command, and a "center ID selection" command. The "left ID selection" command is a command to select an ID image located on the left side of the ID images in the selection area 60, and the voice is, for example, "left". The "right ID selection" command is a command to select an ID image located on the right side of the selection area 60, and the voice is, for example, "right". The "center ID selection" is a command to select an ID image located in the center of the selection area 60, and the voice is, for example, "center".

[0160] As another example of the arrangement in the selection area 60, in the case of one ID image, one ID image is displayed in the center of the selection area 60, and in the case of two ID images, two ID images are displayed on the left and right sides of the selection area 60. In these cases, it is possible to select individual ID images using a predetermined command in the same way. Note that this example is an example in which numbers are arranged in ascending order from the left side in the selection area 60, but this is not limiting.

[0161] As a modified example, the ID image in the selection area 60 and the rectangular image including the reduced image of the AR image may have different colors or shapes for identification. In this case, the user can select and distinguish each ID image or AR image by using a command such as a position designation, a color designation, or a shape designation. When providing a distinction in color or shape, it may be either one or both of the AR image and the ID image. For example, a different color is set for the ID image (or the reduced image of the AR image) to be displayed for each position (e.g., left, center, right) in the selection area 60. For example, the left position in the selection area 60 is set to red, the center position to blue, and the right position to yellow. The user can select the ID image in the left position by inputting, for example, a "red ID selection" command (e.g., voice "red").

[0162] Also, for example, a different shape is set for each position within the selection area 60. For example, the left position within the selection area 60 is set to a circle, the center position to a square, and the right position to a triangle. The user can select the ID image at the right position by inputting, for example, a "select triangle ID" command (for example, the voice "triangle").

[0163] As a modified example, the display of the ID image in the selection area 60 may be omitted, and only a reduced image of the AR image may be displayed.

[0164] Alternatively, multiple ID images may be arranged in the selection area 60, concentrated at any one of the left, center, or right positions.

[0165] As a modified example, in the ID configuration of the ID image, it is possible to apply a form other than a number, for example, a color, a shape, etc. The ID image is not limited to a number form as long as the user can specify it in a distinguishable manner. In the case of an ID image other than a number, the user can select the ID image using a predetermined command by a method such as position specification, color specification, shape specification, etc.

[0166] As a variant, when there is a real image of an object that is the basis for generating the AR image, an image of that object, a reduced image of that object image, a partial image of that object image, etc. may be used as the ID image or in combination with other images.

[0167] [Selection area (3)] As a modification of the second embodiment, when an ID image is arranged and displayed within the selection area 60, the ID image may be arranged and displayed at a position that matches as closely as possible with the position of the corresponding AR image.

[0168] FIG. 24 shows two display examples in this modified example. When displaying ID images "1" to "3" related to AR images a1, a2, and a11 in the line of sight area 20 in the selection area 60 at the bottom side, the HMD 1 arranges the corresponding ID images at positions that correspond as much as possible to the positions of the AR images in the x direction (horizontal direction) of the display surface 2. In this example, the ID image "2" of AR image a11, the ID image "1" of AR image a1, and the ID image "3" of AR image a2 are arranged in order from the left in the selection area 60. In this case, the ID numbers are not arranged in order, but the x direction position indicates the correspondence with the AR images, making it easy for the user to perform a selection operation.

[0169] As another display example, the HMD 1 may provide the selection area 60 at another position in the display surface 2. The selection area 60b is provided near the right side of the display surface 2. When the HMD 1 displays the ID images "1" to "3" related to the AR images a1, a2, and a11 in the line-of-sight area 20 in the selection area 60b on the right side, the HMD 1 arranges the corresponding ID images at positions that correspond as much as possible to the positions of the AR images in the y direction (vertical direction) of the display surface 2. In addition, when it is desired to make the correspondence between the AR image and the ID image easier to understand, a connecting line as indicated by a dashed arrow may be displayed. Similarly, the selection area 60 may be provided on the upper side or left side of the display surface 2. The position at which the selection area 60 is provided on the display surface 2 may be determined as a variable position according to the position and movement of the user's line of sight.

[0170] [Effects etc. (2)] As described above, according to the video display device of the second embodiment, the selection area 60 is used to improve the operability and usability of the user with respect to the AR image. In particular, according to the second embodiment, since the ID image is displayed in the selection area 60, the selection operation of the ID image is easy. Even when the gesture method is used, the detection accuracy is improved compared to the conventional art example, and the selection operation of the ID image is easy. In addition, there is a possibility that various AR images are mixed in the multiple AR images displayed in the display surface 2. Each AR image may have various sizes and shapes. In the second embodiment, even if there are various AR images, ID numbers are assigned, the images are aligned to a certain size, and the information is displayed in an organized manner in the selection area 60. Therefore, the user can easily select the ID image.

[0171] [Variation (1)] The following modification of the second embodiment is also possible. A control example using the selection area 60 by the HMD 1 of the modification is described below. In this control example, the HMD 1 arranges and displays an ID image for an AR image that has gone outside the line-of-sight area 20 within the selection area 60 according to a predetermined rule.

[0172] FIG. 25 shows a display example in this control example. In this example, the range of IDs to be used is "1" to "9", and the maximum number of IDs to be simultaneously displayed in the gaze area 20 is 5. First, the user's gaze is in the gaze area E21. In the gaze area E21, AR images a4, a5, a12, and a13 are included, and corresponding ID images "1" to "4", for example, are displayed. Next, the gaze moves, for example, to the left, and the gaze area E22 is entered. In the gaze area E22, AR images a1, a2, a3, and a4 are included, and corresponding ID images "4" to "7", for example, are displayed. For the AR images a5, a12, and a13 that have gone outside the gaze area E22, when the control of the first embodiment is used, the ID images "1" to "3" are hidden.

[0173] The HMD 1 of this modification hides the corresponding ID images "1" to "3" for the AR images a5, a12, and a13 that are outside the line of sight area E22, and displays the corresponding ID images c1 to c3 (ID images "1" to "3") within the selection area 60. The HMD 1 arranges the three ID images c1 to c3 in aligned positions (right, center, left) within the selection area 60, distributing them at equal intervals k1.

[0174] In this state, the user can select each of the ID images c1 to c3 in the selection area 60. This allows the user to select an AR image a5 outside the line-of-sight area E22. For example, when the user selects the ID image c1=ID image "1" in the selection area 60, the corresponding AR image a5 can be selected.

[0175] Furthermore, suppose that the line of sight moves to the left and the state becomes line of sight area E23. AR images a1 and a11 are in line of sight area E23, and ID images "7" and "8" are displayed for AR images a1 and a11. In this modified example, the HMD1 hides the corresponding ID images "5", "6", and "4" for AR images a2, a3, and a4 that are outside line of sight area E23, and additionally displays the corresponding ID images "5", "6", and "4" in the selection area 60. That is, the ID images "1" to "6" are displayed in a regular arrangement in the selection area 60 as in the example of FIG. 22. Similarly, in this state, the user can select and operate each ID image in the selection area 60. Note that a maximum number may also be set for the number of ID images to be displayed in the selection area 60. For example, when the maximum number is 3, the ID images "4" to "6" are displayed in the selection area 60, instead of the ID images "1" to "6".

[0176] As a modified example, the HMD 1 may display, as a history, the ID images corresponding to the AR images that have gone out of the line-of-sight area 20 in the selection area 60 in the order in which they went out of the line of sight area 20 in time series. In this selection area 60, up to a predetermined number of ID images are displayed adjacent to each other, and when the predetermined number is exceeded, the oldest ID images are deleted. In the example of FIG. 25, when the order of the AR images that have gone out of the line-of-sight area is AR images a13, a12, a5, a4, a3, and a2, the ID images "3", "2", "1", "4", "6", and "5" are displayed in the selection area 60 in that order.

[0177] [Variation (2)] In a modified example of the HMD 1, when the line of sight area deviates from the display surface 2 as in the example of FIG. 21 described above, the ID image that was in the deviated line of sight area may be displayed in the selection area 60. FIG. 26 shows an example of display at the time of deviation. When the HMD 1 determines that the line of sight area E1 has deviated from the display surface 2, the ID image of the AR image that was in the line of sight area E1 but is no longer visible is displayed in the selection area 60. For example, the ID image "2" of the AR image a2 and the ID image "3" of the AR image a3 that were in the line of sight area E1 are arranged near the right side of the selection area 60. In this example, since deviation occurred on the right side of the display surface 2, the ID image is displayed at a position close to the right side in the selection area 60. The ID image displayed at this time may be a reduced image of the AR image, as described above. This modified example can also be effectively used for ID images that have gone outside the display surface 2.

[0178] (Embodiment 3) An image display device according to a third embodiment of the present invention will be described with reference to Fig. 27. The HMD according to the third embodiment differs from the flow in Fig. 3 in the order of processing, with the main change being that step S2 and step S3 are reversed. After setting a line-of-sight area, the HMD according to the third embodiment extracts and displays an AR image within the line-of-sight area, and assigns and displays an ID image.

[0179] FIG. 27 shows a flow in the third embodiment. In step S1, the HMD1 detects the position of the HMD1, etc. In step S2b, the HMD1 detects the gaze direction and sets a gaze area. In step S3b, the HMD1 generates an AR image and displays the AR image in the gaze area if it falls within the gaze area. In step S4b, the HMD1 detects an AR image that falls within the gaze area. In step S5, the HMD1 assigns an ID image to the AR image in the gaze area and displays it. In the third embodiment, with such a flow configuration, the AR image and ID image are displayed limited to within the gaze area.

[0180] [Example] An example of displaying an image in the HMD1 of the third embodiment is as follows. The example of FIG. 14 described above is used. In FIG. 14, the AR image is also displayed outside the line of sight area E1, but in the third embodiment, the AR image and the ID image are displayed only in the line of sight area E1. For example, the AR images a1, a2, and a3 are displayed because their display positions correspond to the line of sight area E1, and the ID images "1", "2", and "3" are displayed. The AR images a4, a5, a11, a12, and a13 are not displayed because their display positions are outside the line of sight area E1, and the ID images are not displayed either. Note that if the line of sight area E1 moves and the display positions are in a corresponding state, the AR image and the ID image are displayed.

[0181] [Effects etc. (3)] According to the third embodiment, the number of AR images and ID images displayed on the display surface 2 is reduced in accordance with the line of sight, so the user can easily perform a selection operation by focusing on the AR images and ID images in the line of sight area. In the third embodiment, there is no need to display all the AR images on the display surface 2, and there is an advantage in that processing efficiency is improved.

[0182] (Embodiment 4) An image display device according to a fourth embodiment of the present invention will be described with reference to Fig. 28. The HMD according to the fourth embodiment continues to display an ID image when the AR image in the line-of-sight area moves outside.

[0183] [Example] FIG. 28 shows an example of an image in the fourth embodiment. (A) shows a first state, and (B) shows a second state. The real image in FIG. 28 is an example of a location near an intersection. In this example, a person is treated as a target for displaying an AR image. An AR application detects a person's face and provides information about that person as an AR image. For example, there are three people, such as person A, person B, and person C, on the display surface 2 of the image in (A). Person A and person C are ordinary people unknown to the user, and person B is an acquaintance of the user. The faces and information of people acquaintances of the user, including person B, are registered in advance in the AR application.

[0184] The AR application extracts a person's face area from the image of the camera 5 by known image processing. For example, the face areas of person A, person B, and person C are extracted. The AR application recognizes and determines the face of a specific person (for example, person B) based on image processing such as feature identification. In other words, the AR application performs personal recognition processing. In addition, at that time, for example, registered information of the face image of person B is used. For example, when the AR application detects the face of person B, it displays an AR image for the face area of ​​person B. This AR image is an image that indicates that the person is a specific person (for example, person B) such as an acquaintance of the user. In this example, the AR image is a frame image that surrounds the face area, such as the AR image a41. This AR image is not limited to this, and may be a frame image that surrounds a body, a speech bubble image, a mark according to a person, or the like.

[0185] In FIG. 28, for the purpose of explanation, the vicinity of person B's face is illustrated in an enlarged manner. In this enlarged view, there is a frame image as an AR image a41 around person B's face. This AR image a41 may also be accompanied by a text image of information such as person B's name. When this AR image a41 is selected and executed, the AR application may display registered information of person B. When this AR image a41 is selected and executed, the AR application may also display the information of person B enlarged using a balloon image or the like.

[0186] In this example, an AR image a42 is displayed in the video of Fig. 28 as an example of an AR image by another AR application. The AR image a42 is an arrow image for navigation such as route guidance. The AR image a42 indicates the direction in which the user should proceed.

[0187] The HMD 1 of the third embodiment assigns and displays the ID images b41 and b42 in association with the AR images a41 and a42 that are within the line-of-sight area E41. For example, the ID image b41 has an ID of "1" and the ID image b42 has an ID of "2."

[0188] Assume that time has passed from the first state (A), and for example, person B has moved to the second state (B). Assume that the line of sight area E41 is in almost the same position. If the basic control of the first embodiment is applied, it will be as follows. In the second state (B), since the AR image a41 is outside the line of sight area E41, the "1" of the ID image b41 is not displayed. In this case, the selection operation of the ID image b41 is not possible. When tracking the AR image a41 of person B, for example, the user moves the line of sight to move the line of sight area and captures the AR image a41 of the moving person. In this case, the ID image b41 of the AR image a41 of person B is also displayed, so the selection operation of the ID image b41 is possible. However, for example, when there are multiple objects or AR images in the line of sight area and each of them moves, or when focusing on some of the AR images, some of the AR images may move and go outside the line of sight area like person B, and the user may not be able to track them.

[0189] In contrast, in the second state, the HMD 1 of the fourth embodiment continues to display the ID "1" of the ID image b41 even for the AR image a41 that has moved outside the line-of-sight area E41. In this case, the AR image a41 and the ID image b41 of person B continue to be displayed even if the user does not move his / her line of sight. As long as person B is present on the display surface 2, the user can track the AR image a41 and the ID image b41 and can perform a selection operation of the ID image b41.

[0190] The control of the fourth embodiment can be set in advance by the user so as to be applied to a specific AR application or a specific type of AR image. Alternatively, this control may be switched to an applied state when the user performs a predetermined operation (for example, an "ID tracking" command). In addition, a form may be combined with the second embodiment, and an ID image of the AR image to be tracked may be displayed within the selection area.

[0191] As another example, the AR image a42 of an arrow is as follows. An AR application sequentially displays a plurality of AR images such as the AR image a42 and the AR image a43 for navigation to a destination. For example, the AR image a42 is displayed at a certain point in time first, and the AR image a43 is displayed at a certain point in time next. A method in which a plurality of AR images are automatically displayed sequentially may be used, or a method in which the user selects the first AR image a42 and the next AR image a43 is displayed may be used. In this way, a plurality of related AR images may be displayed at different positions. In that case, at least some of the AR images may go outside the line of sight area 20. For example, the AR image a43 is outside the line of sight area E41.

[0192] In such a case, the HMD1 displays ID images for multiple related AR images even if they are outside the line of sight area. For example, ID image b42 is displayed for AR image a42, and ID image b43 is displayed for AR image a43. In this example, the ID of ID image b43 is set to "2", the same as the ID of ID image b42. The same applies when there are three or more related AR images.

[0193] [Effects etc. (4)] According to the fourth embodiment, the ID image can be continuously displayed in accordance with the movements of the object, the AR image, and the ID image, so that the user can track and operate the AR image that has moved out of the line of sight area. The control of the fourth embodiment is applicable not only to the case where the object moves. For example, it is also applicable to the case where the object and the AR image are stationary and the line of sight area moves.

[0194] (Embodiment 5) An image display device according to a fifth embodiment of the present invention will be described with reference to Fig. 29. The HMD according to the fifth embodiment realizes control of an ID image by using the line-of-sight depth of a user's line of sight.

[0195] [Line-of-sight depth] FIG. 29 shows a schematic diagram of the concept of gaze depth in the fifth embodiment. (A) shows an example of the size according to the position of an object (e.g. a person) in a real image. (B) shows the gaze depth range according to the position of an object in the depth direction from the position of the user's eyes (= viewpoint position). The gaze detection unit 4 of the HMD1 in the fifth embodiment has a function of detecting gaze depth. The gaze detection unit 4 detects the gaze direction of the left eye and the gaze direction of the right eye as the gaze directions of both eyes of the user. The gaze detection unit 4 can detect the gaze depth from the point where the gaze direction of the left eye and the gaze direction of the right eye intersect.

[0196] When objects are present at different positions in the depth direction as seen by the user, as in the example of Fig. 28 described above, the sizes of the objects are different, as in the example of the image of (A) of Fig. 29. In the image of (A), among three people (person A, person B, person C), person A is relatively close to the HMD 1 and has a shallow gaze depth. Person C is relatively far from the HMD 1 and has a deep gaze depth, and person B is at a distance about intermediate between person A and person C, and has a gaze depth about intermediate between them.

[0197] In (B), the gaze directions from the left eye and right eye of the user are indicated by dashed lines. The point where the two gaze directions intersect corresponds to the gaze depth. The gaze depth range is the range of the effective gaze depth. In (B), the vertical axis indicates the distance between the HMD1 and the target person, and includes the distance ZA to person A, the distance ZB to person B, and the distance ZC to person C. As the gaze depth range according to the distance, the gaze depth ranges HA, HB, and HC are indicated. The HMD1 sets the gaze area 20 in accordance with the detected gaze direction and gaze depth range. For example, when the detected gaze depth range is the gaze depth range HB, the gaze area E60 is set in the image of (A). In this case, for example, the ID image of the AR image corresponding to person B can be easily selected.

[0198] [Control example] As a control example in the fifth embodiment, the HMD1 may automatically adjust the size of the gaze area, etc., according to the gaze depth. The left side of the gaze depth range in FIG. 29 shows a control example of the size of the gaze area in this control example. For example, the HMD1 sets the size (for example, radius) of the gaze area to a level selected from three levels, large, medium, and small, according to the determination of the three levels of the gaze depth, large, medium, and small. In this control example, a gaze area of ​​a suitable size is set according to the state of the gaze direction and gaze depth of the user. The AR image and ID image in the gaze area are determined according to the size of the gaze area. Even if the gaze direction of the user is the same, gaze areas of different sizes are set according to the difference in the gaze depth. As a result, the ID images to be selected are different. For example, in the video of FIG. 14, it is assumed that the base positions of the AR images a2 and a4 are in a position in the back direction compared to the AR image a1. In contrast to the case where the user focuses on the AR image a1, the gaze depth becomes deeper when the user focuses on the AR image a2, etc. In response to this, the HMD1 sets a gaze area 20 of a smaller size. This further narrows down the AR images that fall within the line-of-sight area 20. As a result, the number of ID images displayed is reduced, and the ID image selection operation can be performed in a more narrowed-down state.

[0199] [Effects etc. (5)] According to the fifth embodiment, the gaze state in the depth direction in the three-dimensional space is taken into account as the gaze depth, so that the gaze area and the ID image can be controlled in more detail. For example, the user can easily narrow down the ID images to be selected according to the gaze focus.

[0200] (Embodiment 6) A video display device according to a sixth embodiment of the present invention will be described with reference to Fig. 30. As a control example related to a specific AR application, the HMD according to the sixth embodiment selects and operates a plurality of AR images associated with an object by a plurality of ID images, thereby realizing a predetermined function.

[0201] [Display example (1)] FIG. 30A shows an example of an image in the sixth embodiment. This real image shows a triangular table in a corner of a room as an object 151. The HMD1 in the fifth embodiment grasps the position of the HMD1 in the room using the sensor unit 150. The HMD1 measures the distance between the HMD1 and a wall or floor, and the distance between the HMD1 and the object 151, for example, using the camera 5 or a distance sensor of the sensor unit 150. The HMD1 grasps the positions of various objects in the room in a three-dimensional space, and grasps the positional relationship with the HMD1. As the distance sensor, for example, a known method such as a TOF (Time Of Flight) method, a pattern irradiation method using a laser, or a method of calculation using multiple cameras can be applied.

[0202] The HMD 1 of the fifth embodiment, for example, an AR application, captures an image of a space such as a room using the camera 5, and generates an AR image using the captured image. This AR application has a function of virtually placing an object (corresponding AR object) in a room in response to, for example, a user's operation, and can be used for simulating an interior layout, for example.

[0203] The HMD 1 places and displays an AR object 152 designated by the user on, for example, a triangular table object 151 located within the line of sight area 20. The AR object 152 is, for example, an AR image representing a cylindrical object. This AR object 152 can be various images such as a vase or a model, depending on the AR app or the user's designation. If the AR object 152 falls within the line of sight area 20, the HMD 1 assigns and displays an ID image 153 to the AR object 152.

[0204] Furthermore, the AR application moves and displays the AR object 152 in response to a predetermined operation (e.g., a gesture) of the user. That is, the user can place the AR object 152 at a desired position. The HMD 1 also moves the corresponding ID image 153 in response to the movement of the AR object 152. As another example, the AR object 154 is a rectangular image representing a painting, a window, or the like, placed on a wall.

[0205] For example, when the ID image 153 of the AR object 152 is selected, the HMD 1 executes a predetermined process associated with the AR object 152. This process can be various processes such as putting the AR object 152 in a selected state, turning on / off the display of the AR object 152, etc. For example, there are cases where it is difficult to move the AR object 152 by a gesture such as a direct touch or slide. In this case, the user selects the ID image 153 by voice input to put the AR object 152 in a selected state. Then, the user can move the AR object 152 by using a predetermined command provided by the AR application (for example, commands such as "move right" and "move left").

[0206] [Display example (2)] (B) of FIG. 30 shows another example of an image. The AR application of the HMD 1 can add and display a plurality of AR objects for an object 151 such as a triangular table. One AR application has a function of measuring, for example, the length of an object. The HMD 1 detects edges and feature points of the object 151 based on image recognition by the camera 5. This allows, for example, a left corner point p51, a right corner point p52, and a back point p53 of the upper surface of the object 151 to be detected. For example, the HMD 1 displays AR objects a51 and a52 for the two left and right corner points p51 and p52. The AR object a51 is an arrow image representing the left corner point, and the AR object a52 is an arrow image representing the right corner point. The HMD 1 then assigns and displays ID images to each of the AR objects a51 and a52. For example, the ID image "2" is displayed as the ID image b51 for the AR object a51, and the ID image "3" is displayed as the ID image b52 for the AR object a52.

[0207] A user can select multiple AR objects by selecting ID images in the line of sight area 20. In this example, the user uses the measurement function of this AR application to measure the length between two points of the object 151. To do so, the user selects two ID images b51 and b52 corresponding to the two points. The operation is, for example, input of a "select ID2" command and a "select ID3" command. This allows the two ID images b51 and b52 to be in a selected state. When multiple AR images are in a selected state, the AR application grasps the relationship between those AR images and executes a predetermined process for the multiple AR images based on the relationship. In this example, the AR application calculates the distance between the selected point p51 and point p52 (the length of the straight line between the two points). The AR application may display an AR image of the straight line between the two points. The AR application displays information on the calculated distance as, for example, the AR image 155. Similarly, the AR application can also measure the area of ​​the area between three points according to the designation of three points of an object.

[0208] [Effects etc. (6)] According to the sixth embodiment, it is possible to effectively support the operation input of an AR application that handles a plurality of AR images as in the above example. Even when the AR image is small or when fine operation is required, the operation using the ID image is possible.

[0209] (Embodiment 7) A video display device according to a seventh embodiment of the present invention will be described with reference to Fig. 31. In the seventh embodiment, in a specific AR application, an ID image is directly assigned to a real object, and the object is displayed, enabling virtual operation of the object.

[0210] FIG. 31 shows an example of an image in the seventh embodiment. This real image is taken indoors, and there are multiple (e.g., two) objects OB1 and OB2, such as chairs and tables. The HMD1 distinguishes and detects image areas R1 and R2 of the multiple objects OB1 and OB2 on the display surface 2 based on the recognition of the image from the camera 5. The HMD1 determines the object image area included in the line of sight area 20, and assigns and displays an ID image to the object image area included in the line of sight area 20. For example, an ID image "1" is assigned to and displayed in the area R1 of the object OB1.

[0211] The user selects a desired ID image corresponding to a desired object. For example, a command to select the ID image "1" associated with the object OB1 is input by voice. The HMD1 selects the object image area of ​​the object associated with the selected ID image. The corresponding AR application executes a predetermined process on the object image area of ​​the selected object. An example of the process of the AR application is as follows.

[0212] The AR application of the first example acquires information about the object from the selected object image area, for example, by searching the Internet. The AR application displays the acquired information as an AR image on the display surface 2. For example, if the object is a product, information about the product can be displayed.

[0213] The AR application of the second example is another example of an application that simulates the layout of a room, similar to the fifth embodiment. This AR application virtually moves a selected object image area in response to a predetermined operation of the user. The predetermined operation may be, for example, a sliding gesture of a finger or a voice command to move the object. During this movement, the program makes the object image area invisible by superimposing an image such as a filled-in image on the real object image area, or displays it in a manner that makes it clear that it is the object from which it was moved. The AR application displays the object image area while moving it. This makes it appear to the user that the object has been virtually moved. Thus, the user can simulate the placement of the object, etc.

[0214] FIG. 31 (A) shows a state before the virtual movement of the object OB1, and (B) shows a state after the virtual movement of the object OB1. The user moves the object image area of ​​the object OB1 in (A) to the right. The gaze area also moves to the right. In the state before the movement, the AR application assigns and displays an ID image "1" to the object OB1 in the gaze area E71. The AR application may display a frame image representing the area R1 as an AR image. The user selects the ID image "1" associated with the object OB1 and virtually moves the object OB1 to the right with a predetermined operation. The HMD1 displays a filled-in image g1 in the area R1 of the object OB1 before the movement, so that the object OB1 does not appear to exist after the movement. It is more preferable to perform image processing on the filled-in image g1 to virtually make it appear as the background using a surrounding image. The HMD1 displays an image g2 representing the object OB1 in the area R1b after the movement. Image g2 is a type of AR image, and is an AR image corresponding to the image area of ​​the object. HMD1 maintains the display of ID image "1" for image g2. Note that image g2 of area R1b after movement may be the same as the image of object OB1 before movement, or more preferably, an image in which the orientation, size, etc. of the object are adjusted by image processing.

[0215] [Effects etc. (7)] According to the seventh embodiment, it is possible to virtually operate a real object using an ID image.

[0216] [Note] (1) The image display device automatically enlarges or reduces the gaze area depending on the number or density of generated images within the gaze area. (2) The image display device shall maintain an ID image containing the same identification information for the same generated image in chronological order as much as possible. (3) The image display device reassigns an ID image, including identification information, to the generated image within the line of sight area at each point in time according to a predetermined rule. (4) When a plurality of generated images are densely packed as a generated image, the image display device displays the ID image in a sparsely packed state. (5) When a group of multiple generated images is formed as a generated image, the video display device assigns an ID image to each group and displays them, and selects multiple generated images of the group in response to a selection operation of the group's ID image.

[0217] The present invention has been specifically described above based on the embodiments, but the present invention is not limited to the above-mentioned embodiments and can be modified in various ways without departing from the gist of the invention. Combinations of the embodiments and configurations in which components of the embodiments are added, deleted, or replaced are also possible. The image display device of the embodiment has been described as being applied to an HMD, but the present invention is not limited to this and can be applied to any device that has the function of superimposing a generated image on a real image, and can also be applied to a HUD device or the like. The program processing and information processing described in the embodiment may be implemented by general-purpose software program processing, or may be implemented by dedicated hardware circuits such as FPGA and ASIC. The data of the program may be stored in a computer-readable storage medium such as a disk or memory card. The program may be stored at the time of product shipment, or may be stored in a server or DB on a communication network. The data structure is not limited to a table structure and can be applied. The above-mentioned identification information (ID) can be replaced with a similar expression such as an identifier. [Explanation of symbols]

[0218] 1...HMD, 2...display surface, 3...speaker, 4...gaze detection unit, 5...camera.

Claims

1. a display device for displaying the generated image; a control device that sets a selection area to assist a user in performing an operation on the generated image; A gaze detection device for detecting a gaze direction of the user; Equipped with The control device sets a line-of-sight area according to the detected line-of-sight direction, When the user visually recognizes the displayed generated image, if the generated image falls within the line of sight area, The control device controls to display, in the selection area, a selection image associated with the generated image corresponding to the line of sight area; When a selection operation by the user is received for the selection image, a predetermined process associated with the generated image related to the selection image is executed.

1. A video display system comprising:

2. 2. The image display system according to claim 1, The selection image is a reduced or enlarged image of the generated image.

1. A video display system comprising:

3. 2. The image display system according to claim 1, The selection image is an image including a part or the whole of the generated image.

1. A video display system comprising:

4. 2. The image display system according to claim 1, The control device creates an ID image including identification information for supporting a user's operation on the generated image, and when the generated image falls within the line of sight area, displays the ID image in association with the generated image within the line of sight area and the selection image within the selection area.

1. A video display system comprising:

5. 2. The image display system according to claim 1, The control device expands or reduces the line-of-sight area in response to an operation by the user.

1. A video display system comprising:

6. 6. The image display system according to claim 5, The display device is capable of displaying a plurality of the generated images, The control device sets a maximum number of the generated images that can be simultaneously displayed within the line of sight area, and expands or reduces the line of sight area in accordance with the maximum number.

1. A video display system comprising:

7. 2. The image display system according to claim 1, The control device provides the selection area on the top, bottom, right or left side of the display surface.

1. A video display system comprising:

8. 5. The image display system according to claim 4, The control device controls the ID image and the selection image in the selection area to be displayed at positions corresponding to the display position of the generated image.

1. A video display system comprising:

9. 7. The image display system according to claim 6, The control device controls the display device to set the selection area at a position separated from the position at which the generated image is displayed, and to display a line indicating the correspondence between the generated image and the selection image.

1. A video display system comprising:

10. 9. The image display system according to claim 8, The control device controls the display of the ID image and the selection image at positions corresponding to the positions of the generated image in the left-right or up-down direction of the display screen.

1. A video display system comprising:

11. displaying the generated image; setting a selection area for assisting a user in performing an operation on the generated image; Detecting a gaze direction of the user and setting a gaze area according to the detected gaze direction; When the user visually recognizes the displayed generated image, if the generated image falls within the line of sight area, A step of controlling to display a selection image related to the generated image corresponding to the line of sight area in the selection area; When a selection operation by the user is received on the selection image, executing a predetermined process associated with the generated image related to the selection image; The image display method includes:

Images