Video display system and method

Abstract

To provide a technique capable of improving the operability and usability for a user in relation to a generated image such as augmented reality (AR), with respect to a video display device.SOLUTION: An HMD serving as a video display device displays an AR image 30 on a display screen 2, configures an ID image 40 including identification information (ID) for supporting an operation of a user to the AR image 30, detects a sight line direction of the user to set a sight line region 20, and displays the ID image 40 so as to be associated with the AR image 30 when the AR image 30 falls within a range of the sight line region 20. The HMD receives an operation to select the ID image 40 from the user, selects the AR image 30 associated with the ID image 40 when receiving the operation to select the ID image 40, and executes predetermined processing associated with the AR image 30.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present invention relates to the technology of video display devices, and more particularly to a technology for displaying a computer-generated image (which may also be referred to as a virtual image) with respect to a video of the real space (which may be referred to as a real image).

Background Art

[0002] The development of technologies for displaying generated images such as augmented reality (AR), virtual reality (VR), and mixed reality (MR) for real objects such as real images is underway. The generated image includes a still image or a moving image. Examples of video display devices capable of displaying the generated image include a head-mounted display (HMD, head-mounted display device) and a head-up display (HUD). For example, an HMD having an AR function superimposes and displays an AR image (which may also be referred to as an AR object or the like), which is a generated image, on a real image on a transmissive or non-transmissive display surface. The user selects and operates a target AR image from a plurality of AR images on the display surface. The HMD executes a predetermined process associated with the selected AR image.

[0003] As a prior art example related to the above video display device, Japanese Patent Application Laid-Open No. 2016-218868 (Patent Document 1) can be cited. Patent Document 1 describes that in a display control method or the like, the selectivity of the displayed object data is improved. Patent Document 1 describes that the HMD generates an identification information corresponding to the number of selectable object data included in the display range of the display, and associates each of the generated identification information with each of the object data included in the display range and displays them. That is, as shown in FIG. 2 and the like, Patent Document 1 describes that an identification information image (which may also be referred to as an ID image) is added to and displayed on the AR image. It can be said that this ID image is also a kind of AR image.

Prior Art Documents

[0004] [Patent Document 1] Japanese Patent Publication No. 2016-218868 [Overview of the Initiative] [Problems that the invention aims to solve]

[0005] Conventional video display devices such as HMDs present difficulties in users selecting a desired AR image when, for example, a large number of objects and AR images are present on the display surface. Furthermore, conventional methods for selecting AR images include detecting hand gestures using a camera. These gestures include, for example, virtually touching or pinching the target AR image with the fingers. However, this method can sometimes be difficult to detect when an AR image is selected.

[0006] Conventional video display devices have room for improvement in terms of user operability and usability regarding generated images such as augmented reality (AR). The object of the present invention is to provide a technology for video display devices that can improve user operability and usability regarding generated images such as AR. Other issues, configurations, effects, etc. will be described in the embodiments for carrying out the invention. [Means for solving the problem]

[0007] A representative embodiment of the present invention has the following configuration. The video display device of one embodiment is a video display device that superimposes a generated image onto a real image, displays the generated image on the display surface, configures an ID image that includes identification information to support user operation on the generated image, detects the user's gaze direction and sets a gaze region corresponding to the gaze direction, displays the generated image in association with the generated image when the generated image falls within the gaze region, accepts the user's selection operation on the ID image, selects the generated image associated with the ID image when the selection operation on the ID image is received, and executes a predetermined process associated with the generated image. [Effects of the Invention]

[0008] According to a representative embodiment of the present invention, with respect to the technology of an image display device, it is possible to improve the user operability and usability of generated images such as AR. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows the external appearance of an HMD, which is an image display device according to Embodiment 1 of the present invention. [Figure 2] This figure shows the functional block configuration of the video display device according to Embodiment 1. [Figure 3] This diagram shows the main processing flow of the video display device according to Embodiment 1. [Figure 4] This diagram shows the processing flow for ID images in the video display device of Embodiment 1. [Figure 5] This figure shows an example of the software configuration for the video display device of Embodiment 1. [Figure 6] This diagram shows the basics of ID image display in the video display device of Embodiment 1. [Figure 7] This figure shows an example of the configuration of the line of sight area in the video display device of Embodiment 1. [Figure 8] This figure shows an example of the configuration of the management table in the video display device of Embodiment 1. [Figure 9]It is a diagram showing a command table for voice input in the video display device of Embodiment 1. [Figure 10] It is a diagram showing a first example of a real image, an AR image, and an ID image in the video display device of Embodiment 1. [Figure 11] It is a diagram showing a second example of a real image, an AR image, and an ID image in the video display device of Embodiment 1. [Figure 12] It is a diagram showing a first example of an ID image in the video display device of Embodiment 1. [Figure 13] It is a diagram showing a second example of an ID image in the video display device of Embodiment 1. [Figure 14] It is a diagram showing a first state of a video in the video display device of Embodiment 1. [Figure 15] It is a diagram showing a second state of a video in the video display device of Embodiment 1. [Figure 16] It is a diagram showing an example of a video in the video display device of Embodiment 1. [Figure 17] It is a diagram showing an example when an ID image is selected in the video display device of Embodiment 1. [Figure 18] It is a diagram showing a first example of enlargement / reduction of a gaze area in the video display device of Embodiment 1. [Figure 19] It is a diagram showing a second example of enlargement / reduction of a gaze area in the video display device of Embodiment 1. [Figure 20] It is a diagram showing an example of fixation of a gaze area in the video display device of Embodiment 1. [Figure 21] It is a diagram showing an example of deviation of a gaze area in the video display device of Embodiment 1. [Figure 22] It is a diagram showing an example of a video in the video display device of Embodiment 2 of the present invention. [Figure 23] It is a diagram showing a first example of a selection area in the video display device of Embodiment 2. [Figure 24] It is a diagram showing a second example of a selection area in the video display device of Embodiment 2. [Figure 25] It is a diagram showing a third example of a selection area in the video display device of Embodiment 2. [Figure 26]This figure shows a fourth example of the selection area in the video display device of Embodiment 2. [Figure 27] This figure shows the processing flow in the video display device according to Embodiment 3 of the present invention. [Figure 28] This figure shows an example of an image in the image display device according to Embodiment 4 of the present invention. [Figure 29] This figure shows the line of sight depth in the video display device according to Embodiment 5 of the present invention. [Figure 30] This figure shows an example of an image in the image display device according to Embodiment 6 of the present invention. [Figure 31] This figure shows an example of an image in the image display device according to Embodiment 7 of the present invention. [Modes for carrying out the invention]

[0010] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In principle, the same parts will be denoted by the same reference numerals in all the drawings used to describe the embodiments, and repeated descriptions will be omitted.

[0011] (Embodiment 1) The video display device of Embodiment 1 of the present invention will be described using Figures 1 to 21. The video display device of Embodiment 1 is shown when applied to an HMD. The case shown is a transparent HMD, but it can be similarly applied to other types such as non-transparent HMDs.

[0012] [HMD(1)] Figure 1 shows the external appearance of the HMD1, which is an image display device of Embodiment 1. The HMD1 is a head-mounted housing and includes a display surface 2, a speaker 3, a gaze detection unit 4, a camera 5, and a microphone (not shown). The housing of the HMD1 houses a control unit and a display unit. The control unit is composed of an integrated circuit and controls the entire HMD1, including the display unit. The display unit includes the display surface 2 and displays images on the display surface 2. The display surface 2 is the area on which the user can view the images. The display surface 2 is, for example, a spectacle-shaped display element placed in front of both of the user's eyes, and is, for example, a transparent display surface. That is, the HMD1 is a transparent HMD. The control unit controls the display of images, including AR images, on the display surface 2. The display unit is, for example, a projection display device and includes a light source, an optical system such as a lens, a drive circuit, etc. In the case of a transparent HMD, the real image in front is transmitted through the display surface 2 as seen by the user's eyes, and generated images such as AR images are superimposed on that real image. In other embodiments where a non-transparent HMD is used, the display unit including the display surface 2 is composed of, for example, a liquid crystal display element. In this case, the display surface 2 displays an image obtained by combining a generated image with the forward-facing image captured by the camera 5. The method for displaying the image on the display surface 2 is not particularly limited, but for example, a projection display method can be applied.

[0013] Speaker 3 emits sound according to control from the control unit. The microphone inputs the user's voice and sends it to the control unit. The control unit performs voice recognition processing from the microphone input sound to detect predetermined commands, etc. The control unit executes predetermined processing according to the detected command. This enables various functions, including AR control. The HMD1's memory is pre-loaded with a dictionary of voice commands (command table described later), and various commands are registered therein. In Embodiment 1, the HMD1 can use voice commands to select ID images in AR images and operate the gaze area.

[0014] The gaze detection unit 4 detects the state of the user's gaze, at least the direction of the gaze. The gaze detection unit 4 may also detect other states, such as blinking. The control unit works in conjunction with the gaze detection unit 4 to constantly understand the user's gaze direction. The camera 5 captures an image of the area in front of the HMD1. The camera 5 may be a stereo camera or the like. The control unit processes the image from the camera 5 to perform, for example, object detection.

[0015] The HMD1's housing is equipped with sensors such as a GPS receiver, electronic compass, gyroscope, and accelerometer. The HMD1 uses these sensors to detect its position, orientation, acceleration, and other parameters.

[0016] The HMD1 in Figure 1, in other words, the main unit of the video display device, may be connected to an operating device (in other words, a remote control) via a wireless communication interface or a wired communication interface. The control unit works in conjunction with the operating device and receives operation input from the operating device. In this case, the user can issue commands to the main unit by operating buttons, etc., on the operating device with their fingers. The operating device is not limited to a dedicated device; a general-purpose portable information terminal device such as the user's smartphone may also be used. In this case, the user can operate the HMD1 from the smartphone, etc. For example, the user's smartphone may contain AR apps and data. In that case, the control unit of the HMD1 communicates with the AR apps and data on the smartphone.

[0017] [HMD(2)] Figure 2 shows the functional block configuration of HMD1. HMD1 includes a main control unit 100, a memory unit 110, a video processing unit 120, an audio processing unit 130, a communication processing unit 140, a sensor unit 150, an operation input unit 160, a battery 170, a system bus 180, etc. Each unit sends and receives commands and data via the system bus 180. The main control unit 2 is a processor composed of an MPU, etc., and controls the entire HMD1 according to the program 111 in the memory unit 120. The processor reads the program data from the storage device into processing memory and executes processing according to the program to realize predetermined functions, etc.

[0018] The memory unit 120 is composed of a non-volatile storage device such as a flash ROM or SSD. The memory unit 120 stores data and information such as a program 111 for operation control, a database 112, and setting information 113. The program 111 includes a program that implements the OS of the HMD1, a program that implements AR functions, a program that implements support functions, and so on.

[0019] DB112 is a database that stores various types of data, including content data such as AR images, data such as ID images for support, and management information, and includes data in data structures such as files, tables, or lists. Setting information 113 includes system setting information and user setting information. Storage unit 120 includes a work area for processing. Storage unit 120 also stores various types of information acquired from communication networks and external devices, such as AR programs, video and audio content data, and other information. Examples of various types of information include search information related to objects on a map. Storage unit 120 also stores video data captured using camera 5. However, various types of data and programs may also be stored in external server devices that are linked to HMD1.

[0020] The sensor unit 150 includes various sensors for detecting the state of the HMD1 and its surroundings. The sensor unit 150 includes, for example, a GPS receiver, a geomagnetic sensor (electronic compass), an accelerometer, a gyroscope, a distance sensor, etc. Using these sensors, the HMD1 can detect its position (e.g., latitude, longitude, altitude), orientation (e.g., azimuth and elevation), tilt, movement such as acceleration, distance to an object, etc. The sensor unit 150 may also include other sensors such as an illuminance sensor, a proximity sensor, a barometric pressure sensor, etc.

[0021] The communication processing unit 140 is composed of an IC chip or the like and includes a communication processing circuit and an antenna. The communication processing unit 140 includes a LAN communication unit 141, a telephone network communication unit 142, etc. The LAN communication unit 141 performs data communication processing with the LAN via an access point using a method such as Wi-Fi (registered trademark). The telephone network communication unit 142 enables telephone and data communication by performing wireless communication processing with base stations of a mobile telephone communication network using a method such as W-CDMA (registered trademark), GSM, or LTE. The communication processing unit 140 may also include communication units for other methods such as Bluetooth (registered trademark) or infrared.

[0022] The video processing unit 120 is composed of an IC chip and the like, and includes an imaging unit 121, a gaze detection unit 4, a display unit 123, etc. The imaging unit 121 includes a camera 5. The display unit 123 includes a display surface 2. The camera 5 of the imaging unit 121 obtains image data by converting light input from the lens into an electrical signal using an element such as a CCD or CMOS. The gaze detection method of the gaze detection unit 4 can be a known method. For example, this method includes a method of detecting the gaze by photographing the inner corner of the eye and the iris with a visible light camera, or a method of detecting the gaze using the corneal reflection method with an infrared LED and an infrared camera. The display unit 123 includes an image processing circuit and video RAM, and superimposes an AR image on the display surface 2 based on the video data input to the video RAM. A touch panel or the like may also be applied to the display unit 123.

[0023] The voice processing unit 130 is composed of an IC chip or the like and includes a voice input unit 131, a voice recognition unit 132, and a voice output unit 133. The voice input unit 131 includes a microphone and converts input sound into voice data. The voice output unit 133 includes a speaker 3 and converts the voice data into sound for output. The voice recognition unit 132 analyzes the input voice data and detects commands, etc. The voice recognition unit 132 may be composed of a dedicated IC or may be composed of program processing by the main control unit 100.

[0024] The operation input section 160 is the part that receives basic operation inputs to the HMD1 (e.g., power on / off, volume adjustment, etc.), and includes, for example, hardware buttons and touch sensors. The HMD1 operates using the power of the battery 180 even when not supplied with external power.

[0025] [Processing Flow (1)] Figure 3 shows the main processing flow of HMD1. The flow in Figure 3 has steps S1 to S9. The steps will be explained below in order. The main processing is carried out by the processor.

[0026] In step S1, the HMD1 detects its position and orientation. The position of the HMD1 is, for example, its position in a geographical three-dimensional space, and is obtained as coordinates (latitude, longitude, altitude) using a GPS receiver or the like. The orientation of the HMD1 corresponds to the direction (Z direction) in which the display surface 2 in Figure 1 is viewed, and is obtained as azimuth and elevation angles using an electronic compass. In addition, the HMD1 detects its current state using the sensor unit 150.

[0027] In step S2, HMD1 (e.g., an AR program) generates and displays an AR image in a predetermined manner. For example, HMD1 detects a predetermined object that is the target of AR from within the real image on the display surface 2. For example, HMD1 uses a camera 5 or the like to detect the object's 3D position and distance from HMD1, and detects the object's 2D position and image area within the display surface 2. HMD1 acquires object information about the detected object. For example, HMD1 may search for object information corresponding to the object's location from map data or the internet. Alternatively, HMD1 may acquire information about objects located near its current position from map data based on HMD1's current position. HMD1 may also use a service such as Google Street View to acquire information about objects such as stores on a map.

[0028] HMD1 generates an AR image based on acquired object information and displays it on display surface 2. For example, the AR program of HMD1 displays the AR image by associating it with the position of the object on display surface 2 and the image area.

[0029] HMD1 detects the AR image generated and displayed by the AR program described above and obtains information about the AR image. HMD1 may refer to the AR object information stored in DB112 (the management table described later) or obtain the AR object information from the internet.

[0030] In step S3, the HMD1 uses the gaze detection unit 4 to detect the user's gaze direction in real time, such as the gaze direction ED in Figure 1. The HMD1 sets a gaze region (Figure 6, described later) on the display surface 2 according to the detected gaze direction. The gaze region is set as a circular area with a predetermined radius, for example, with the point where the gaze direction and the display surface 2 intersect as the center point. In the first embodiment, the HMD1 is configured not to display the gaze region on the display surface 2 in the basic settings, but it may also be configured to display the gaze region on the display surface 2 in the basic settings. Furthermore, the HMD1 in the first embodiment can switch the display and hiding of the gaze region in response to user operations (commands, described later). The position of the gaze region, etc., is updated at each control timing. If the movement of the gaze is small, the same gaze region as last time is used.

[0031] Furthermore, during the processing in step S3, HMD1 also determines whether the line of sight has deviated from the display surface 2. If the line of sight has deviated, HMD1 executes a predetermined process. This process may include, for example, the reconstruction of the ID image within the line of sight. Also, when HMD1 is first used, the initial line of sight is used.

[0032] In step S4, HMD1 makes a determination regarding the AR image in step S2 and the viewing area in step S3 within the display surface 2. That is, HMD1 determines whether the position or area of ​​the AR image is within the viewing area at that time, whether they overlap or not, etc. For example, HMD1 determines whether the "display position" of the "AR image" in the management table (Figure 8) described later is within the range of the viewing area at that time.

[0033] In step S5, HMD1 generates an ID image to associate with the corresponding AR image within the line of sight, and displays the ID image at a position corresponding to the AR image on the display surface 2. This ID image indicates that the corresponding AR image is selectable. At this time, if there is one or more AR images within the line of sight, HMD1 assigns ID numbers to them sequentially, for example, starting from 1. Then, HMD1 generates an ID image with that ID and displays it near the AR image.

[0034] In Embodiment 1, the HMD1 displays ID images only for AR images within the line of sight, and does not display ID images for AR images outside the line of sight. The HMD1 may internally prepare IDs and ID images for AR images outside the line of sight, but will not display them. Selectable AR images are those within the line of sight that have an assigned ID image. As described above, the number of ID images displayed on the display surface 2 is reduced, thereby reducing the amount of information, making it easier for the user to select an ID image (corresponding AR image).

[0035] In step S6, the HMD1 accepts user input. In particular, the HMD1 accepts input for selection operations related to ID images. The HMD1 of Embodiment 1 accepts at least ID image selection operations via voice recognition command input.

[0036] In step S7, HMD1 checks whether the ID image selection operation input in step S6 was received. If the ID image selection operation input was received (Y), it proceeds to step S8; otherwise, it proceeds to step S9.

[0037] In step S8, the HMD1 selects (in other words, puts into a selected state) the AR image indicated by the ID of the selected ID image, and causes the corresponding AR program to execute a predetermined process associated with that AR image.

[0038] In step S9, HMD1 checks whether the predetermined termination operations related to the AR function and assistance function have been performed. If the termination operations have been performed (Y), the flow ends; otherwise, it returns to the initial step S1 and repeats the same process.

[0039] Furthermore, in the loop of the above flow, in step S1, HMD1 may determine whether the change in position, orientation, etc. of HMD1 exceeds a predetermined degree, and if it does, execute subsequent update processing (e.g., updating the gaze area or ID image). Similarly, in step S3, HMD1 may determine whether the change in gaze direction exceeds a predetermined degree, and if it does, execute subsequent update processing. In addition, when determining the change in the gaze area, etc., HMD1 may use a set threshold. For example, if the angle representing the gaze direction is greater than or equal to a set angle threshold (e.g., 10 degrees), HMD1 may determine that there has been a change in gaze direction and perform update processing. Alternatively, HMD1 may determine that there has been a change in gaze direction if the angle representing the gaze direction changes continuously for a predetermined time threshold (e.g., 1 second) or longer. Also, for example, if the difference in the position of HMD1 between time points is greater than or equal to a set distance threshold (e.g., 0.1 m), HMD1 may determine that there has been a change in the position of HMD1 and perform update processing. If there is no or only a small change in the gaze area, HMD1 will continue to use the gaze area settings that were set previously.

[0040] [Processing Flow (2)] Figure 4 shows a detailed processing example of the ID image processing in steps S6 to S8. Figure 4 has steps S21 to S27. In step S21, HMD1 detects a predetermined command based on speech recognition. HMD1 processes according to the command table (Figure 9) described later. In step S22, HMD1 executes processing of a predetermined function corresponding to the detected command. Commands include commands related to the gaze area and commands related to the ID image, and each command has a corresponding processing defined.

[0041] In step S23, HMD1 checks whether an ID image has been selected. If it has not been selected (N), the flow ends; if it has been selected (Y), it proceeds to step S24. In step S24, HMD1 differentiates the display state on display surface 2 for the selected ID image. That is, HMD1 displays the ID image on display surface 2 with a predetermined display effect (e.g., flashing display, high brightness display, specific color display, etc.). This informs the user which ID image has been selected.

[0042] In step S25, HMD1 selects an AR image associated with the selected ID image. For example, HMD1 sends a selection notification to the AR program associated with the selected AR image (Figure 5, described later). Upon receiving the selection notification, the AR program selects the AR image and executes a predetermined process corresponding to the selection of the AR image. As an example, in the case of an AR application that provides information on stores on a map, the AR application obtains detailed information about the store corresponding to the selected AR image, generates an AR image that displays that detailed information, and displays it. In another example, in the case of an AR application that allows the user to arbitrarily place AR images within a real-world image, the AR application selects the AR image and, in response to user operations, moves or deletes the AR image, or issues commands to the AR image.

[0043] In Embodiment 1, a gesture method can also be used as an operation input method. The flow in Figure 4 shows the flow when this method is used and has corresponding steps S26 and S27. In step S26, the HMD1 acquires an image of the front using the camera 5, and determines and detects a predetermined gesture from the image by image recognition processing. The gesture to be detected is, for example, a touch gesture in which the user places their fingers on the position of the ID image for a predetermined time or longer within the area corresponding to the display surface 2. Alternatively, another gesture to be detected is a gesture in which the user virtually pinches the ID image with their fingers.

[0044] In step S27, the HMD1 performs processing of a predetermined function that is defined in advance in response to the detected gesture. This processing includes at least an ID image selection operation corresponding to the predetermined gesture. After step S27, the process proceeds to step S23, where it is similarly confirmed whether an ID image has been selected.

[0045] [Software Configuration Example] Figure 5 shows an example of the software configuration of the HMD1 in Embodiment 1. Figure 5 shows an example of implementing support functions at the OS and middleware level of the HMD1. The HMD1 has an OS 500, an AR program 501, an AR application 502, and a program 510. The OS 500 includes middleware, etc. The AR program 501 is a program included in the OS 500 that has the function of generating and displaying AR images. The AR program 501 is a program that displays information such as the time, battery status, and communication status of the HMD1, as well as icons for various functions, as AR images on the display surface 2. In this example, the case where the AR program 501 generates an AR image Ax and displays it on the display surface 2 is shown.

[0046] AR app 502 is a variety of application programs running on OS 500 that have the function of generating and displaying AR images. AR app 502 may have multiple AR apps, such as AR apps P1 to Pm. This example shows a case where each AR app 502 (P1 to Pm) generates and displays AR images A1 to Am on display surface 2.

[0047] As in this example, the premise is that HMD1 displays AR images on the display surface 2 using the AR program 501 of OS500 and each AR application 502. The AR program 501 and each AR application 502 each have the function of executing predetermined processing associated with the AR image.

[0048] Furthermore, the HMD1 of Embodiment 1 has a program 510 on which support functions are implemented. The program 510 may be implemented as part of the OS 500 or at the level of a single application. The main control unit 100 realizes the support functions by executing processing according to the program 510 on the OS 500. This support function is an AR image manipulation support function. This support function uses ID images to facilitate selection operations for multiple AR images from the AR program 501 and each AR application 502. This support function assigns and displays ID images to AR images according to the gaze area. This support function enables the selection of AR images by selecting ID images.

[0049] Program 510 works in conjunction with AR program 501 and each AR application 502. Program 510 generates and displays ID images 504 associated with AR images 504 displayed on display surface 2. For example, this shows a case where ID images B1-Bm and ID image Bx are associated with and displayed for AR images A1-Am and AR image Ax, respectively.

[0050] Program 510 receives input from the user for selecting an ID image 504 (for example, a voice command specifying the ID). Program 510 interprets this input as a selection of an ID image and an AR image. If an AR image 503 associated with the selected ID image 504 is selected, Program 510 sends a selection notification to the corresponding program (AR program 501 or AR app 502). Upon receiving the selection notification, the program selects the target AR image and executes a predetermined process associated with that AR image.

[0051] In another embodiment, the support functions of program 510 may be implemented only in a specific application. In this case, the specific application provides support related to selection operations by displaying ID images, etc., only for specific AR images that it controls.

[0052] [ID Image Display] Figure 6 illustrates the basics of assigning and displaying ID images to video containing AR images on display surface 2. In Figure 6, the rectangular frame schematically represents display surface 2. The two-dimensional coordinate system of display surface 2 is indicated by (x,y). The x direction is the horizontal direction within the plane, and the y direction is the vertical direction within the plane. Within display surface 2, one or more AR images 30 may be displayed depending on objects not shown, geographical location, or independently of objects. In this example, AR images 30 include AR image a1 of a heart-shaped mark and AR image a2 of a star-shaped mark. AR image a1 is displayed at position p2(x2,y2) within display surface 2.

[0053] On the other hand, on the display surface 2, a viewing area 20 is set according to the user's viewing direction. The viewing area 20 is dynamically set in response to changes in the viewing direction. Let's assume that the point where the user's viewing direction intersects with the display surface 2 is, for example, position p1(x1,y1). In this case, a circular area with a predetermined radius r1, with position p1 as the center point, is set as the viewing area 20. The viewing area 20 is not displayed in the default settings, but is shown as a dashed line.

[0054] For example, an AR image 30(a1) is located within the line of sight area 20. In this case, the HMD1 generates an ID image 40(b1) for the AR image 30(a1) and displays it in association with it. In this example, the ID image 40 has a rectangular ID image b1 with the ID number "1". The ID image b1 is, for example, an image in which the number "1" is placed within a rectangular frame. The area within the frame of the ID image b1 is, for example, a transparent area, but it may also be an opaque area. In this example, the ID image b1 is displayed at a nearby position p3(x3,y3) relative to the position p2 of the AR image a1, for example, at a position above the AR image a1. Since the AR image a2 is not within the line of sight area 20, the ID image 40 is not displayed. The position in which the ID image 40 is displayed relative to the position of the AR image 30 is not limited to the position above it; for example, a suitable available position is selected depending on the display state of the AR image 30 and the ID image 40 on the display surface 2 at that time.

[0055] [View area] Figure 7 shows an example configuration of the gaze area 20. In Figure 7, gaze areas 20a, 20b, and 20c are shown as examples of gaze areas 20. Gaze area 20a is an example with a circular shape, similar to Figure 6. The shape of the gaze area 20 is not limited to this, and various shapes can be set. Gaze area 20b is an example with a horizontally elongated ellipse shape. Gaze area 20c is an example with a horizontally elongated rectangle. Furthermore, the size of the gaze area 20, such as its radius, may be a fixed size set in advance, or it may be changeable by user settings. For example, the vertical and horizontal width of the gaze area 20 may be user-configurable. Also, as will be described later, when using the HMD1, the size of the gaze area 20 can be changed in response to predetermined operation inputs by the user.

[0056] Furthermore, the image shows the case where the viewing area 20a is set to the center position pc(xc,yc) of the display surface 2. In Embodiment 1, the position of the viewing area 20 changes according to the user's viewing direction. For example, if the position at the end of the viewing direction changes from position pc to position pd(xd,yd), the viewing area 20a moves to the viewing area 20d.

[0057] In the modified HMD1, the viewing area 20 may be fixedly set at a predetermined, fixed position within the display surface 2, for example, at the center position pc(xc,yc), regardless of the user's viewing direction. The vertical and horizontal dashed lines passing through position pc are the bisectors of the vertical and horizontal widths of the display surface 2. This modified version corresponds to a form in which the viewing area 20 is controlled by considering the user's viewing direction as approximately the same as the orientation of the HMD1. In this modified version, the hardware and software configuration can be made simpler. The user changes the orientation of the HMD1 by moving their head. This causes the viewing area relative to space to change.

[0058] Furthermore, as will be described later, it is possible to fix or release the position of the gaze area 20 on the display surface 2 in response to predetermined user input. This makes it possible to fix the display of the ID image within the gaze area 20 when the user is focusing on a particular direction of gaze.

[0059] As will be described later, it is possible to display an image representing the gaze area 20 (gaze area boundary) as a type of AR image on the display surface 2 in response to a predetermined operation input by the user, or to switch it to be hidden. The image representing the gaze area 20 is, for example, a boundary line, and may be an image such as a dashed line as shown in the figure. For example, when displaying the gaze area 20a, the dashed circle shown in the figure is displayed as the boundary line.

[0060] [Management Table] Figure 8 shows an example configuration of a management table for controlling AR images and ID images. This management table contains a list of all AR images and ID images. HMD1 reads and writes information about all AR images and ID images to this management table. The HMD1's processor creates and maintains the management table in memory as part of DB112. By referring to this management table, HMD1 can understand the current state of the AR images and ID images.

[0061] The management table in Figure 8 has the following items: Management ID, AR Object, Display Position, Base Position, and Identification Information. "Management ID" is identification information for management purposes, such as an address or row number. "AR Object" is an AR image, and includes a corresponding image data file. "Display Position" is the 2D coordinate (x,y) that serves as the reference position when displaying the AR image on display surface 2. "Base Position" is the 3D position coordinate (X,Y,Z) that serves as the basis for generating the AR image. "Base Position" corresponds to, for example, a location on a map or in real space. Note that AR images generated independently of real-world objects may not have a "Base Position". "Identification Information" includes the ID and ID image. The ID is information such as a number used to identify and select the AR image. The ID image is a display image containing that ID, and includes a corresponding image data file. ID numbers are generated sequentially, for example, 1, 2, 3, ... If an ID image is not attached to the AR image, the "Identification Information" item is set to a value (0) indicating the absence of an ID image. Other data items may include a flag indicating whether or not an ID image can be selected.

[0062] [Command List] Figure 9 shows an example of the command table configuration corresponding to the speech recognition method in Embodiment 1. This command table has pre-configured usable commands. This command table has the following items: Command ID, Target (Type), Command, Voice, and Remarks. A command is set for each row in the table. The HMD1 detects the "command" in the command table from the input voice. The "Command ID" indicates the identifier of the "command". The "Target (Type)" indicates the target and type of that "command". The "Voice" shows an example of the voice used when inputting that "command". The "Voice" is not limited to the example voice, but can be set in various ways as long as it is a voice that can uniquely distinguish each command. The "Remarks" shows a description of the processing content, etc.

[0063] Commands with IDs C01 to C06 have "Gaze" as the "Target (Type)" and are a group of commands related to controlling the gaze area. Commands with IDs C11 to C42 have "ID" as the "Target (Type)" and are a group of commands related to controlling the ID image.

[0064] The command with command ID=C01, "Show gaze area on," switches the display to show an image representing the gaze area 20 (e.g., a boundary line) within the display surface 2, as shown in Figure 6. The corresponding voice command is, for example, "Show gaze area on." The command with command ID=C02, "Show gaze area off," switches the display to a state where the image representing the gaze area 20 is not displayed within the display surface 2. The corresponding voice command is, for example, "Show gaze area off."

[0065] Command ID=C03, "Look-line area lock," is a command that switches the position of the look-line area 20 within the display surface 2 to a fixed state, and the voice command is, for example, "Look-line lock" or "Freeze." Command ID=C04, "Look-line area unlock," is a command that unlocks the position of the look-line area 20 and switches it to a free state, and the voice command is, for example, "Look-line area unlock."

[0066] Command ID=C05, "Expand Gaze Area," is a command that expands the size of the gaze area 20 within display surface 2, and the accompanying voice command is, for example, "Expand Gaze Area" or "Make it Bigger." Command ID=C06, "Reduce Gaze Area," is a command that reduces the size of the gaze area 20, and the accompanying voice command is, for example, "Reduce Gaze Area" or "Make it Smaller."

[0067] Command ID=C11, "ID display on," is a command that switches the display to show the ID image on display screen 2, and the accompanying voice command is, for example, "Shikibetsu on" or "ID on." Command ID=C12, "ID display off," is a command that switches the display to not show the ID image on display screen 2, and the accompanying voice command is, for example, "Shikibetsu off" or "ID off."

[0068] Commands from "Select ID1" (command ID=C21) to "Select ID9" (command ID=C29) are used to individually select each of the nine IDs, for example, from ID "1" to ID "9". For example, the "Select ID1" command selects ID image "1" (for example, ID image b1 in Figure 6), which has ID number "1", and the corresponding voice is, for example, "Number 1", "Number 1", or "ID 1". In this example, nine IDs from "1" to "9" are used, but it is also possible to use 10 or more IDs, and separate commands may be provided for numbers of 10 or more.

[0069] If the voice recognition detected from the user's input corresponds to, for example, "Ichiban" (number one), it is interpreted as the "Select ID1" command, and thus as an operation to select ID image "1". This operation to select ID image "1" is then interpreted as an operation to select the AR image associated with that ID image "1" at that time. As a result, that AR image is selected, and the predetermined processing associated with that AR image is executed.

[0070] The "Select All IDs" command (command ID=C30) selects all ID images within the gaze area, and the voice command is, for example, "All". The "Select One ID" command (command ID=C31) selects any one ID image within the gaze area according to a predetermined rule, and the voice command is, for example, "Select". This command, in particular, selects the single ID image closest to the center point of the gaze area (i.e., the direction of gaze) if there are multiple ID images within the gaze area.

[0071] Command ID=C32, "Deselect All IDs," is a command that deselects all ID images within the gaze area if any ID images are currently selected, returning them to a deselected state. The corresponding voice command is, for example, "Deselect All IDs." Command ID=C33, "Deselect One ID," is a command that deselects any one ID image within the gaze area, returning it to a deselected state. Commands C32 and C33 can be applied when, for control purposes, selected and deselected states are distinguished for ID images and their corresponding AR images.

[0072] The "Execute ID" command with command ID=C34 is a command that executes the processing of the AR image associated with a selected ID image. For example, the voice input is "Jikkou" (execute). The C34 command can be applied when, for AR images, selection and execution are separate in terms of control. If, for AR images, selection and execution are not separate in terms of control, the selection operation directly corresponds to the instruction to execute the processing.

[0073] The "Display AR Information" command with command ID=C41 displays information about the ID image within the line of sight, or the AR image associated with the selected ID image. The voice prompt is, for example, "Info On". The "Hide AR Information" command with C42 switches the AR information generated by command C41 to a hidden state, and the voice prompt is, for example, "Info Off". Information about the AR image includes, for example, the name and function of the AR program that generated the AR image. By viewing this information, the user can confirm the function and meaning of the AR image. Note that this command is separate from data output as an example of processing execution of an AR image by an AR program. A certain AR program will output data as an example of predetermined processing when an AR image is selected. For example, in the case of an AR app that provides information about stores on a map, this data output is the display of the store information as an AR image.

[0074] In addition to the above example, the HMD1 may also have voice commands for operating various functions via the OS, etc. Furthermore, in the case where the AR application and support functions are implemented together, an example of a voice command may be a command corresponding to the execution of AR image processing by the AR application. For example, a "data on" command that displays data corresponding to the AR image could be used. For example, an AR application that provides store information would, when the "data on" command is entered, display an AR image containing store information for the selected AR image.

[0075] The voice processing unit 130 in Figure 2 may also be equipped with a speech synthesis function. The HMD1 may use the speech synthesis function to output voice responses to the user as appropriate. For example, the HMD1 may output a corresponding voice response when the user inputs a predetermined voice command. Examples of voice responses include voice confirmations of input commands and voice questions.

[0076] [AR images and ID images] Figures 10 and 11 show examples of AR images and ID images displayed on display surface 2. The example in Figure 10 schematically shows an example of video when using HMD1 outdoors, including real images of buildings such as shops near a road. The example in Figure 11 schematically shows an example of video when using HMD1 indoors, including real images of tables, walls, etc. First, the following are examples of AR images that can be displayed by existing AR programs. The method for generating and displaying AR images is not particularly limited and can be based on publicly known technologies. Various AR methods may be mixed. For example, the AR methods of each AR application 503 in Figure 5 may be different.

[0077] Figure 10 shows AR images A1 to A13 as examples of AR images (=AR objects). AR images A1 and A2 are examples of AR images that the user has arbitrarily placed within the real image for purposes such as relaxation. These AR images are, for example, animals such as birds or cats, or virtual characters, and are composed of, for example, animated images. The user places the selected desired AR image within the display surface 2 through a predetermined operation. This AR image is displayed, for example, at a designated fixed position within the display surface 2. Alternatively, this AR image may move autonomously within the display surface 2 according to a program. For example, the bird AR image A1 moves as if flying within the display surface 2. Furthermore, this AR image may always be displayed within the display surface 2 regardless of the position of the user and HMD1.

[0078] AR images A3 and A4 are examples of AR images that display store information, such as images with text like the store's name displayed in a rectangular area. For example, an AR application guides users to store information using AR images. The AR application searches for nearby stores based on map data, according to the current location and orientation of the HMD1. The AR application generates AR image A3, etc., using the searched information and displays it on the display surface 2, corresponding to the geographical location of the store. Alternatively, an AR application detects objects such as buildings and signs of stores from images taken by the camera 5, obtains information about those objects, and generates and displays an AR image using that information. The object information can be any set information, such as name, type, address, and status.

[0079] AR images A5 and A6 are examples of images that display information such as arrows and speech bubbles for user navigation. For example, an AR app has a navigation function that includes route search, and generates AR images for route navigation from the current location to the destination according to the destination entered by the user. For example, AR image A5 is an image that indicates to turn right at the next intersection and that the destination is 30m away. AR image A6 is an image that indicates that there is a station in the direction of the arrow.

[0080] AR image A7 is an example of displaying a frame image on a detected person's face. For example, a certain AR app pre-registers information about people the user knows. The AR app detects the faces of these people based on the image from camera 5. The AR app then adds a frame image and the person's information to the detected face area and displays it as AR image A7. This allows the user to find people they know and to check their information.

[0081] AR images A8, A81, A82, and A83 illustrate an example of a group of AR images that displays information about the recommended dishes of the day offered by a restaurant, as shown in AR image A4. AR image A8 is a text image of "Today's Recommendations." AR images A81, A82, and A83 are images of various dishes associated with AR image A8. For example, an AR app might automatically display this kind of detailed information as push notifications for a restaurant.

[0082] Figure 11 shows AR images A21 to A28. AR image A21 is an AR image of multiple birds, etc., placed by the user, and is moving autonomously. AR image A22 is an AR image of a cat, etc., placed by the user, and is, for example, stationary. AR images A23 and A24 are AR images of vases, food, etc., placed by the user on a real-world table. AR images A25, A26, and A27 are AR images of calendars, paintings, clocks, etc., placed by the user on a real-world wall. For example, one AR application can place various AR objects at desired locations in a room. AR image A28 is an AR image of icons of multiple documents, etc., placed by the user at predetermined locations on display surface 2. When AR image A28 is selected, the corresponding program displays the contents of the documents, etc., on display surface 2.

[0083] As shown in the examples above, the display form of AR images can be various, including icons, marks, text, shapes, animations, 3D models, and other images, and is not particularly limited. Also, as shown in the examples above, some AR images are displayed in correspondence with real-world objects, while others are displayed independently of objects. Furthermore, AR images may be accompanied by audio output (e.g., sound effects, animal sounds, etc.).

[0084] Another example of the latter type is when the HMD1's OS or other program displays control-oriented AR images. Examples of these AR images include AR images of icons for various functions and apps, AR images representing status such as date and time, battery level, and communication status, and AR images representing email notifications, etc. This type of AR image is displayed on the display surface 2 for a certain period of time, regardless of the position of real-world objects.

[0085] The AR methods are not limited to the examples above; various known methods can be applied. For example, an AR marker method may be applied. In this method, the HMD1 detects a predetermined marker (e.g., a 2D code) within the real image by recognizing the image from the camera 5, and generates and displays a predetermined AR image corresponding to the position of that marker. Another method may be used that uses the user's gaze direction to detect an object or point in that direction, and generates and displays an AR image corresponding to that object or point.

[0086] The following is an example of how ID images are displayed for the above AR images. In Figure 10, as an example of a gaze area 20, there is a gaze area 20 located slightly to the upper right within the display surface 2. Within this gaze area 20 are a total of 7 AR images: AR images A1, A4, A6, A8, A81, A82, and A83. HMD1 assigns ID numbers to the multiple AR images within the gaze area 20 in order of proximity to the center point and displays the ID images. In this example, 7 ID images, ID images "1" to "7", are generated and displayed in order of proximity to the center point of the gaze area 20. For example, AR image A81 is assigned ID image "1". AR image A82 is assigned ID image "2". AR image A83 is assigned ID image "3". AR image A8 is assigned ID image "4". AR image A1 is assigned ID image "5". AR image A4 is assigned ID image "6". AR image A6 is assigned the ID image "7". This example shows how to assign and display an ID image to all AR images within the gaze area 20.

[0087] In Figure 11, as an example of a viewing area 20, the viewing area 20 is located slightly above the display surface 2. Within this viewing area 20 are a total of seven AR images: three AR images A21, three AR images A23, two AR images A24, and two AR images A25 and A26. In this example, seven ID images, ID images "1" to "7", are generated and displayed in order of proximity to the center point of the viewing area 20. For example, the three AR images A21 are assigned ID images "1" to "3". AR image A23 is assigned ID image "4". AR image A25 is assigned ID image "5". AR image A24 is assigned ID image "6". AR image A26 is assigned ID image "7".

[0088] [ID Image] In the example above, numbers such as "1" and "2" are used as IDs for the ID images. The range of IDs that can be used for the ID images may also be set. For example, in the default settings for HMD1, the range of IDs is set to "1" to "9". It is also possible to set the number of ID images that can be displayed simultaneously on the display surface 2, or the number of ID images that can be displayed simultaneously in the line of sight area 20. For example, in the default settings for HMD1, the maximum number of ID images in the line of sight area 20 is set to 5. These limitations help to reduce the amount of information on the display surface 2, making it easier to view.

[0089] The assignment of ID numbers to ID images is carried out according to a predetermined rule. For example, numbers are assigned sequentially starting from "1" in order of proximity to the center point within the line of sight. Furthermore, the assignment of ID numbers is done dynamically depending on the situation. Even for the same AR image, different IDs may be assigned depending on the situation. In the first control example described later, HMD1 controls the system so that, over time, the same ID image is maintained for the same AR image as much as possible. For example, after an ID is assigned to an AR image, the same ID may be maintained for at least a certain period of time. In the second control example described later, HMD1 reassigns ID images to the AR images within the line of sight according to a predetermined rule at each point in time of control.

[0090] Furthermore, the HMD1 allows users to configure settings regarding the controlled objects, such as displaying an ID image for AR images from a specific AR application, but not displaying an ID image for AR images from other specific AR applications.

[0091] Figure 12 shows examples of various display modes for ID images, all of which are applicable. In example (A), AR image a1 displays ID image b1 with ID "1" without a frame on the left side. In example (B), AR image a1 displays ID image b1 with ID "1" superimposed on the same position. In example (C), AR image a1 displays ID image b1 with ID "A" in a circular frame on the upper side. This is an example using the letter "A" as the ID. IDs are not limited to numbers; various characters can be used.

[0092] Example (D) shows how to place ID image b1, which has ID "1", at a position slightly away from AR image a1. Furthermore, AR image a1 and ID image b1 are connected by a connecting line to indicate their relationship. For example, this type of display may be applied if there is no available space near AR image a1.

[0093] Example (E) has an AR image a101 that displays information about the real store j1 at a position slightly away from the actual store j1 location, via a connecting line of association. The AR image a101 displays text such as the name of the store j1 within a rectangular area. Above the AR image a101, an ID image b1 with ID "1" is displayed.

[0094] In example (F), an ID image b1 with ID "1" is displayed in relation to an AR image A1 moving within display surface 2. This ID image b1 is basically displayed in a position that follows the movement of AR image A1. As a variation, this ID image b1 is displayed in a fixed position instead of a position that follows the movement of AR image A1. Alternatively, this ID image b1 may be displayed to move at a slower speed than the movement of AR image A1.

[0095] Example (G) shows how, for AR image A5 such as an arrow, an ID image may be added or not displayed even within the line of sight, depending on user settings. If displaying an ID image for certain types of AR images makes them difficult to understand, they can be hidden in this way. Also, for example, if an AR image originally contains characters or shapes such as numbers, the ID image can be hidden.

[0096] Other examples of IDs include those that are identified by differences in color or shape. For example, shapes such as circles, triangles, and squares can be used. Alternatively, colors such as red, blue, and yellow can be used.

[0097] In the example in (A) of Figure 13, multiple AR images, for example, three AR images a1, a2, and a3, are densely arranged in close proximity. The HMD1 basically displays each ID image near each AR image. In this case, for example, if the ID images are displayed above each AR image, the situation becomes like (A). The three ID images b1, b2, and b3 with IDs "1" to "3" are displayed densely. In this case, the information is somewhat difficult for the user to see. An example of improvement is shown in (B), etc. In the example in (B), for the same three AR images a1 to a3, the ID images are displayed in different positions, for example, in positions that spread outwards. As a result, the multiple ID images b1 to b3 do not overlap and are easier to see. In the example in (C), for the same three AR images a1 to a3, the three ID images b1 to b3 are organized and displayed within a speech bubble image. Within the speech bubble image, each AR image (which may also be a reduced image) is first displayed vertically. Then, within the speech bubble image, the corresponding ID image is displayed next to each AR image.

[0098] Furthermore, example (D) shows an example where multiple AR images (e.g., AR images a1~a3) are grouped together and assigned a single ID image b101 for display. ID image b101 is, for example, an image with ID=group ID="A". For example, ID image b101 with group ID "A" is displayed near AR images a1~a3, or near a speech bubble image. In this case, it is possible to select multiple AR images a1~a3 in the group together in response to the selection operation of the group-unit ID image b101. As a variation, HMD1 initially displays only the group-unit ID image b101 near the cluster of multiple AR images a1~a3. When the user selects ID image b101, HMD1 displays information about the multiple AR images a1~a3 and ID images b1~b3 included in that group using a speech bubble image or the like, allowing them to be selected individually.

[0099] (E) shows an example of assigning and displaying a group-specific ID image b101 when there are multiple AR images that are originally related as a group, such as AR images A8, A81~A83 in Figure 10. HMD1 encloses the group of AR images A8, A81~A83, for example, with a frame, and assigns and displays an ID image b101 that includes that frame. (F) shows an example of assigning and displaying a group-specific ID image b101 in the case of the three AR images A21 in Figure 11.

[0100] [Display control example (1)] A more detailed example of ID image display control in Embodiment 1 is described below. The HMD1 controls the assignment and display of an ID image in accordance with the changes in the AR image corresponding to changes in the gaze area 20.

[0101] Figure 14 shows the first control example. The first control example maintains the same ID image for a given AR image as much as possible. For example, when a given first AR image enters the line of sight, the ID image with ID "1" is assigned and displayed. Subsequently, when the first AR image leaves the line of sight, the assignment of ID "1" is maintained internally, and the ID image is hidden. Subsequently, when the first AR image enters the line of sight again, the same ID image with ID "1" is displayed again. On the other hand, the second control example does not maintain the same ID image for a given AR image as much as possible, but rather assigns a new ID each time according to the state of the line of sight at that time.

[0102] Figure 14 shows the first state of the image on display surface 2. In this example, the real image shows three store buildings near a road. Examples of AR images displayed in the first state are AR images a1, a2, a3, a4, a5, a11, a12, and a13. AR images a1 to a5 are examples of marks of each shape, and AR images a11, a12, and a13 are examples of rectangular images that display store information. In the first state, the viewing area 20 is viewing area E1, with a center position p1 (x1, y1) and a radius r1. In the first state, AR images a1, a2, and a3 are within viewing area E1. AR images a4, a5, a11, a12, and a13 are outside viewing area E1. In this example, the determination of whether an AR image is within the viewing area is made by checking whether the "display position" (Figure 8) of the AR image is within the viewing area, but this is not the only possible method. Alternatively, the method could involve determining whether or not a portion of the AR image falls within the line of sight.

[0103] HMD1 assigns IDs to AR images in order from those closest to the center position p1 of the line of sight E1. In this example, the AR images a1, a2, and a3 are in order of proximity to position p1. Therefore, HMD1 assigns and displays three ID images "1", "2", and "3" to the three AR images a1, a2, and a3.

[0104] The ID numbering method is not limited to this example; various methods can be applied. Examples of other methods include assigning numbers in the order in which AR images appear within the line of sight. Another example is scanning the line of sight in a predetermined order (for example, sequentially scanning from the top left to the bottom right) and assigning numbers in that order.

[0105] As shown in the example above, when there are multiple AR images on the display surface 2, the HMD1 of Embodiment 1 displays the ID image only within the line of sight. Therefore, the amount of information displayed to the user is reduced, making it easier to manipulate the AR images. The user can manipulate the AR images by selecting the ID image for the AR image they are focusing on in the vicinity of their line of sight. The HMD1 may also be configured to display the ID image on the AR images within the line of sight and then erase it after a certain period of time. The ID image may be a semi-transparent image.

[0106] Figure 15 shows the second state of the image on display surface 2. Assume that the orientation and line of sight of HMD1 have shifted slightly to the left from the line of sight area E1 in the first state, resulting in the line of sight area E2 in the second state. In the second state, AR image a1 remains within the line of sight area E2, a new AR image a11 has entered, and AR images a2 and a3 have moved outside. AR image a11 is an example of a rectangular image displaying store information ("Store AAA").

[0107] HMD1 maintains the display of ID image "1" for AR image a1. HMD1 hides ID images "2" and "3" for AR images a2 and a3. HMD1 assigns and displays ID image "4" for AR image a11. In the first control example, even though ID images "2" and "3" for AR images a2 and a3 are hidden, their assigned numbers are maintained, and for AR image a11, the next number after "3", "4", is assigned. In the second state, both ID images "1" and "4" are displayed within the gaze area E2.

[0108] When using the second control example, the following occurs: HMD1 cancels the assignment of ID images "2" and "3" to AR images a2 and a3 that are outside the line of sight E2. HMD1 assigns and displays ID image "2", which has the next number after ID image "1", to the newly incoming AR image a11. In this state, ID images "1" and "2" are displayed within the line of sight E2.

[0109] Furthermore, for example, if the user returns to the gaze region E1 of the first state from the gaze region E2 of the second state, the first control example will be as follows: HMD1 will again display the same ID images "2" and "3" for AR images a2 and a3 that have entered the gaze region E1, and will hide the ID image "4" for AR image a11 that has left the gaze region E1.

[0110] [Display control example (2)] Figure 16 shows an example of the image on display surface 2. Assume that the user's gaze moves slightly to the right from the first state of gaze area E1 in Figure 14, resulting in the state of gaze area E3 in Figure 16. Within gaze area E3, AR images a1, a2, and a3 remain, and new AR images a4 and a5 are added. Here, assume that the user setting state is that the ID range is "1" to "9", and the maximum number of ID images that can be displayed simultaneously within the gaze area is 3. Although there are positions for 5 AR images within gaze area E3, HMD1 limits the display of ID images to the maximum number = 3.

[0111] In the first control example, the following occurs: HMD1 first assigns and displays ID image "4" to the first incoming AR image a4, and hides ID image "1" for AR image a1. At this point, the maximum number of ID images "2", "3", and "4" (3 in total) are displayed. Next, HMD1 assigns and displays ID image "5" to the incoming AR image a5, and hides ID image "2" for AR image a2. At this point, the maximum number of ID images "3", "4", and "5" (3 in total) are displayed. In this way, HMD1 displays an ID image for newly incoming AR images while sequentially hiding older IDs (or IDs behind the movement of the gaze area, etc.) to keep the number down to the maximum. Similarly, when HMD1 has used up to the last ID in the range ("9"), it returns to the first ID in the range ("1") and uses them cyclically.

[0112] In a similar situation, the second control example is as follows. Similarly, assume that the ID range is "1" to "3" and the maximum number is 3. HMD1 first assigns and displays ID image "1" to the first incoming AR image a4, and then removes the assignment of ID image "1" to AR image a1. At this point, the maximum number of ID images "1", "2", and "3" (3) are displayed. HMD1 then assigns and displays ID image "2" to the next incoming AR image a5, and removes ID image "2" from AR image a2. At this point, the maximum number of ID images "1", "2", and "3" (3) are displayed. In this way, HMD1 assigns and displays ID images to newly incoming AR images, and also removes the assignment of ID images to any of the AR images, so as to always use ID images "1", "2", and "3" within the predetermined range.

[0113] As described above, the HMD1 of Embodiment 1 allows the maximum number of ID images that can be displayed within the viewing area to be set in advance, thereby limiting the number of ID images that can be displayed. However, for example, if multiple AR images are densely packed within the viewing area, exceeding the maximum number, it may not be possible to display multiple ID images related to all of those AR images. The HMD1 may automatically control the expansion or contraction of the viewing area depending on the density of multiple AR images in the viewing area. For example, the HMD1 may reduce the size of the viewing area when multiple AR images are densely arranged. For example, the HMD1 may expand the size of the viewing area when multiple AR images are sparsely arranged. For example, the HMD1 may adjust the size of the viewing area to match the maximum number of images that can be displayed simultaneously. By keeping the number of ID images displayed on the display surface 2 roughly constant, the user can easily perform selection operations. It is also possible to set the maximum number of images that can be displayed simultaneously to one. In this case, only a maximum of one ID image is displayed on the display surface 2 and within the viewing area 20, making selection operations easy.

[0114] [Select ID image, execute AR image processing] FIG. 17 shows a display example when a user selects an AR image a11 within a line-of-sight region E2 using a corresponding ID image b11 (= ID image “4”), in an example of a real image similar to the example of the video in FIG. 15. Within the line-of-sight region E2, AR images a1, a11 are present, and ID images “1” and “4” are displayed. A certain AR app displays an AR image a11 having store information. This AR app further provides, as a predetermined process in response to a selection operation of this AR image a11, detailed information regarding the store as a further AR image.

[0115] Suppose the user wants to view detailed information regarding the store indicated by the AR image a11. In this case, basically, the user selects the AR image a11. At that time, instead of the AR image a11, the user can use the ID image b11 displayed for assistance. In this example, the user selects the ID image b11. When using a voice command, the user inputs the ID number “4” indicated by the ID image b11 (corresponding “ID4 selection” command) by voice. For example, the user inputs voice such as “yonban”. The HMD1 recognizes the voice and detects it as an “ID4 selection” command. The HMD1 interprets the command as a selection operation of the ID image b11 having ID “4” and, at that time, as a selection operation of the AR image a11 associated with that ID image b11.

[0116] The HMD1 performs display differentiation of the ID image b11 (FIG. 4, step S24) in response to the selection operation of the ID image b11. Correspondingly, display differentiation may also be performed for the target AR image a11. Also, the HMD1 notifies the corresponding AR app of the selection of the AR image a11 (FIG. 5). Thereby, the AR app sets the AR image a11 to a selected state and performs a predetermined process defined corresponding to the selection operation, in this example, display of store detailed information.

[0117] The AR application constructs an AR image a11b containing detailed information about the store and displays it at or near the location of AR image a11. In this example, the detailed information in AR image a11b includes information such as the type of store "AAA" and its business hours. In the AR application, an AR image may consist of multiple AR images (e.g., multiple layers or multiple pages). The HMD1 may assign and display an ID image to each of these multiple AR images. Data output by the AR application may also be audio output. Furthermore, for example, if the AR application receives a predetermined operation from the user on AR image a11 or AR image a11b (e.g., a second selection operation), it will erase AR image a11b.

[0118] [Delete ID image] This section describes an example of controlling the deletion of ID images. The HMD1 deletes, or in other words, hides, the ID images within the line of sight in response to a predetermined user operation. This will be explained using the example video in Figure 14. Within the line of sight E1, three ID images "1" to "3" are displayed, corresponding to AR images a1 to a3. If the user wants to delete the ID images within the line of sight E1 in this state, they input the "ID display off" command, for example, C12 in Figure 9. In response, the HMD1 switches the ID images "1" to "3" within the line of sight E1 to a hidden state. When an ID image is hidden, the HMD1 disables the selection operation for that ID image. When the ID images are deleted, the corresponding AR images are displayed, but they cannot be selected. For example, even if the user inputs the "Select ID1" command in this state, they cannot select ID image "1" or AR image a1.

[0119] Furthermore, if the user wishes to display the ID images again, they can enter the "ID Display On" command, for example, C11 in Figure 9. In response, the HMD1 switches the display state of ID images "1" to "3" within the gaze area E1.

[0120] As a variation, the system may be configured to simultaneously delete the ID image and the corresponding AR image in response to a predetermined command.

[0121] [Select multiple ID images] This section describes an example of control when selecting one or more ID images from multiple ID images within the gaze area. First, in the example video in Figure 14, the user can specify and select individual AR images "1" to "3" within the gaze area E1 using individual commands (Figure 9, C21 to C29). The user can also select multiple AR images at once by specifying multiple ID images consecutively within a certain time period. For example, when using the voice method, the user speaks multiple ID numbers consecutively within a certain time period. During voice recognition processing, the HMD1 detects multiple words (corresponding voice units) within a certain time period and interprets them as a command based on the sequence of multiple words. The input voice might be, for example, "one, three, five". This allows the user to select ID images "1", "3", and "5" together. Alternatively, for example, if the user wants to select a range of ID images "2", "3", "4", and "5" with consecutive numbers, the user speaks that range. The input voice might be, for example, "two, from, five" (meaning 2 to 5). This allows you to select that range all at once.

[0122] Alternatively, if the user wants to select all ID images "1" through "3" within the gaze area E1, they can enter the "Select All IDs" command, for example, as shown in C30 of Figure 9. In response, the HMD1 selects all ID images "1" through "3" within the gaze area E1. This allows all three corresponding AR images a1 through a3 to be selected.

[0123] Alternatively, the user can select the single ID image closest to the center of their line of sight within the gaze region E1. In this case, the user enters the "Select 1 ID" command, for example, C31 in Figure 9. In response, the HMD1 selects the ID image closest to the center within the gaze region E1, for example, ID image "1". This ensures that only the corresponding single AR image a1 is selected.

[0124] [Enlarge / Reduce field of view] Figure 18 shows an example of control when the viewing area is enlarged or reduced in response to a predetermined user operation. The example image in Figure 18 is the same as in Figure 14. Initially, the viewing area is in the state of E1. If the user wants to enlarge the viewing area, for example, they input the "Enlarge Viewing Area" command C05 in Figure 9. In response, HMD1 enlarges and displays the viewing area E1. The enlarged viewing area E11 is shown. The viewing area E11 has a larger radius r11 than the original (r11>r1). Also, if the user wants to reduce the viewing area, for example, they input the "Reduce Viewing Area" command C06 in Figure 9. In response, HMD1 reduces and displays the viewing area E1. The reduced viewing area E12 is shown. The viewing area E12 has a smaller radius r12 than the original (r12>r12). <r1)。

[0125] In the case of the enlarged viewing area E11, AR images a4 and a11 are newly included. Accordingly, for example, HMD1 assigns and displays ID images "4" and "5" to AR images a4 and a11, within the maximum number (e.g., 5). If the maximum number is, for example, 3, ID images "4" and "5" are not assigned and displayed to AR images a4 and a11. In the case of the reduced viewing area E12, AR images a2 and a3 are outside the viewing area, and only AR image a1 remains. Accordingly, HMD1 switches the ID images "2" and "3" for AR images a2 and a3 to hidden. In this way, the number of ID images displayed on display surface 2 can be changed by enlarging / reducing the viewing area. Since the number of ID images can be changed according to the state viewed by the user, it is easier for the user to select ID images.

[0126] The modified HMD1 may also use the following as another example of control regarding the enlargement / reduction of the gaze area. In this control example, HMD1 enlarges / reduces the gaze area based on automatic judgment, rather than in response to user operation as shown in Figure 18. It is assumed that the maximum number of ID images that can be simultaneously displayed within the gaze area is set to, for example, 3, based on user settings. A command to change this maximum number may also be provided. An example of voice command would be "Saidai, San" (meaning "Side, Three").

[0127] In the example of Figure 19 (A), multiple AR images (e.g., AR images a1, a2, a3) are arranged on the display surface 2 at relatively low density and wide intervals. In this case, the HMD1 detects up to 3 AR images in order of proximity from the center of the line of sight p1. For example, in order of proximity, these are AR images a1, a2, and a3. The HMD1 constructs a line of sight area 20 with a size that matches the arrangement of these 3 AR images. In this example, a radius r1a is chosen that can encompass the 3 AR images a1 to a3, and a line of sight area E1a with this radius r1a is constructed. The HMD1 then assigns and displays the ID images "1" to "3" to each of the AR images a1 to a3 within this line of sight area E1a.

[0128] In the example of (B) in Figure 19, multiple AR images (e.g., AR images a1, a2, a3) are arranged relatively densely and at close intervals within the display surface 2. For example, suppose the state of the AR images changes from state (A) to state (B). In this case, HMD1 similarly detects up to 3 AR images in order of proximity from the center of the line of sight position p1. HMD1 constructs a line of sight area 20 with a size that matches the arrangement of these 3 AR images. In this example, a radius r1b is taken that can encompass the 3 AR images a1 to a3, and a line of sight area E1b with this radius r1b is constructed. Then, HMD1 assigns and displays the ID images "1" to "3" to the AR images a1 to a3 within this line of sight area E1b.

[0129] In this modified version, the viewing area is automatically adjusted according to the arrangement of the AR images on display surface 2. This modified version is particularly effective when displaying an image representing the viewing area.

[0130] As another control example, the shape of the viewing area may be deformed using predetermined commands. For example, commands such as vertical zoom in, vertical zoom out, horizontal zoom in, and horizontal zoom out may be provided. The user can select the size and shape of the viewing area in advance on the user settings screen provided by the HMD1.

[0131] [Fixing the line of sight] Figure 20 shows an example of control related to fixing the gaze area and fixing the ID image within the gaze area. The user can temporarily fix the gaze area or release it in response to a predetermined operation. Fixing the gaze area fixes the display of the ID image within the gaze area, and the display of the ID image can be maintained even when the gaze moves.

[0132] In the example video in Figure 20, the initial state is that of the gaze region E21. Within the gaze region E21, centered at position p21, there are AR images a1 and a11, and the corresponding ID images "1" and "2" are displayed. If the user wants to fix the state of ID images "1" and "2" within this gaze region E21, they can input the "Looking Region Fix" command, for example, C03 in Figure 9. Upon receiving this operation, the HMD1 fixes the state of the gaze region E21, including its position and size. Along with this fixation, the state of ID images "1" and "2" displayed within that gaze region E21 is also fixed.

[0133] After fixing the position as described above, suppose the user moves their gaze to the right, and the direction of their gaze moves from position p21 to position p22, and then from position p22 to position p23. In this case, if the position is not fixed, the gaze area E21 will change to gaze areas E22 and E23. A new AR image a13 will be placed within gaze area E23, and an ID image will be assigned and displayed. On the other hand, if the position is fixed as in this control example, the gaze area E21 will not change. Therefore, the ID images "1" and "2" of AR images a1 and a11 will remain displayed, and no ID image will be assigned or displayed for AR image a13. Even when the user's gaze is at position p23, the user can still select ID images "1" or "2".

[0134] Furthermore, if the user wishes to release the fixed gaze area 20, they input the "Release Gaze Area Lock" command, for example, C04 in Figure 9, as a predetermined operation. In response, the HMD1 releases the fixed state of the gaze area E21. The HMD1 then resets the gaze area to match the position of the gaze at the time of release. For example, if the gaze is at position p23 at the time of release, the gaze area E23 is configured. As a result, for example, the ID image "1" is assigned and displayed to the AR image a13 within the gaze area E23, and the ID image "1" etc. is removed from AR images a1 etc. outside the gaze area E23.

[0135] [Deviation from the line of sight] Figure 21 shows an example of control in the case of deviation from the line of sight. When setting the line of sight in step S3 of Figure 3, HMD1 determines whether the line of sight deviates from the display surface 2. If HMD1 determines that there is a deviation from the line of sight, it readjusts the line of sight accordingly. Initially, assume the state is as shown in (A) of Figure 21. In this state, AR images a1 to a3 are contained within the line of sight E1, and ID images "1" to "3" are displayed. Also, AR image a11 corresponding to a store, etc., is displayed on the display surface 2. Note that there is a building to the left of the building shown in AR image a11, but since information could not be acquired, a corresponding AR image was not generated. In this state, the line of sight E1 does not deviate from the display surface 2.

[0136] Next, suppose the orientation of HMD1 changes due to the user's head movement, resulting in state (B). Assume the user's gaze is still directed at the same position p1. In this state, the gaze region E1 extends about halfway outside the area of ​​display surface 2, with a semicircular portion remaining within display surface 2. HMD1 determines this state as a "deviation" from the gaze region if, for example, the position p1, the center of the gaze, moves outside the area of ​​display surface 2. Note that in (B), for clarity in explanation, an AR image corresponding to (A) is also illustrated outside of display surface 2. This example shows a deviation from the gaze region due to a change in the orientation of HMD1, but the gaze region may also deviate due to the movement of the user's gaze.

[0137] If position p1 is within the display surface 2, the line of sight remains valid, even if it is only a semicircular portion. For AR image a1 within that line of sight, the display of ID image "1" is maintained. On the other hand, AR images a2 and a3 that were within a semicircular portion outside the display surface 2 are naturally not displayed, along with their corresponding ID images "2" and "3". If position p1 moves outside the display surface 2, HMD1 determines that it has deviated from the line of sight E1 and resets the ID images that were within that line of sight. For example, HMD1 revokes the assignment of ID images "2" and "3" to AR images a2 and a3.

[0138] Furthermore, when the user's gaze returns to the display surface 2, the HMD1 sets the gaze area accordingly. For example, if the gaze moves from position p1 in (B) to position p5, the gaze area E5 is configured to correspond to position p5. In this case, corresponding to the new AR image a11 entering the gaze area E5, the ID image "2" is added to and displayed on the AR image a11.

[0139] As a variation, the HMD1 may determine whether the entire image area of ​​the line of sight region 20 has completely deviated outside the display surface 2.

[0140] [Effects etc. (1)] As described above, the video display device of Embodiment 1 can improve user operability and usability regarding AR images by setting a gaze area according to the direction of the user's gaze and displaying ID images within the gaze area. In particular, according to Embodiment 1, since the display of ID images is limited to the gaze area, the amount of information on the display surface 2 from the user's perspective is suppressed, making it easy to select AR images using IDs. The user only needs to select from a small number of IDs, making the operation easy.

[0141] Furthermore, in Embodiment 1, as shown in Figure 3, after detecting the AR image (step S2), a gaze area is set (step S3), and an ID image is displayed (step S5). The user can set the gaze area while viewing and confirming the AR image on the display surface 2, and can select an ID image within that gaze area. Therefore, the chances of missing an AR image on the display surface 2 can be reduced.

[0142] [Comparative Example] As a comparative example of the video display device HMD (Head-Mounted Display) for Embodiment 1, we will use the configuration example described in Patent Document 1 for comparison. The HMD of the comparative example, for example, acquires one AR image from a list of AR images (=AR objects) displayed on the display surface, and if that AR image is a selectable AR image, it assigns an ID number to it. The HMD then displays the image with the assigned ID number near the AR image. In the comparative example, the ID image is assigned and displayed to the AR image regardless of the user's line of sight. Therefore, on the display surface, there may be many AR images and corresponding ID images (corresponding numbers), or multiple ID images may be densely clustered. In that case, it may be difficult for the user to select an ID image.

[0143] On the other hand, the HMD1 of Embodiment 1 displays ID images primarily within the line of sight, even when there are numerous AR images on the display surface 2. Furthermore, the HMD1 of Embodiment 1 displays multiple ID images in a way that makes them as easy to see as possible. Therefore, users can easily select the ID image associated with the desired AR image. In addition, the HMD1 of Embodiment 1 allows for operation of the line of sight using voice commands, thereby enhancing the operability of the ID images.

[0144] [Modified form (1)] The following modifications are also possible for the HMD1 of Embodiment 1. In Embodiment 1, the selection of an ID image was, as a basic concept, defined as including the selection of an AR image and the execution of a process associated with the AR image. That is, when a user selects an ID image, a predetermined process related to the AR image associated with that ID image is automatically executed by the corresponding program. In the modified HMD, the selection of an ID image may be defined as a separate concept of AR image selection and AR image processing execution. In this case, when a user selects an ID image, the AR image associated with that ID image is selected by the corresponding program. Then, when a user performs a predetermined operation on the selected AR image, a predetermined process related to that AR image is executed by the corresponding program. Furthermore, the above ID image selection concepts may be mixed, and different actions may be performed when selecting an ID image depending on the AR application or a specific AR image.

[0145] [Modified form (2)] The modified HMD has the functionality to select multiple ID images as a single group. For example, a user may want to select multiple desired AR images within display surface 2 as a single group; in other words, they may want to select multiple AR images at once. First, using the group ID shown in Figure 13 above, it is possible to select a group by specifying the group ID. Furthermore, by using the "Select All IDs" command mentioned above, all ID images within the line of sight can be selected at once.

[0146] Alternatively, the user can, for example, enter a predetermined "group selection" command and then sequentially specify individual ID images. For instance, to select ID images "1," "3," and "5" within the gaze area, the voice input might be something like "select, one, three, five." Furthermore, if AR image selection and AR image processing are separate processes, the user can select the desired AR image and then execute the AR image processing by entering a predetermined "execute" command.

[0147] (Embodiment 2) The video display device of Embodiment 2 of the present invention will be described using Figures 22 to 26. The basic configuration of Embodiment 2 and others is the same as that of Embodiment 1, and the following will describe the components of Embodiment 2 and others that differ from Embodiment 1. The HMD of Embodiment 2 has an area for selecting ID images (sometimes referred to as the "selection area" or "identification information area") on the display surface. The user can select not only ID images displayed in areas other than the selection area on the display surface, but also ID images displayed within the selection area.

[0148] In the second embodiment, in the flow shown in Figure 4 above, the confirmation in step S23 includes checking whether an ID image within the selection area has been selected. Also, the gesture determination in steps S26 and S27 includes determining the gesture for the ID image within the selection area.

[0149] [Selected area (1)] Figure 22 shows an example of the image on the display surface 2 of the HMD1 of Embodiment 2. A selection area 60 related to the ID image is provided within the display surface 2. In this example, this selection area 60 is provided as a horizontally elongated rectangular area near the bottom edge of the display surface 2, and is, for example, a transparent area. This selection area 60 may be hidden from the user, or it may be made visible to the user by displaying the selection area boundary line, etc.

[0150] Within the line of sight area 20, as described above (Figure 11), there are a total of seven AR images: three AR images A21, and four AR images A23, A24, A25, and A26. These AR images are placed relatively close together, making selection difficult without further processing. When using the control method of Embodiment 1, ID images "1" to "7" are assigned and displayed to the seven AR images. These ID images "1" to "7" are also placed relatively close together, which may make selection difficult without further processing.

[0151] Therefore, the HMD1 of Embodiment 2 displays corresponding ID images in the selection area 60 for seven AR images within the gaze area 20. The HMD1 arranges multiple ID images within the selection area 60 according to a predetermined rule. In this example, seven ID images "1" to "7" are arranged in the selection area 60 at equal intervals, from left to right, in ascending order of number. For example, from left to right in the selection area 60, there are ID images c1, c2, c3, ..., c7. Also, in this example, multiple ID images are displayed in the selection area 60 at a larger size than the ID images in the gaze area 20. These ID images function like ID selection buttons.

[0152] The user can select each ID image within the selection area 60. In this case, the input method is not limited to the voice method described above; a gesture method can also be used. For example, suppose the user wants to select ID image c3 = ID image "3". In the voice method, by entering the aforementioned "Select ID3" command, the user can select ID image c3, and the corresponding AR image A21 can be selected. When using the gesture method, the user makes a virtual touch gesture with their fingers to the location of ID image c3. This gesture involves, for example, moving the fingertip to a position that overlaps with the rectangular area of ​​ID image "3" from the user's perspective, and keeping the fingertip in that position for a predetermined time or longer. During this gesture, the user's fingers are outside the front of the display surface 2 of the HMD1 and are visible through the display surface 2 and the ID images. Therefore, the user and camera 5 can understand the state of the fingers.

[0153] The HMD1 detects the gesture based on the image recognition from the camera 5 and interprets it as a selection operation for the ID image c3. This detection of ID image selection operation by gesture is easier and more accurate than detecting gestures for AR images or ID images at various positions on the display surface in conventional HMDs. In Embodiment 2, detection is easy because it is only necessary to determine the gesture for the position of the ID image within a predetermined selection area 60. Even if the AR image or ID image moves within the display surface 2, detection is easy because the ID image is displayed at a fixed position within the selection area 60.

[0154] In the control example shown in Figure 22, the respective ID images are displayed in both the gaze area 20 and the selection area 60. For example, for AR image A21, the ID image "1" is displayed twice, in both the gaze area 20 and the selection area 60. This is not the only variation; as a modification, the ID image may be displayed only in the selection area 60. However, in that case, it is necessary to make the correspondence between the AR image and the ID image clear. For example, the correspondence can be made clear by displaying a reduced image of the AR image together with the ID image, or by connecting the AR image and the ID image with a connecting line.

[0155] Alternatively, the HMD1's input unit 160 may be used as another input method. For example, the user can select an ID image in the selection area 60 by pressing a predetermined hardware button on the main unit or control device.

[0156] [Selection Area (2)] As a modification of Embodiment 2, within the selection area 60, the ID image may be displayed in association with a corresponding reduced image of the AR image.

[0157] Figure 23 shows an example of a modified display. The line of sight area 20 contains AR images a1, a2, and a11. The HMD1 places the ID images "1," "2," and "3" associated with AR images a1, a2, and a11 in the line of sight area 20 into positions (left, center, and right) within the selection area 60, distributed and aligned at equal intervals k1. Furthermore, the HMD1 places reduced images of each AR image a1 to a3 in pairs with their respective ID images. Within the selection area 60, for example, the left position displays a pair consisting of the ID image "1" of AR image a1 and a rectangular image containing a reduced image of AR image a1. For example, the center position displays a pair consisting of the ID image "2" of AR image a11 and a rectangular image containing a reduced image of AR image a11. The selection area 60 may display not only reduced images of AR images, but also partial images cropped from parts of AR images, or enlarged images of some text, etc.

[0158] Because the reduced-size images of the AR images clearly show their correspondence to the original AR image, users can easily select ID images with a feeling similar to directly manipulating the original AR image.

[0159] Multiple ID images within this selection area 60 can also be identified by their position (e.g., left, center, right). The HMD1 of Embodiment 2 also allows selection of ID images within the selection area 60 using a position-specifying command. Examples of voice commands include the "Select Left ID" command, the "Select Right ID" command, and the "Select Center ID" command. The "Select Left ID" command selects the ID image located on the left side of the ID images in the selection area 60, and the voice command is, for example, "Hidari" (left). The "Select Right ID" command selects the ID image located on the right side of the selection area 60, and the voice command is, for example, "Migi" (right). The "Select Center ID" command selects the ID image located in the center of the selection area 60, and the voice command is, for example, "Chuou" (center).

[0160] Other arrangement examples within the selection area 60 include the case of a single ID image, where the single ID image is displayed in the center of the selection area 60, and the case of two ID images, where the two ID images are displayed on the left and right sides of the selection area 60. In these cases as well, it is possible to select individual ID images using a predetermined command. Note that this example shows the ID images arranged in ascending order from left to right within the selection area 60, but this is not the only possible arrangement.

[0161] As a variation, the rectangular image containing the ID image or the thumbnail of the AR image in the selection area 60 may have different colors or shapes for identification. In this case, the user can distinguish and select each ID image or AR image using commands such as position specification, color specification, or shape specification. When distinguishing by color or shape, it may be applied to either the AR image or the ID image, or both. For example, different colors may be set for the ID image (or thumbnail of the AR image) displayed at each position within the selection area 60 (e.g., left, center, right). For example, the left position within the selection area 60 may be set to red, the center position to blue, and the right position to yellow. The user can select the ID image at the left position by entering a command such as "select red ID" (e.g., voice command "red").

[0162] Furthermore, different shapes are assigned to each position within the selection area 60. For example, the left position within the selection area 60 might be a circle, the center a square, and the right position a triangle. The user can then select the ID image on the right by, for example, entering a "triangle ID selection" command (e.g., the voice command "triangle").

[0163] As a variation, the display of the ID image within the selected area 60 may be omitted, and only a reduced-size image of the AR image may be displayed.

[0164] As a variation, multiple ID images may be concentrated in the selection area 60 at any of the following positions: left, center, or right.

[0165] As a variation, it is also possible to apply aspects other than numbers, such as color or shape, to the ID configuration of an ID image. An ID image is not limited to numbers; it only needs to be something that the user can distinguish and specify. In the case of ID images other than numbers, the user can select an ID image using predetermined commands that specify position, color, shape, etc.

[0166] As a variation, if there is a real-world object that serves as the basis for generating the AR image, an image of that object, a reduced image of that object, or a partial image of that object may be used as an ID image, or used in combination with other ID images.

[0167] [Selection Area (3)] As a modification of Embodiment 2, when placing and displaying an ID image within the selection area 60, it may be placed and displayed in a position that is as close as possible to the position of the corresponding AR image.

[0168] Figure 24 shows two display examples of this modified configuration. When HMD1 displays ID images "1" to "3" related to AR images a1, a2, and a11 within the line of sight area 20 in the selection area 60 at the bottom, it places the corresponding ID images in the x-direction (horizontal direction) of the display surface 2 at positions that correspond as closely as possible to the positions of the AR images. In this example, within the selection area 60, the ID images "2" for AR image a11, "1" for AR image a1, and "3" for AR image a2 are placed from left to right. In this case, the ID numbers are not in order, but the x-direction positions indicate the correspondence with the AR images, making it easy for the user to make selections.

[0169] As another display example, the HMD1 may provide a selection area 60 at another location within the display surface 2. Selection area 60b is an example of a selection area provided near the right edge of the display surface 2. When the HMD1 displays ID images "1" to "3" related to AR images a1, a2, and a11 within the gaze area 20 within the selection area 60b on the right edge, it places the corresponding ID images at a position in the y-direction (vertical direction) of the display surface 2 that corresponds as closely as possible to the position of the AR images. Furthermore, if the correspondence between the AR images and ID images is to be made clearer, connecting lines such as dashed arrows may be displayed. Similarly, selection areas 60 may be provided on the top or left edge of the display surface 2. In addition, the position of the selection area 60 within the display surface 2 may be determined as a variable position depending on the user's gaze position and movement.

[0170] [Effects etc. (2)] As described above, the video display device of Embodiment 2 enhances user operability and usability regarding AR images by using the selection area 60. In particular, Embodiment 2 makes it easier to select ID images because the ID image is displayed within the selection area 60. Furthermore, even when using the gesture method, the detection accuracy is improved compared to the conventional example, making it easier to select ID images. In addition, multiple AR images displayed on the display surface 2 may be a mix of various types. Each AR image may have various sizes and shapes. In Embodiment 2, even when there are various AR images, an ID number is assigned, they are standardized to a certain size, and the information is displayed in an organized manner within the selection area 60. Therefore, it is easy for the user to select ID images.

[0171] [Modified form (1)] The following are also possible modifications of Embodiment 2. An example of control using the selection area 60 by the modified HMD1 is described below. In this control example, the HMD1 displays ID images for AR images that have moved outside the line of sight area 20, by arranging them within the selection area 60 according to a predetermined rule.

[0172] Figure 25 shows an example of the display in this control example. In this example, the range of IDs used is set to "1" to "9", and the maximum number of simultaneous displays within the gaze area 20 is set to 5. Initially, the user's gaze is in the state of gaze area E21. Within gaze area E21 are AR images a4, a5 and AR images a12, a13, and correspondingly, ID images "1" to "4" are displayed. Next, the gaze moves, for example, to the left, and the state becomes gaze area E22. Within gaze area E22 are AR images a1, a2, a3, a4, and correspondingly, ID images "4" to "7" are displayed. For AR images a5, a12, a13 that have moved outside gaze area E22, when using the control of Embodiment 1, ID images "1" to "3" are hidden.

[0173] In this modified version, HMD1 hides the corresponding ID images "1" to "3" for AR images a5, a12, and a13 that are outside the line of sight area E22, and displays the corresponding ID images c1 to c3 (ID images "1" to "3") within the selection area 60. HMD1 places the three ID images c1 to c3 within the selection area 60 at positions (right, center, left) that are distributed and aligned with equal intervals k1.

[0174] In this state, the user can select each ID image c1 to c3 within the selection area 60. This allows the user to select AR images such as a5 that are outside the line of sight area E22. For example, if the user selects ID image c1 = ID image "1" within the selection area 60, the corresponding AR image a5 can be selected.

[0175] Furthermore, suppose the gaze moves to the left, reaching the state of gaze region E23. Within gaze region E23 are AR images a1 and a11, and correspondingly, ID images "7" and "8" are displayed for AR images a1 and a11. In this modified example, HMD1 hides the corresponding ID images "5," "6," and "4" for AR images a2, a3, and a4 that are outside gaze region E23, and adds the corresponding ID images "5," "6," and "4" to the selection region 60. That is, ID images "1" to "6" are displayed within the selection region 60 in a regular arrangement, similar to the example in Figure 22. Similarly, in this state, the user can select each ID image within the selection region 60. Note that a maximum number of ID images can also be set for displaying within the selection region 60. For example, if the maximum number is 3, ID images "4" to "6" will be displayed within the selection region 60 instead of ID images "1" to "6."

[0176] As a variation, the HMD1 may display corresponding ID images in the selection area 60 as a history, in chronological order of the order in which the AR images went outside the line of sight area 20. Within this selection area 60, up to a predetermined number of ID images are displayed adjacent to each other, and if the number exceeds the predetermined number, the oldest ID images are deleted in order. In the example in Figure 25, if the order in which the AR images went outside the line of sight area is AR images a13, a12, a5, a4, a3, a2, then the ID images "3", "2", "1", "4", "6", and "5" are displayed in the selection area 60 in that order.

[0177] [Modified form (2)] As an alternative HMD1, if the line of sight deviates from the display surface 2, as in the example in Figure 21 above, the ID images that were within the deviated line of sight may be displayed within the selection area 60. Figure 26 shows an example of display during deviation. When HMD1 determines that the line of sight E1 has deviated from the display surface 2, it displays the ID images of the AR images that were within the line of sight E1 but are no longer visible within the selection area 60. For example, the ID images "2" of AR image a2 and "3" of AR image a3, which were within the line of sight E1, are positioned near the right side of the selection area 60. In this example, since the deviation occurred on the right side of the display surface 2, the ID images are displayed in a position shifted to the right within the selection area 60. The ID images displayed in this case may be reduced images of the AR images, as described above. This alternative can also effectively utilize ID images that have moved outside the display surface 2.

[0178] (Embodiment 3) The third embodiment of the present invention's video display device will be described using Figure 27. The HMD of the third embodiment differs from the flow in Figure 3 in the order of processing, with the main change being that steps S2 and S3 are reversed. The HMD of the third embodiment sets the gaze area, then extracts and displays the AR image within the gaze area, assigns and displays an ID image.

[0179] Figure 27 shows the flow in Embodiment 3. In step S1, HMD1 detects its position, etc. In step S2b, HMD1 detects the direction of gaze and sets the gaze area. In step S3b, HMD1 generates an AR image and displays the AR image within the gaze area if it falls within the gaze area. In step S4b, HMD1 detects an AR image that falls within the gaze area. In step S5, HMD1 assigns an ID image to the AR image within the gaze area and displays it. In Embodiment 3, this flow configuration displays AR images and ID images only within the gaze area.

[0180] [Example Display] An example of image display in the HMD1 of Embodiment 3 is shown below. The example in Figure 14 mentioned above is used. In Figure 14, AR images were displayed outside the line of sight E1, but in Embodiment 3, AR images and ID images are displayed only within the line of sight E1. For example, AR images a1, a2, and a3 are displayed because their display positions are within the line of sight E1, and ID images "1," "2," and "3" are displayed. AR images a4, a5, a11, a12, and a13 are not displayed because their display positions are outside the line of sight E1, and their ID images are not displayed either. Note that when the line of sight E1 moves and the display position becomes appropriate, the AR images and ID images will be displayed.

[0181] [Effects etc. (3)] According to Embodiment 3, the number of AR images and ID images displayed on the display surface 2 is reduced according to the user's line of sight, so the user only needs to focus on the AR images and ID images within their line of sight, making selection easier. Embodiment 3 also has advantages such as improved processing efficiency, as it does not need to display all AR images on the display surface 2.

[0182] (Embodiment 4) The fourth embodiment of the present invention will be described using Figure 28. The HMD of the fourth embodiment continues to display the ID image when the AR image within the line of sight moves outside the line of sight.

[0183] [Example Display] Figure 28 shows an example of an image in Embodiment 4. (A) shows the first state, and (B) shows the second state. The real image in Figure 28 is an example of an area near an intersection. In this example, people are used as the target for displaying the AR image. An AR application detects a person's face and provides information about that person as an AR image. Within the display surface 2 of the image in (A), there are three people, for example, person A, person B, and person C. Persons A and C are ordinary people unknown to the user, while person B is an acquaintance of the user. The AR application has pre-registered the faces and information of people the user knows, including person B.

[0184] The AR application extracts the facial regions of people from the image of camera 5 using known image processing techniques. For example, the facial regions of person A, person B, and person C are extracted. The AR application recognizes and identifies the face of a specific person (e.g., person B) based on image processing such as feature recognition. In other words, the AR application performs personal recognition processing. In this process, for example, the registered information of person B's facial image is used. If the AR application detects the face of person B, for example, it displays an AR image over person B's facial region. This AR image represents a specific person (e.g., person B), such as an acquaintance of the user. In this example, this AR image is a frame image surrounding the facial region, as in AR image a41. This AR image is not limited to this; it could also be a frame image surrounding the body, a speech bubble image, a mark corresponding to the person, etc.

[0185] Figure 28 shows a magnified view of the area around person B's face for illustrative purposes. In this magnified view, there is a frame image as AR image a41 around person B's face. This AR image a41 may also include text images containing information such as person B's name. The AR application may also display the registered information of person B when this AR image a41 is selected and executed. The AR application may also display person B's information in a larger size using a speech bubble image or the like when this AR image a41 is selected and executed.

[0186] In this example, the video in Figure 28 also displays AR image a42 as an example of an AR image from another AR application. AR image a42 is an arrow image for navigation, such as route guidance. This AR image a42 indicates the direction the user should go.

[0187] In Embodiment 3, the HMD1 assigns and displays corresponding ID images b41 and b42 to AR images a41 and a42 that are within the line of sight area E41. For example, ID image b41 has ID "1" and ID image b42 has ID "2".

[0188] Suppose that time has passed from the first state (A), and for example, person B has moved, resulting in the second state (B). Assume that the line of sight area E41 is in almost the same position. If we apply the basic control of Embodiment 1, it will be as follows: In the second state (B), since AR image a41 is outside the line of sight area E41, the "1" in ID image b41 is hidden. In this case, it is not possible to select ID image b41. If the user is tracking, for example, person B's AR image a41, they move their gaze to move the line of sight area and capture the moving person's AR image a41. In this case, the ID image b41 of person B's AR image a41 is also displayed, so it is possible to select ID image b41. However, for example, if there are multiple objects or AR images within the line of sight area and each moves, or if the user focuses on some of the AR images, some of the AR images may move outside the line of sight area, as with person B, and the user may not be able to track them.

[0189] In contrast, the HMD1 of Embodiment 4, in this second state, continues to display the ID "1" of the ID image b41 even for the AR image a41 that has moved outside the line of sight area E41. In this case, the display of the AR image a41 and ID image b41 of person B continues even if the user does not move their gaze. As long as person B is within the display surface 2, the user can track the AR image a41 and ID image b41 and select the ID image b41.

[0190] Furthermore, the control in Embodiment 4 can be configured in advance by the user to apply to specific AR applications or specific types of AR images. Alternatively, the control may be switched to the applied state when the user performs a predetermined operation (e.g., the "ID Tracking" command). In addition, it may be combined with Embodiment 2 to display the ID image of the AR image to be tracked within the selection area.

[0191] As another example, consider the arrow AR image a42. An AR app sequentially displays multiple AR images, such as AR image a42 and AR image a43, for navigation to a destination. For example, AR image a42 is displayed at some point, and AR image a43 is displayed at a later point. The multiple AR images may be displayed automatically in sequence, or the user may select the first AR image a42 to display the next AR image a43. In this way, multiple related AR images may be displayed in different locations. In this case, at least some of the AR images may be outside the line of sight area 20. For example, AR image a43 is outside the line of sight area E41.

[0192] In such cases, HMD1 displays ID images for multiple related AR images, even those outside the line of sight. For example, ID image b42 is displayed for AR image a42, and ID image b43 is displayed for AR image a43. In this example, the ID of ID image b43 is set to "2," the same as the ID of ID image b42. The same applies when there are three or more related AR images.

[0193] [Effects etc. (4)] According to Embodiment 4, the display of the ID image can continue in accordance with the movement of the object, AR image, and ID image, so the user can track and operate on the AR image even if it moves outside the line of sight. Furthermore, the control in Embodiment 4 is applicable not only when the object is moving. For example, it can be applied even when the object and AR image are stationary and only the line of sight is moving.

[0194] (Embodiment 5) The video display device of Embodiment 5 of the present invention will be described with reference to Figure 29. The HMD of Embodiment 5 controls the ID image using the user's line of sight depth.

[0195] [Line-of-sight depth] Figure 29 schematically illustrates the concept of gaze depth in Embodiment 5. (A) shows an example of the size of an object (e.g., a person) according to its position in a real image. (B) shows the gaze depth range according to the position of an object in the depth direction from the user's eye position (= viewpoint position). The gaze detection unit 4 of the HMD1 in Embodiment 5 has a function to detect gaze depth. The gaze detection unit 4 detects the gaze direction of the left eye and the gaze direction of the right eye as the gaze directions of both of the user's eyes. The gaze detection unit 4 can detect gaze depth from the point where the gaze direction of the left eye and the gaze direction of the right eye intersect.

[0196] As in the example in Figure 28 mentioned above, when objects are located at different positions in the depth direction from the user's perspective, the size of each object will differ, as shown in the example image in Figure 29 (A). In image (A), of the three people (person A, person B, and person C), person A is relatively close to the HMD1 and has a shallow depth of sight. Person C is relatively far from the HMD1 and has a deep depth of sight. Person B is at a distance somewhere between person A and person C, and has a depth of sight somewhere in between them.

[0197] In (B), the gaze directions from the user's left and right eyes are shown by dashed lines. The point where the two gaze directions intersect corresponds to the gaze depth. The gaze depth range is the effective range of gaze depth. In (B), the vertical axis shows the distance between HMD1 and the target person, with distance ZA to person A, distance ZB to person B, and distance ZC to person C. The gaze depth ranges HA, HB, and HC are shown as gaze depth ranges corresponding to the distance. HMD1 sets the gaze region 20 corresponding to the detected gaze direction and gaze depth range. For example, if the detected gaze depth range is gaze depth range HB, the gaze region E60 is set in the image of (A). In this case, for example, the selection operation of the ID image of the AR image corresponding to person B can be easily performed.

[0198] [Control example] As an example of control in Embodiment 5, the HMD1 may further automatically adjust the size of the gaze area, etc., according to the gaze depth. The left side of the gaze depth range in Figure 29 shows an example of controlling the size of the gaze area in this control example. For example, the HMD1 sets the size of the gaze area (e.g., radius) to a level selected from the three levels of large, medium, and small, according to the determination of three levels of gaze depth: large, medium, and small. In this control example, a gaze area of ​​a suitable size is set according to the user's gaze direction and gaze depth. The AR image and ID image within the gaze area are determined according to the size of that gaze area. Even if the user's gaze direction is the same, gaze areas of different sizes are set according to the difference in gaze depth. As a result, the ID images that become candidates for selection will differ. For example, in the video in Figure 14, assume that AR image a2 and AR image a4 have their base positions in the background relative to AR image a1. When the user focuses on AR image a1, the gaze depth becomes deeper when the user focuses on AR image a2, etc. Correspondingly, the HMD1 sets a smaller gaze area 20. This further narrows down the AR images that fall within the line of sight area 20. As a result, the number of displayed ID images decreases, allowing for selection of ID images in a more refined state.

[0199] [Effects etc. (5)] According to Embodiment 5, since the gaze state in the depth direction within three-dimensional space is taken into account as the gaze depth, the gaze region and ID images can be controlled in more detail. For example, the user can easily narrow down the ID images to be selected according to their gaze focus.

[0200] (Embodiment 6) The video display device of Embodiment 6 of the present invention will be described using Figure 30. As an example of control related to a specific AR application, the HMD of Embodiment 6 selects and operates multiple AR images associated with an object using multiple ID images to realize a predetermined function.

[0201] [Example Display (1)] Figure 30(A) shows an example of an image in Embodiment 6. This real image shows a room with a triangular table as object 151 in the corner. The HMD1 in Embodiment 5 uses the sensor unit 150 to determine its position in the room. The HMD1 measures the distance between the HMD1 and the wall or floor, and the distance between the HMD1 and object 151, for example, using the camera 5 or the distance sensor of the sensor unit 150. The HMD1 determines the position of various objects in the room in three-dimensional space and understands their positional relationship with the HMD1. As for the distance sensor, known methods such as the TOF (Time Of Flight) method, the laser pattern irradiation method, or the method that uses multiple cameras for calculation can be applied.

[0202] In Embodiment 5, the HMD1, for example, an AR application, uses the camera 5 to capture images of a space such as a room, and generates an AR image using the captured video. This AR application has a function to virtually place objects (corresponding AR objects) in a room according to user operations, and can be used, for example, to simulate interior layouts.

[0203] HMD1 places and displays a user-specified AR object 152 on an object 151 of a triangular table within the line of sight 20, for example. The AR object 152 is, for example, an AR image representing a cylindrical object. Depending on the AR application or user specification, this AR object 152 can be various images, such as a vase or a model. HMD1 assigns and displays an ID image 153 to the AR object 152 if it is within the line of sight 20.

[0204] Furthermore, the AR application moves and displays the AR object 152 in response to a predetermined user operation (e.g., gesture). In other words, the user can place the AR object 152 in a desired position. The HMD1 also moves the corresponding ID image 153 in response to the movement of the AR object 152. As another example, the AR object 154 is an example of placing a rectangular image representing a painting, window, etc., on a wall.

[0205] When, for example, the ID image 153 of the AR object 152 is selected, the HMD1 executes a predetermined process associated with the AR object 152. This process can perform various operations, such as selecting the AR object 152 or turning its display on / off. For example, it may be difficult to move the AR object 152 using direct gestures such as touch or slide. In this case, the user can select the ID image 153 using voice input to select the AR object 152. Then, the user can move the AR object 152 using predetermined commands provided by the AR application (for example, commands such as "move right" or "move left").

[0206] [Example Display (2)] Figure 30(B) shows an example of another image. The AR application on HMD1 can add and display multiple AR objects to an object 151, such as a triangular table. One AR application has a function to measure the length of an object, for example. Based on image recognition by camera 5, HMD1 detects the edges and feature points of object 151. This allows for the detection of points such as p51 at the left corner of the top surface of object 151, p52 at the right corner, and p53 at the back. For example, HMD1 displays AR objects a51 and a52 for the two corner points p51 and p52. AR object a51 is an arrow image representing the left corner point, and AR object a52 is an arrow image representing the right corner point. HMD1 then assigns and displays an ID image to each of the AR objects a51 and a52. For example, AR object a51 displays ID image b51 with ID image "2", and AR object a52 displays ID image b52 with ID image "3".

[0207] The user can select multiple AR objects by selecting ID images within the line of sight area 20. In this example, the user uses the measurement function of this AR app to measure the distance between two points on object 151. To do this, the user selects two ID images b51 and b52, which correspond to the two points. This operation is performed, for example, by inputting the "Select ID2" command and the "Select ID3" command. This selects the two ID images b51 and b52. When multiple AR images are selected, the AR app understands the relationship between those AR images and performs predetermined processing on the multiple AR images based on that relationship. In this example, the AR app calculates the distance (the length of the straight line between the two points) between the selected points p51 and p52. The AR app may also display an AR image of the straight line between the two points. The AR app displays the calculated distance information, for example, as AR image 155. Similarly, the AR app can also measure the area of ​​the region between three points, depending on the specification of three points on an object.

[0208] [Effects etc. (6)] According to Embodiment 6, as in the example above, it is possible to effectively support the operation input of an AR application that handles multiple AR images. Even when the AR image is small or when fine-tuning is required, operation using the ID image is possible.

[0209] (Embodiment 7) The video display device of Embodiment 7 of the present invention will be described with reference to Figure 31. In Embodiment 7, an ID image is directly assigned to and displayed on a real-world object in a specific AR application, enabling virtual manipulation of that object.

[0210] Figure 31 shows an example of an image in Embodiment 7. This real image is of an indoor setting and contains multiple objects (e.g., two) such as chairs and tables, OB1 and OB2. Based on the image recognition from the camera 5, the HMD1 distinguishes and detects the image regions R1 and R2 of the multiple objects OB1 and OB2 within the display surface 2. The HMD1 identifies the object image regions included within the line of sight region 20 and assigns and displays an ID image to the object image regions included within the line of sight region 20. For example, the ID image "1" is assigned and displayed to region R1 of object OB1.

[0211] The user selects the desired ID image corresponding to the desired object. For example, a voice command is input to select ID image "1" associated with object OB1. The HMD1 selects the object image region of the object associated with the selected ID image. The corresponding AR application performs predetermined processing on the object image region of the selected object. An example of the AR application's processing is as follows.

[0212] The first example of an AR application retrieves information about a selected object from its image region, for example, by searching the internet. The AR application then displays this retrieved information as an AR image on display surface 2. For example, if the object is a product, information about that product can be displayed.

[0213] The second example of the AR application is another example of an application that simulates the layout of a room, similar to the fifth embodiment. This AR application virtually moves a selected object image area in response to a predetermined operation by the user. The predetermined operation may be, for example, a finger sliding gesture or a voice command instructing the movement of the object. During this movement, the program either makes the real object image area invisible by superimposing an image such as a fill, or displays it in a way that indicates it is the original object. The AR application displays the object image area while moving it. As a result, the user perceives the object as having moved virtually. Therefore, the user can simulate the placement of objects, etc.

[0214] Figure 31(A) shows the virtual state of object OB1 before movement, and (B) shows the virtual state of object OB1 after movement. The user moves the object image area of ​​object OB1 in (A) to the right. The line of sight area also moves to the right. In the state before movement, the AR application assigns and displays the ID image "1" to object OB1 within the line of sight area E71. The AR application may also display a frame image representing area R1 as the AR image. The user selects the ID image "1" associated with object OB1 and virtually moves object OB1 to the right with a predetermined operation. HMD1 displays a fill image g1 in the area R1 of object OB1 before movement to make it appear as if object OB1 is not there after movement. It is preferable to use the surrounding image to process the fill image g1 to make it appear as a virtual background. HMD1 displays an image g2 representing object OB1 in the area R1b after movement. Image g2 is a type of AR image, specifically an AR image that corresponds to the image region of an object. HMD1 maintains the display of ID image "1" for image g2. Note that image g2 of the region R1b after movement may be the same as the image of object OB1 before movement, or it is more preferable to use an image in which the orientation and size of the object have been adjusted through image processing.

[0215] [Effects etc. (7)] According to Embodiment 7, it is possible to virtually manipulate a real object using an ID image.

[0216] [Note] (1) The video display device automatically enlarges or reduces the line of sight area according to the number or density of generated images within the line of sight area. (2) The video display device shall maintain an ID image containing the same identification information for the same generated image over time. (3) The video display device reassigns an ID image containing identification information to the generated image within the line of sight area at each point in time, according to a predetermined rule. (4) When the generated image contains multiple generated images clustered together, the video display device displays the ID image in a non-clustered state. (5) When multiple generated images form a group as generated images, the video display device assigns an ID image to each group and displays them, and selects multiple generated images from the group according to the selection operation of the group's ID image.

[0217] Although the present invention has been specifically described above based on embodiments, the present invention is not limited to the embodiments described above and can be modified in various ways without departing from the spirit of the invention. Configurations can also be made by combining each embodiment, or by adding, deleting, or replacing the components of the embodiments. The image display device in the embodiment was shown as being applied to an HMD, but it is not limited to this, and any device that has the function of superimposing a generated image on a real image is acceptable, and it can also be applied to HUD devices, etc. The program processing and information processing described in the embodiments may be implemented by general-purpose software program processing, or by dedicated hardware circuits such as FPGAs and ASICs. The program data may be stored on a computer-readable storage medium such as a disk or memory card. The program may be stored at the time of product shipment, or it may be stored on a server or DB on a communication network. The data structure is not limited to a table structure and is applicable. The aforementioned identification information (ID) can be replaced with similar expressions such as identifiers. [Explanation of symbols]

[0218] 1...HMD, 2...Display surface, 3...Speaker, 4...Eye-tracking unit, 5...Camera.

Claims

1. A display device for displaying the generated image, A control device for setting a selection area to assist user operation on the generated image, A gaze detection device for detecting the direction of the user's gaze, Equipped with, The control device sets a line of sight area corresponding to the detected line of sight direction, In response to the user viewing the displayed generated image, if the generated image falls within the line of sight area, The control device controls the selection region to display a selection image associated with the generated image that falls within the line of sight region. When the user makes a selection operation on the selection image, the system executes a predetermined process associated with the generated image related to the selection image. A video display system characterized by the following features.

2. The video display system according to claim 1, The selection image is a reduced version of the generated image. A video display system characterized by the following features.

3. The video display system according to claim 1, The aforementioned selection image is an image that includes part or all of the aforementioned generated image. A video display system characterized by the following features.

4. The video display system according to claim 1, The control device configures an ID image containing identification information to assist user operation on the generated image, and when the generated image falls within the line of sight area, it displays the ID image in association with the generated image in the line of sight area and the selection image in the selection area. A video display system characterized by the following features.

5. The video display system according to claim 1, The control device enlarges or reduces the line of sight area in response to the user's operation. A video display system characterized by the following features.

6. The video display system according to claim 5, The display device is capable of displaying multiple generated images, The control device sets the maximum number of generated images that can be simultaneously displayed within the viewing area, and expands or shrinks the viewing area according to the maximum number. A video display system characterized by the following features.

7. The video display system according to claim 1, The control device provides the selection area on the top, bottom, right, or left edge of the display surface. A video display system characterized by the following features.

8. The video display system according to claim 4, The control device controls the display of the ID image and the selection image of the selected region at positions corresponding to the display positions of the generated image. A video display system characterized by the following features.

9. The video display system according to claim 6, The control device sets the selection area at a location separate from the position where the generated image is displayed, and controls the device to display a line indicating the correspondence between the generated image and the selection image. A video display system characterized by the following features.

10. The video display system according to claim 8, The control device controls the display surface to display the ID image and the selection image at positions corresponding to the positions of the generated image in the left-right or up-down direction. A video display system characterized by the following features.

11. The steps include displaying the generated image and The steps include setting a selection area to assist user interaction with the generated image, The steps include detecting the user's gaze direction and setting a gaze region corresponding to the detected gaze direction, In response to the user viewing the displayed generated image, if the generated image falls within the line of sight area, The steps include controlling the selection region to display a selection image associated with the generated image that falls within the viewing region, When the user makes a selection operation on the selection image, the steps include: executing a predetermined process associated with the generated image related to the selection image; A method for displaying images, comprising the characteristics of an image display method.

Images