Display device and head mount display

Abstract

To provide a display device and a head mount display which can make a user properly visually recognize information by performing a visual line guidance.SOLUTION: A display device makes an image recognize by condensing light corresponding to an image to be displayed at a position between a pupil and a retina of a user. In other words, the display device is a visual line direction-dependent display device. Further, the display device is configured to be able to realize AR additionally presenting information to an actual environment recognized by a user. Visual line guidance means of the display device outputs information for guiding the visual line of a user to a guidance point which is a position to which the visual line of the user should be directed. The guidance point is a position of the information of an object existing in the actual environment or the position of the information displayed to the object existing in the actual environment.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to the technical field of display devices that allow a user to view only an image in the direction in which the user's gaze is directed. [Background technology]

[0002] This type of technology is described, for example, in Patent Document 1. Patent Document 1 describes a display device (hereinafter referred to as a "gaze direction dependent display device") that can display an image only in the direction that the user's gaze is directed by setting the light focusing point corresponding to the image to be displayed approximately at the center of the eyeball. In a gaze direction dependent display device, by setting the light focusing point approximately at the center of the eyeball, only light rays in the gaze direction pass through the pupil, so that only the image in the gaze direction is visible. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2002-90688 Summary of the Invention [Problem to be solved by the invention]

[0004] In the above-mentioned gaze direction dependent display device, the user can see the image in the gaze direction, but cannot see the image in any other direction. As a result, there are cases where the information presented by the display device cannot be provided appropriately to the user.

[0005] The above-mentioned examples are examples of problems that the present invention aims to solve. An object of the present invention is to provide a display device and a head-mounted display that can guide the user's gaze to allow the user to properly view information. [Means for solving the problem]

[0006] The invention described in claim 1 is a display device that focuses light corresponding to an image to be displayed at a position between the user's pupil and retina, and presents additional information to the real environment viewed by the user, and is equipped with a detection means for detecting objects present in the real environment, an audio output unit that outputs audio, and a gaze guidance means that outputs audio to the audio output unit to guide the user's gaze to a guidance point that is the position of the detected object or the position of information displayed for the object.

[0007] The invention described in claim 2 is a display device that focuses light corresponding to an image to be displayed at a position between the user's pupil and retina, and presents additional information to the real environment viewed by the user, and is equipped with a detection means that detects an object present in the real environment, a gaze detection means that detects the user's gaze, and a gaze guidance means that notifies the user of information to guide the user's gaze to a guidance point that is the position of the detected object or the position of information displayed for the object, and the gaze guidance means terminates the notification when the gaze detected by the gaze detection means is directed toward the guidance point.

[0008] The invention described in claim 3 is a display device that focuses light corresponding to an image at a position between a user's pupil and retina, and is equipped with a gaze detection means that detects the user's gaze, and a gaze guidance means that notifies the user of information to guide the user's gaze to a guidance point, which is a position corresponding to an image that should be displayed by the display device outside the viewing area according to the user's gaze direction, and the gaze guidance means terminates the notification when the gaze detected by the gaze detection means is directed toward the guidance point.

[0009] The invention described in claim 4 is a display device that focuses light corresponding to an image to be displayed at a position between the user's pupil and retina, and adds and presents information to the real environment viewed by the user, and is equipped with a detection means that detects an object present in the real environment, a movement detection means that detects movement of the user's head, and a gaze guidance means that notifies the user of information to guide the user's gaze to a guidance point that is the position of the detected object or the position of information displayed for the object, and the gaze guidance means terminates the notification when a predetermined time has elapsed since the movement of the head toward the guidance point was detected by the movement detection means.

[0010] In the invention described in claim 16, the head-up display is characterized by comprising a gaze guidance means for notifying information for guiding the gaze of the user to a guidance point, which is a position corresponding to an image that should be displayed outside the viewing area according to the direction of the user's gaze. [Brief explanation of the drawings]

[0011] [Figure 1] The schematic configuration of a head-mounted display (HMD) is shown. [Figure 2] FIG. 2 is a block diagram showing the configuration of a control unit. [Figure 3] 1A and 1B are diagrams for explaining problems with a conventional line-of-sight direction dependent display device. [Figure 4] FIG. 10 is a diagram illustrating a first display example of a guidance image. [Figure 5] FIG. 10 is a diagram showing a second display example of a guidance image. [Figure 6] FIG. 10 is a diagram illustrating a third display example of a guidance image. [Figure 7] FIG. 10 is a diagram showing a fourth display example of a guidance image. [Figure 8] FIG. 10 is a diagram showing a first example of the end of gaze guidance. [Figure 9] FIG. 10 is a diagram showing a second example regarding the end of gaze guidance. [Figure 10] FIG. 1 shows a diagram for explaining display by an AR system. [Figure 11] Specific examples of visual guidance by area are shown below. DETAILED DESCRIPTION OF THE INVENTION

[0012] In one aspect of the present invention, a display device that focuses light corresponding to an image to be displayed at a position between the user's pupil and retina and presents additional information to the real environment viewed by the user is equipped with a gaze guidance means that displays a guidance image to guide the user's gaze to a guidance point, which is the position of an object present in the real environment or the position of information displayed for an object present in the real environment.

[0013] The display device allows the user to view an image by focusing light corresponding to the image to be displayed at a position between the user's pupil and retina (e.g., approximately the center of the eyeball). In other words, the display device is a gaze direction-dependent display device. The display device is also configured to realize AR (Augmented Reality), which presents additional information to the real environment viewed by the user. Such gaze direction-dependent display devices have a problem in that the user cannot see images other than those in the gaze direction, making it easy for the user to miss information presented by the device. Therefore, the gaze guidance means displays a guidance image to guide the user's gaze to a guidance point, which is a position to which the user should direct their gaze. The guidance point is the position of an object present in the real environment or the position of information displayed relative to an object present in the real environment. The display device allows the user to properly view objects present in the real environment or information displayed relative to the object by guiding the user's gaze.

[0014] In one aspect of the display device, the visual line guidance means displays, as the guidance image, an image that shrinks in size toward the guidance point over time, thereby making it possible to appropriately guide the user's visual line regardless of the area of ​​the display area that the user can see.

[0015] In the above display device, preferably, the gaze guiding means sets an initial size of the guidance image to a size corresponding to the distance between the guidance point and the end position of the display area of ​​the display device farthest from the guidance point, and reduces the size of the guidance image from the initial size. This makes it possible to eliminate gaze directions in which the gaze guidance effect cannot be obtained and to suppress deterioration in the time efficiency of the gaze guidance effect.

[0016] In another aspect of the display device, the visual line guidance means displays, as the guidance image, an image that changes spatially according to the distance to the guidance point, thereby making it possible to appropriately guide the user's visual line regardless of the area of ​​the display area that the user can see.

[0017] In a preferred example, the spatial change is a change in brightness depending on the distance to the induction point. In another preferred example, the spatial change is a change in color depending on the distance to the induction point. In another preferred example, the spatial change is a change in image density depending on the distance to the induction point. For example, the change in image density is a change in the density of lines or points.

[0018] In another aspect of the display device, the visual line guidance unit displays an image indicating the direction of the guidance point as the guidance image, thereby making it possible to appropriately guide the user's visual line regardless of the area of ​​the display area that the user can see.

[0019] In another aspect of the display device, the display device further includes a gaze detection means for detecting the gaze of the user, and the gaze guidance means displays the guidance image only in an area corresponding to the gaze detected by the gaze direction detection means, thereby enabling the guidance image to be displayed in a manner that is easy for the user to view, centered on the gaze direction.

[0020] In another aspect of the display device, the visual guidance means does not display the guidance image in a first area corresponding to a visual recognition area when the user directs his / her gaze to the guidance point. This makes it possible to appropriately prevent the user from feeling discomfort because the guidance image is no longer visible when the user moves his / her gaze to the guidance point.

[0021] Preferably, the line-of-sight guidance means changes the size of the first area according to the size of the user's pupil or the brightness of the user's surroundings, because the pupil diameter changes depending on the brightness of the user's surroundings (brightness of ambient light), and the size of the visual recognition area that defines the first area changes accordingly.

[0022] In another aspect of the display device, the visual guidance means displays an image in a second area surrounding the first area such that the visual guidance effect of the guidance image decreases as the area approaches the first area. For example, the visual guidance means decreases the brightness of the guidance image in the second area as the area approaches the first area. This makes it possible to appropriately deal with fluctuations in the visual recognition area due to changes in pupil diameter caused by brightness of ambient light.

[0023] In another aspect of the display device, the display device further includes a motion detection means for detecting a head movement of the user, and the gaze guidance means stops displaying the guidance image when a predetermined time has elapsed since the motion detection means detected the head movement toward the guidance point. This makes the guidance image invisible when the user turns his or her face toward the guidance point, thereby appropriately preventing the user from feeling discomfort.

[0024] In another aspect of the display device, the display device further includes a gaze detection means for detecting the gaze of the user, and the gaze guidance means stops displaying the guidance image when the gaze detected by the gaze detection means is directed toward the guidance point. This makes it possible to appropriately prevent the user from feeling discomfort when the guidance image is not visible when the user directs their gaze toward the guidance point.

[0025] In another aspect of the display device, the gaze guiding means stops displaying the guidance image when the guidance point is located in front of the user's face. This makes it possible to appropriately prevent the user from feeling discomfort when the user turns his or her face toward the object corresponding to the guidance point, as the guidance image is no longer visible.

[0026] In another aspect of the display device, the visual guidance means can display the guidance image over the entire display area of ​​the display device, thereby eliminating any visual direction in which the visual guidance effect cannot be obtained in the display area.

[0027] In another aspect of the display device, the gaze guidance unit displays the guidance image in a part of the display area of ​​the display device. This makes it possible to prevent the user from viewing guidance images related to information that the user is not interested in, for example. This makes it possible to prevent the user from viewing unnecessary information.

[0028] Preferably, the partial area corresponds to a specific area of ​​the real environment.

[0029] Preferably, the gaze guiding means uses a plurality of the guidance points located in different locations and displays the guidance image for each of the plurality of guidance points, thereby making it possible to appropriately guide the user's gaze to information that should be focused on for each classified space.

[0030] In another aspect of the present invention, a head-mounted display that focuses light corresponding to an image to be displayed at a position between the user's pupil and retina and presents additional information to the real environment viewed by the user is provided with a gaze guidance means that displays a guidance image to guide the user's gaze to a guidance point, which is the position of an object present in the real environment or the position of information displayed for an object present in the real environment.

[0031] In yet another aspect of the present invention, a display method executed by a display device that focuses light corresponding to an image to be displayed at a position between a user's pupil and retina and presents additional information to a real environment viewed by the user includes a gaze guidance step of displaying a guidance image for guiding the user's gaze to a guidance point that is the position of an object present in the real environment or the position of information displayed for an object present in the real environment.

[0032] In yet another aspect of the present invention, a display program executed by a display device that has a computer and focuses light corresponding to an image to be displayed at a position between the user's pupil and retina, and presents additional information to the real environment viewed by the user, causes the computer to function as a gaze guidance means that displays a guidance image to guide the user's gaze to a guidance point, which is the position of an object present in the real environment or the position of information displayed for an object present in the real environment.

[0033] The display program can be conveniently handled in a state in which it is recorded on a recording medium. [Example]

[0034] Preferred embodiments of the present invention will now be described with reference to the drawings.

[0035] (1) First Example First, the first embodiment will be described.

[0036] (1-1) Equipment configuration FIG. 1 shows a schematic configuration of a head-mounted display (hereinafter referred to as "HMD") 10 according to a first embodiment. The HMD 10 mainly comprises a display unit 1, a mask 2, a spherical mirror 3, a half mirror 4, and a control unit 5. By using the spherical mirror 3, the HMD 10 is configured to be able to display an image directly on the retina without using the lens function of the eye. For example, the HMD 10 is configured in the form of glasses, and is worn on the user's head when used. The HMD 10 is an example of a "display device" in the present invention.

[0037] The display unit 1 is configured with an LCD (Liquid Crystal Display), DLP (Digital Light Processing), organic EL, etc., and emits light corresponding to the image to be displayed. Note that the display unit 1 may be configured to scan light from a laser light source with a mirror.

[0038] Light emitted from the display unit 1 is masked by a mask 2, and the light that passes through the mask 2 is reflected by a spherical mirror 3 and a half mirror 4 to reach the eyeball. In this case, the light that is diffused beyond the mask 2 is focused at a position approximately at the center of the eyeball. The optical system of the HMD 10 according to this embodiment is designed so that light corresponding to the image to be displayed is focused at a position approximately at the center of the eyeball. As a result, only light in the direction of the line of sight passes through the pupil, and therefore only the image in the direction of the line of sight (more specifically, a virtual image; the same applies hereinafter) is visible to the user. In other words, the HMD 10 functions as a gaze direction-dependent display device. Details of the functions and actions of a gaze direction-dependent display device are described in, for example, Patent Document 1.

[0039] The control unit 5 has a CPU, RAM, ROM, etc. (not shown), and performs overall control of the HMD 10. As will be described in detail later, the control unit 5 is an example of the "gaze guidance means" in the present invention.

[0040] It should be noted that the present invention is not limited to focusing light corresponding to an image to be displayed at a position approximately at the center of the eyeball. The above-described gaze direction dependent display device can be realized by focusing light at a position between the pupil and the retina, even if the position is not approximately at the center of the eyeball.

[0041] Furthermore, although Figure 1 illustrates a device that uses a spherical mirror 3 as a line-of-sight dependent display device, the present invention can be applied to various devices, such as a device that uses a lens instead of the spherical mirror 3, or a device that uses the reflection characteristics of a hologram element instead of the spherical mirror 3.

[0042] Next, a specific configuration of the control unit 5 according to the first embodiment will be described with reference to Fig. 2. Fig. 2 is a block diagram showing the configuration of the control unit 5 according to the first embodiment. As shown in Fig. 2, the control unit 5 has a guidance point designation unit 5a, a guidance image generation unit 5b, a display image generation unit 5c, and a display synthesis unit 5d. In this embodiment, the control unit 5 performs control to display a guidance image for guiding the user's line of sight to a guidance point, which is a position where the user's line of sight should be directed.

[0043] The guidance point designation unit 5a designates a point (guidance point) to which the user's line of sight is to be guided. For example, the guidance point designation unit 5a designates a position corresponding to an image to be presented to the user by the HMD 10 as the guidance point.

[0044] The guidance image generating unit 5b generates a guidance image for guiding the user's line of sight to a guidance point. Specific examples of the guidance image will be described later.

[0045] The display image generating unit 5c generates an image (an image different from the guidance image) to be presented to the user by the HMD 10. For example, the display image generating unit 5c generates images such as CG (Computer Graphics) and characters.

[0046] The display composition unit 5d outputs to the display unit 1 data of an image obtained by combining the guidance image generated by the guidance image generation unit 5b and the image generated by the display image generation unit 5c.

[0047] (1-2) Problems with the conventional technology Here, referring to Fig. 3, problems with conventional gaze direction dependent display devices (such as the device described in Patent Document 1) will be described. The rectangular areas shown in Figs. 3(a) and (b) indicate the display area of ​​a gaze direction dependent display device (the same applies to the following drawings). Here, an example is shown in which an incoming email notification (see reference numeral 21) is displayed in the center of the display area, and the time (see reference numeral 22) is displayed in the upper right corner of the display area. Furthermore, reference numerals 20a and 20b indicate viewing areas corresponding to the field of view according to the user's gaze direction.

[0048] As shown in Figure 3(a), when the user looks toward the center, the user notices the email notification. However, as shown in Figure 3(b), when the user looks toward the upper right to check the time, the user does not notice the email notification in the center. In this case, even if the email notification is highlighted by flashing, the highlighted display is not in the user's field of vision, so the user does not notice the email notification.

[0049] As described above, conventional gaze direction dependent display devices have the problem that the user cannot see images other than those in the gaze direction, making it easy for the user to miss information presented by the device. Therefore, in the first embodiment, a position corresponding to an image to be presented to the user by the HMD 10 is used as a guidance point, and a guidance image is displayed to guide the user's gaze to the guidance point.

[0050] (1-3) Display example Next, specific examples of guidance images according to Example 1 will be described. Note that the guidance images are generated by the guidance image generating unit 5b of the control unit 5.

[0051] (1-3-1) First display example A first display example of the guidance image will be described with reference to Fig. 4. In Fig. 4(a) to (d), stars 23 indicate guidance points, thick lines 24 (24a to 24d) indicate guidance images, and multiple thin lines indicate the trajectory of the guidance image 24. Specifically, Fig. 4(a) to (d) show the change in the guidance image 24 over time. In this case, it is assumed that time has passed in the order of Fig. 4(a) → Fig. 4(b) → Fig. 4(c) → Fig. 4(d).

[0052] 4, the stars indicating the guidance points 23 are shown for the sake of convenience and are not actually displayed (actually, an image generated by the display image generating unit 5c is displayed). Also, the multiple thin lines indicating the trajectory of the guidance image 24 are shown for the sake of convenience and are not actually displayed. The same applies to the stars and thin lines in the drawings shown below.

[0053] In the first display example, as shown in FIGS. 4(a) to 4(d), the HMD 10 displays, as the guidance image 24, an image of a circle (more specifically, an image showing the curves (circumference, arc) that make up the circle, hereinafter referred to as a "reducing circle") that shrinks in size toward the guidance point 23 over time. That is, the HMD 10 displays a reducing circle that shrinks in size over time, with the guidance point 23 as the convergence point. By displaying such a reducing circle, the user's line of sight can be appropriately guided regardless of the area of ​​the display area that the user can see. This makes it possible to intentionally present information and guide the user's line of sight.

[0054] In the first display example, the HMD 10 starts displaying the reduction circle from the edge of the display area. Specifically, the HMD 10 sets the distance between the guidance point 23 and the edge position of the display area farthest from the guidance point 23 (approximately the lower left position in the example shown in FIG. 4) as the initial radius of the reduction circle, and displays the reduction circle with a radius that gradually decreases from the initial radius. This is because, for example, if the radius of the reduction circle shown in FIG. 4(b) is set as the initial radius, the reduction circle will not be displayed at the lower left edge, and therefore, when the user's gaze is at that edge, the gaze guidance effect will not be obtained. Furthermore, if the initial radius is too large, there will be a time when the reduction circle is not displayed, which can be said to result in poor time efficiency of the gaze guidance effect. Therefore, the HMD 10 sets the initial radius of the reduction circle to the distance between the edge position farthest from the guidance point 23 and the guidance point 23. This makes it possible to eliminate gaze directions in which the gaze guidance effect is not obtained and to suppress deterioration in the time efficiency of the gaze guidance effect.

[0055] Furthermore, if the shrinking speed of the shrinking circle is defined as "δr / δt" (r: radius of the shrinking circle, t: time), in one example, the HMD 10 keeps the shrinking speed roughly constant. This allows the same gaze guidance effect to be generated throughout the entire display area. Furthermore, if the shrinking speed is too fast, the user will not be able to follow the shrinking circle with their eyes, and if the shrinking speed is too slow, it will take too much time to guide the gaze. Therefore, it is preferable to set the shrinking speed taking these factors into consideration. Specifically, it is preferable to set the shrinking speed so that the user can follow the shrinking circle with their eyes and so that the time efficiency of the gaze guidance effect is ensured. In another example, the HMD 10 can speed up the shrinking speed of the shrinking circle in areas where the gaze guidance effect is not required. This can improve the time efficiency of the gaze guidance effect.

[0056] Furthermore, the HMD 10 can repeatedly display the reduced circle. Specifically, after the reduced circle has converged (at which point the reduced circle disappears), the HMD 10 can repeatedly display a reduced circle with the initial radius again and reduce it. This makes it possible to properly guide the line of sight to the guidance point 23 even if the user does not look at the reduced circle for a while due to blinking or the like.

[0057] In the above, an example of shrinking a circular image (shrinking circle) has been shown as the first display example of the guidance image, but the shape of the image to be reduced is not limited to a circle. For example, images of various shapes such as a square, a star, or a character can be reduced toward the guidance point 23. In such a case, the reduction speed described for the shrinking circle can also be applied in the same way. For example, the reduction speed can be defined as the speed of change of the radius of the circle circumscribing the image to be reduced.

[0058] (1-3-2) Second display example In the first display example, an image that changes over time is displayed in the display area as a guidance image, but in the second display example, an image that changes spatially is displayed in the display area as a guidance image. Specifically, in the second display example, an image that changes spatially according to the distance to the guidance point 23 is displayed as a guidance image.

[0059] Fig. 5 is a diagram showing a second display example of the guidance image. As shown in Fig. 5, in the second display example, the HMD 10 displays, as the guidance image, an image whose brightness changes toward the guidance point 23. This image is configured to become brighter as it approaches the guidance point 23 and darker as it moves away from the guidance point 23.

[0060] Note that the guidance image is not limited to an image whose brightness changes toward the guidance point 23, and instead, an image whose color changes toward the guidance point 23 may be displayed as the guidance image. For example, an image whose color becomes more yellow as it approaches the guidance point 23 and whose color becomes more blue as it moves away from the guidance point 23 may be displayed.

[0061] (1-3-3) Third display example In the third display example, an image having a spatial change in the display area is displayed as a guidance image, similar to the second display example. However, while in the second display example, an image whose brightness or color changes toward the guidance point 23 is displayed as a guidance image, in the third display example, an image whose density of line segments changes toward the guidance point 23 is displayed as a guidance image.

[0062] 6 is a diagram showing a third display example of the guidance image. As shown in FIG. 6, in the third display example, the HMD 10 displays, as the guidance image, an image in which the density of line segments changes toward the guidance point 23. This image is composed of line segments extending radially from the guidance point 23. In this image, the density of the line segments increases as the user approaches the guidance point 23, and decreases as the user moves away from the guidance point 23.

[0063] 6, for the sake of convenience, only a portion of the line segments constituting the guidance image is shown, and in reality, many more line segments are displayed. The number of line segments constituting the guidance image is determined, for example, so that at least one line segment is visible in any line-of-sight direction.

[0064] Furthermore, the guidance image is not limited to an image in which the density of line segments changes toward the guidance point 23, and instead, an image in which the density of dots changes toward the guidance point 23 may be displayed as the guidance image. For example, an image in which the density of dots increases as the image approaches the guidance point 23 and decreases as the image moves away from the guidance point 23 may be displayed (the relationship between the distance from the guidance point 23 and the density of dots may be reversed).

[0065] (1-3-4) Fourth display example The fourth display example differs from the first to third display examples in that an image pointing to the direction of the guidance point 23 is displayed as a guidance image. Specifically, in the fourth display example, an arrow image pointing to the direction of the guidance point 23 is displayed as a guidance image.

[0066] 7 is a diagram showing a fourth display example of the guidance image. As shown in FIG. 7, in the fourth display example, the HMD 10 displays, as the guidance image, a plurality of arrow images each pointing toward the guidance point 23. The plurality of arrow images extend radially from the guidance point 23.

[0067] 7 shows only a portion of the arrow images constituting the guidance image for the sake of convenience, and in reality, many more arrow images are displayed. The number of arrow images as guidance images is determined, for example, so that at least one arrow image is visible in any line of sight direction.

[0068] (1-3-5) Fifth display example In the first display example, an image that changes over time in the display area is displayed as the guidance image, and in the second to fourth display examples, an image that has a spatial change in the display area is displayed as the guidance image. In the fifth display example, an image that combines the first display example with the second to fourth display examples, that is, an image that has both a temporal change and a spatial change in the display area, is displayed as the guidance image.

[0069] (1-3-6) Other display examples In the above display example, a guidance image is displayed using one guidance point, but guidance images corresponding to multiple guidance points may be displayed using multiple guidance points. In other words, the above display example can also be applied to cases where guidance is to be given to multiple locations. In one example, circular images (shrinking circles) that shrink in size toward each of multiple guidance points are displayed simultaneously. In another example, the guidance point is changed every certain period of time or every cycle, and a guidance image corresponding to the guidance point at that time is displayed, that is, a time-division display is performed. In yet another example, a guidance image is displayed in each of the areas obtained by dividing the display area horizontally or vertically, that is, a space-division display is performed.

[0070] (1-4) Limiting the display area of ​​guidance images All of the above-described guidance images may be displayed only in a viewing area corresponding to the user's line of sight, rather than in the entire display area. Specifically, by detecting the user's line of sight using a known method, guidance images can be displayed only in a viewing area corresponding to the detected line of sight. This makes it possible to display guidance images that are easy for the user to view, centered on the line of sight. For example, by displaying the arrow image shown in the fourth display example centered on the user's line of sight, the direction the arrow image is pointing can be easily viewed.

[0071] (1-5) End of gaze guidance The guidance image described above basically becomes unnecessary after the user's line of sight is directed to the guidance point 23. Furthermore, if the guidance image is displayed even after the user's line of sight is directed to the guidance point 23, the user may feel uncomfortable. Therefore, it is preferable to end the display of the guidance image (i.e., end the line of sight guidance) after the user's line of sight is directed to the guidance point 23. A specific example of the end of line of sight guidance is shown below.

[0072] The specific examples of the end of gaze guidance shown below are applied to guidance images such as those shown in the first to fourth display examples described above.

[0073] (1-5-1) First example A first example of the end of gaze guidance will be described with reference to Fig. 8. In Fig. 8, area 25a in the display area is an area where a guidance image is not displayed, and area 26a in the display area is an area where a guidance image is displayed. Area 25a corresponds to the visual recognition area when the gaze is directed toward guidance point 23. Area 25a is an example of the "first area" in the present invention.

[0074] 8, in the first example, the HMD 10 does not display a gaze image in the area 25a corresponding to the viewing area when the user's gaze is at the guidance point 23. As a result, when the user moves his / her gaze to the guidance point 23, the guidance image is no longer visible, that is, the display of the guidance image is terminated, so that it is possible to appropriately prevent the user from feeling discomfort. On the other hand, when the user's gaze is different from the guidance point 23, it is possible to appropriately guide the gaze by making the user view the guidance image.

[0075] The size of the area 25a where the guidance image is not displayed may be changed according to the pupil diameter of the user. This is because the size of the viewing area that defines the area 25a varies depending on the pupil diameter. For example, the HMD 10 can detect the pupil diameter of the user using a known method, estimate the viewing area based on the detected pupil diameter, and set the size of the area 25a according to the estimated viewing area.

[0076] Furthermore, because the pupil diameter changes depending on the brightness of the environmental light (the brightness around the user), instead of directly using the pupil diameter as described above, the size of the area 25a where the guidance image is not displayed may be changed according to the brightness of the environmental light. For example, the HMD 10 can detect the brightness of the environmental light, estimate a viewing area corresponding to the pupil diameter based on the detected brightness, and set the size of the area 25a according to the estimated viewing area.

[0077] (1-5-2) Second example The second example differs from the first example in that, taking into consideration fluctuations in the visible area due to changes in pupil diameter caused by the brightness of ambient light, etc., a guidance image is displayed in a surrounding area (hereinafter referred to as a "buffer area") surrounding an area in which a guidance image is not displayed, so that the gaze guidance effect gradually decreases. That is, in the first example, the area 25a in which a guidance image is not displayed and the area 26a in which a guidance image is displayed are used to switch the display of the guidance image between on and off, but in the second example, the buffer area provided between the area in which a guidance image is not displayed and the area in which a guidance image is displayed is used to gradually transition from an on state to an off state of the display of the guidance image.

[0078] Fig. 9 is a diagram showing a second example regarding the end of gaze guidance. In Fig. 9, area 25b in the display area is an area in which a guidance image is not displayed, area 26b in the display area is an area in which a guidance image is displayed, and area 26c in the display area is the buffer area described above. Specifically, a guidance image with a normal gaze guidance effect is displayed in area 26b, and a guidance image with a lower gaze guidance effect than area 26b is displayed in buffer area 26c. Buffer area 26c is an example of the "second area" in the present invention, and its size is set according to the fluctuation range of the viewing area due to changes in pupil diameter caused by the brightness of ambient light, etc.

[0079] In a second example, when the user's line of sight is in the buffer area 26c, the HMD 10 displays a guidance image such that the effect of guiding the line of sight gradually decreases as the line of sight approaches the area 25b where no guidance image is displayed. For example, the HMD 10 reduces the brightness of the guidance image in the buffer area 26c, thins the lines constituting the guidance image, and increases the speed at which the guidance image is reduced compared to the area 26b.

[0080] (1-5-3) Third example In the third example, the HMD 10 ends the display of the guidance image when a predetermined time has elapsed since the guidance image was displayed. Such a third example may be implemented instead of the first and second examples described above, or may be implemented in combination with the first or second example.

[0081] (1-5-4) Fourth example In a fourth example, the HMD 10 ends the display of the guidance image when a predetermined time has elapsed since the user's head moved toward the guidance point 23. Specifically, in the fourth example, the HMD 10 detects the user's head movement based on the output of sensors such as a camera, an acceleration sensor, and a gyro sensor. Then, when the HMD 10 detects the head movement toward the guidance point 23, it ends the display of the guidance image when a predetermined time has elapsed since the detection.

[0082] The fourth example may be implemented in place of the first to third examples described above, or may be implemented in appropriate combination with the first to third examples.

[0083] (1-5-5) Fifth example In the fifth example, the HMD 10 ends the display of the guidance image when the user's line of sight is directed toward the guidance point 23. Specifically, in the fifth example, the HMD 10 detects the user's line of sight using a known method, and ends the display of the guidance image when the detected line of sight is directed toward the guidance point 23.

[0084] The fifth example may be implemented in place of the first to fourth examples described above, or may be implemented in appropriate combination with the first to fourth examples.

[0085] (2) Second Example Next, a second embodiment will be described. In the second embodiment, the configuration for guiding the user's line of sight to the guidance point 23, as shown in the first embodiment, is applied to an AR (Augmented Reality) configuration that presents additional information such as CG and text in the real environment. Hereinafter, a configuration that can realize AR will be referred to as an "AR system" as appropriate.

[0086] With a gaze direction dependent display device (HMD 10) such as that shown in Fig. 1, the half mirror 4 allows the user to simultaneously view the real environment ahead and the displayed image. Therefore, by providing a means (such as a camera) for detecting an object that actually exists in the real environment, it is possible to display an image that matches the object. In other words, an AR system can be realized.

[0087] A specific example of display by the AR system will be described with reference to FIG. 10. FIG. 10(a) shows images 30a to 30c displayed by the display unit 1 of the HMD 10, and FIG. 10(b) shows the image visually recognized in the viewing area 31 when the images 30a to 30c are displayed. In this case, only the image 30b is visually recognized. On the other hand, FIG. 10(c) shows objects 32a to 32c existing in the real environment, and FIG. 10(d) shows the image visually recognized by the AR system. Specifically, FIG. 10(d) shows the image visually recognized in the viewing area 31 as shown in FIG. 10(b) when the images 30a to 30c as shown in FIG. 10(a) are displayed in the real environment as shown in FIG. 10(c). As shown in FIG. 10(d), the AR system can display an image 30b that matches an object 32b existing in the real environment.

[0088] In the above-mentioned AR system, when the gaze direction dependent display device (HMD 10) is worn on the head, the position of the guidance point 23 on the display changes depending on the movement of the head, etc. Below, a specific example of gaze guidance in the AR system is shown.

[0089] (2-1) Gaze guidance in AR systems In an AR system using a gaze direction-dependent display device, it is possible to display information (images) by aligning the positions of objects existing in the real environment with those of the object. It is also possible to align the guidance point 23 with the position of an object existing in the real environment. Therefore, when the configuration shown in the first embodiment (specifically, the configuration for displaying a guidance image for guiding the line of sight to the guidance point 23) is applied to the AR system, the position of an object existing in the real environment or the position of display information (images) associated with an object existing in the real environment can be set as the guidance point 23, and a guidance image for guiding the line of sight to such guidance point 23 can be displayed. For example, guidance images such as those shown in the first to fourth display examples of the first embodiment can be displayed in a similar manner.

[0090] (2-2) End of gaze guidance in AR systems In the AR system, too, the guidance image becomes unnecessary after the user's gaze is directed toward the guidance point 23. Furthermore, if the guidance image is displayed even after the user's gaze is directed toward the guidance point 23, the user may feel uncomfortable. Therefore, in the AR system, too, it is preferable to terminate the display of the guidance image after the user's gaze is directed toward the guidance point 23.

[0091] As described above, in an AR system using a gaze direction-dependent display device, the position of an object in the real environment or the position of display information (image) associated with the object in the real environment is set as a guidance point 23, and a guidance image is displayed to guide the gaze to the guidance point 23. Here, when a user focuses on an object in the real environment, the user tends to turn his or her face toward the object. Therefore, in an AR system using a gaze direction-dependent display device, when the guidance point 23 linked to the position of an object in the real environment moves to a position close to the front of the user's face, it is determined that the user's gaze is directed toward the guidance point 23, and the display of the guidance image is terminated. This makes it possible to achieve both appropriate gaze guidance and suppression of user discomfort. In one example, it is possible to determine whether the guidance point 23 linked to the position of the object has moved to a position close to the front of the user's face, based on an image captured by a camera capturing the real environment in front of the user.

[0092] Note that various methods for terminating gaze guidance, such as those described in the section "(1-5) Terminating gaze guidance," may be applied to AR systems using gaze direction-dependent display devices.

[0093] (2-3) Area-specific visual guidance By providing multiple areas within the display area corresponding to specific areas in the real environment and displaying a guidance image for each area, it is possible to prevent the user's gaze from being guided to information that is unnecessary for the user. This makes it possible to achieve both efficient gaze guidance and prevention of the provision of unnecessary information to the user.

[0094] FIG. 11 shows a specific example of gaze guidance by area. FIG. 11 illustrates a case where two areas 40a and 40b corresponding to two guidance points 23a and 23b are provided in a display area. In the areas 40a and 40b, guidance images 42a and 42b similar to the guidance image 24 shown in the first display example of the first embodiment are displayed. Specifically, in the areas 40a and 40b, circular images (reduced circles) that shrink in size toward the guidance points 23a and 23b over time are displayed as the guidance images 42a and 42b. Note that the guidance images 42a and 42b are displayed only in the areas 40a and 40b, respectively, and are not displayed overlapping each other.

[0095] For example, guidance images 42a and 42b as shown in FIG. 11 are effective when applied to a space where items are separated by type, such as a mall or supermarket, and there are objects or information that the user should look at. In one example, in a mall where a watch store and a clothing store are side by side, area 40a can be used for the watch store, and guidance point 23a can be the location of the product that the user should look at in the watch store. Area 40b can be used for the clothing store, and guidance point 23b can be the location of the product that the user should look at in the clothing store. In this case, if a user is interested in watches and directs their gaze toward the watch store, the viewing area 43 will enter the area 40a for the watch store, causing the guidance image 42a for the watch store to be viewed. As a result, the user's gaze will be guided to the guidance point 23a for the watch store, and the user will pay attention to the product at guidance point 23a. In this case, the user cannot see guidance image 42b for the clothing store, and therefore the user's gaze will not be guided to guidance point 23b for the clothing store.

[0096] As described above, by displaying guidance images 42a and 42b for each area 40a and 40b, it is possible to appropriately guide the user's gaze to objects or information that should be focused on for each classified space without detecting the user's line of sight, etc. Furthermore, since guidance images to objects or information that the user is not interested in are not visible to the user, it is possible to appropriately prevent the user from feeling uncomfortable, etc.

[0097] The display shown in FIG. 11 can be realized by "information on the three-dimensional real environment" and "means for detecting the position and orientation relationship between the image and the real environment." The "information on the three-dimensional real environment" includes three-dimensional area information on the watch shop or clothing store, the three-dimensional positions of guidance points, and display information. The "means for detecting the position and orientation relationship between the image and the real environment" can be realized, for example, by using an image captured by a camera that captures the real environment in front of the user. By using such a "means for detecting the position and orientation relationship between the image and the real environment," the "information on the three-dimensional real environment" can be displayed in association with each area.

[0098] 11 shows an example in which two areas 40a, 40b are provided in the display area to display two guidance images 42a, 42b, but three or more areas may be provided in the display area to display three or more guidance images.Furthermore, it is not limited to providing a plurality of areas in the display area to display a plurality of guidance images, and only one area (a part of the display area) may be provided in the display area to display a guidance image only in that area.

[0099] (3) Variations Although the above example shows an example in which a guidance image is used to guide the user's gaze, the present invention is not limited to using a guidance image. In another example, instead of a guidance image, the user's gaze can be guided using blinking LEDs, vibration, sound, or the like. For example, a display device can be provided with multiple LEDs, and LEDs located in positions corresponding to the direction in which the gaze should be guided can be blinked, or a portion of the display device corresponding to the direction in which the gaze should be guided can be vibrated, or sound can be generated from a portion of the display device corresponding to the direction in which the gaze should be guided.

[0100] The present invention is not limited to application to the HMD 10. The present invention can also be applied to display devices such as a head-up display (HUD). [Industrial Applicability]

[0101] The present invention can be used in display devices such as head-mounted displays and head-up displays. [Explanation of symbols]

[0102] 1 Display 2. Mask 3 Spherical mirror 4 Half mirror 5. Control section 5a Guidance point designation section 5b Guidance image generation unit 5c Display image generation section 5d display composition section 10 Head-mounted display (HMD)

Claims

[Claim 1] A display device that focuses light corresponding to an image to be displayed at a position between a user's pupil and a retina, and presents additional information to a real environment visually recognized by the user, a detection means for detecting an object present in the real environment; an audio output unit that outputs audio; A display device characterized by comprising: a gaze guidance means for outputting, to the audio output unit, audio for guiding the user's gaze to a guidance point, which is the position of the detected object or the position of information displayed for the object.

Images