Head-mounted display apparatus, image display control apparatus, and image display control program

Abstract

To suppress deterioration in image quality according to the position of an observer eye.SOLUTION: The head-mounted display apparatus includes: a detection unit for detecting the position of eyes of a user; an eyepiece lens; a first display panel for displaying a first image; a second display panel disposed closer to the eyepiece lens than the first display panel for displaying a second image; and a control unit for moving at least one of the first image and the second image on the basis of a change in the eye position detected by the detection unit.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a head-mounted display device, a video display control device, and a video display control program. [Background technology]

[0002] In recent years, light field technology, which controls the light rays emitted from a display device and incident on the viewer's eye, has been attracting attention. Humans can see an object when the light rays reflected from the object enter the pupil. Light field technology artificially reproduces the reflected light rays on a display device, allowing the viewer to see the image with a sense of depth (three-dimensionality) by adjusting the focus in the same way that humans see objects in nature.

[0003] Light field technology is also known as a ray space. The light field is described by a Plenoptic function, which is expressed as a five-dimensional function including the position and direction of the array camera. However, a method is usually used to record and represent light rays in a space called a 4D light field, which reduces the position dimension by one by arranging the array camera on a plane. This makes it possible to reproduce 3D images and arbitrarily focused images.

[0004] There are several types of head-mounted display devices that use light field technology, and representative examples include a microlens array type and a stacked type. The microlens array method attaches an array of lenses with diameters on the order of micrometers to the surface of a display panel. In this method, the microlenses separate the light emitted from the display panel into bundles of light rays in specific directions, and direct the light to the viewer's eyes. The microlens array method has the disadvantage that, in principle, a decrease in display resolution is unavoidable, as one microlens covers multiple display pixels. The stacking method has the disadvantage of requiring careful alignment between the first and second display panels, but has the advantage of easily manufacturing a display device that creates a sense of depth (three-dimensional effect) simply by stacking multiple existing display panels.

[0005] As a stacked type head mounted display, for example, the technology described in Patent Document 1 is known. The head-mounted display described in Patent Document 1 has a structure in which two display panels are stacked for each eye with a medium layer sandwiched between them. The first display panel located farther from the viewer is equipped with a backlight and has the function of emitting light. The second display panel located closer to the viewer does not have the function of emitting light and is illuminated by the light emitted from the first display panel. In the stacked type, light emitted from a pixel on the first display panel passes through pixels on the second display panel before entering the viewer's eye, so the light must be controlled by the combination of pixels through which it passes. For this reason, stacked type head-mounted displays have the advantage of not reducing display resolution. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Special Publication No. 2020-521174 Summary of the Invention [Problem to be solved by the invention]

[0007] However, in a stacked head-mounted display, because light rays are incident on the observer's pupil through a combination of desired pixels on the first and second display panels, if the observer's eye position deviates from a predetermined range or if the observer's eye position changes while viewing the image, the image cannot be viewed correctly. Furthermore, if the range of the observer's eye that can view the image is widened, the reproducibility of the image will decrease.

[0008] The present disclosure has been made in consideration of these circumstances, and aims to provide a head-mounted display device, an image display control device, and an image display control program that can suppress degradation in image quality depending on the position of the observer's eyes. [Means for solving the problem]

[0009] The present disclosure has been made to solve the above-mentioned problems, and one aspect of the present disclosure is a head-mounted display device comprising: a detection unit that detects the position of a user's eye; an eyepiece; a first display panel that displays a first image; a second display panel that is positioned closer to the eyepiece than the first display panel and that displays a second image; and a control unit that moves at least one of the first image and the second image based on changes in the eye position detected by the detection unit.

[0010] Another aspect of the present disclosure is an image display control device for a head-mounted display device comprising: a detection unit that detects the position of a user's eyes; an eyepiece; a first display panel that displays a first image; and a second display panel that is positioned closer to the eyepiece than the first display panel and that displays a second image, the image display control device comprising: a control unit that causes the first display panel to display the first image and the second display panel to display the second image, and that controls the movement of at least one of the first image and the second image based on changes in the position of the eyes detected by the detection unit.

[0011] Another aspect of the present disclosure is an image display control program that causes a computer of a head-mounted display device that includes a detection unit that detects the position of a user's eyes, an eyepiece, a first display panel that displays a first image, and a second display panel that is positioned closer to the eyepiece than the first display panel and displays a second image to perform the following steps: detecting the position of the user's eyes; displaying the first image on the first display panel; displaying the second image on the second display panel; and moving at least one of the first image and the second image based on changes in the position of the eyes detected by the detection unit. [Effects of the Invention]

[0012] According to one aspect of the present invention, it is possible to suppress degradation in image quality depending on the position of the observer's eyes. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a diagram schematically showing a head-mounted display device 1 according to an embodiment. [Figure 2] FIG. 1 is a block diagram showing an example of a functional configuration of a head-mounted display device 1 according to an embodiment. [Figure 3] 2A to 2C are diagrams illustrating a virtual image and a reproduced image displayed by the head mounted display device 1 according to the embodiment. [Figure 4] FIG. 10 is a diagram for explaining a first method according to the embodiment. [Figure 5] FIG. 1 is a diagram showing an example of a head-mounted display device 1 for realizing a first method in an embodiment. [Figure 6] FIG. 10 is a diagram for explaining another example of the first method in the embodiment. [Figure 7] FIG. 10 is a diagram showing another example of the head-mounted display device 1 for realizing the first method in the embodiment. [Figure 8]FIG. 10 is a diagram for explaining a second method according to the embodiment. [Figure 9] FIG. 10 is a diagram showing an example of a head-mounted display device 1 for realizing a second method in the embodiment. [Figure 10] FIG. 10 is a diagram for explaining a third method according to the embodiment. [Figure 11] FIG. 10 is a diagram showing an example of a head-mounted display device 1 for realizing a third method in the embodiment. [Figure 12] FIG. 10 is a diagram for explaining an observation range in the embodiment. [Figure 13] FIG. 3 is a diagram showing an example of an operation procedure of the head mounted display device 1 in the embodiment. [Figure 14] FIG. 1A is a perspective view showing a multi-view camera and a target object, and FIG. 1B is a diagram showing nine images of the object captured synchronously by nine multi-view cameras. [Figure 15] FIG. 2 is a diagram showing an example of a three-dimensional image (VR image) of a target object constructed in a virtual space in the embodiment. [Figure 16] FIG. 10 is a diagram showing a light field image generated from 25 simultaneously captured images in an embodiment. [Figure 17] 1A and 1B show images two-dimensionally compressed by the NMF method in the embodiment, where (A) shows an image to be displayed on the first display panel 104 and (B) shows an image to be displayed on the second display panel 106. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, a head-mounted display device, an image display control device, and an image display control program to which the present invention is applied will be described with reference to the drawings.

[0015] 1 is a diagram schematically illustrating a head-mounted display device 1 according to an embodiment. The head-mounted display device 1 is a stacked light field type head-mounted display device. The head-mounted display device 1 includes a housing 100, backlights 102R and 102L, first display panels 104R and 104L, second display panels 106R and 106L, right-eye eyepieces 108R and 108L, and right-eye detection units 110R and 110L.

[0016] The backlights 102R and 102L emit illumination light. The illumination light emitted from the backlights 102R and 102L is incident on the right and left eyes of the viewer via the first display panel 104R and 104L, the second display panel 106R and 106L, and the right eyepiece 108R and left eyepiece 108L.

[0017] The first display panel 104R and the first display panel 104L display a first image. The first display panel 104R and the first display panel 104L are liquid crystal display panels. The first display panel 104R is attached in front of the right eye of the viewer. The first display panel 104L is attached in front of the left eye of the viewer. Backlights 102R and 102L are attached to the surfaces of first display panels 104R and 104L opposite the viewer side. In first display panels 104R and 104L, a liquid crystal layer and a color filter are arranged in this order from the backlight 102R and backlight 102L side to the viewer side. The first display panels 104R and 104L transmit the illumination light emitted from the backlights 102R and 102L and guide it to the second display panels 106R and 106L.

[0018] The second display panels 106R and 106L are arranged closer to the right eyepiece 108R and left eyepiece 108L than the first display panels 104R and 104L, and display a second image. The second display panel 106R and the second display panel 106L are liquid crystal display panels. The second display panel 106R is attached in front of the viewer's right eye. The second display panel 106L is attached in front of the viewer's left eye. The second display panel 106R and the second display panel 106L have a liquid crystal layer and a color filter arranged in this order from the first display panel 104R and the first display panel 104L side toward the viewer side. The second display panel 106R and the second display panel 106L transmit illumination light incident from the backlight 102R and the backlight 102L via the first display panel 104R and the first display panel 104L and guide the light to the right eyepiece 108R and the left eyepiece 108L. As a result, the user observes through the right eyepiece 108R and the left eyepiece 108L, and thus observes virtual images of the display images (first images) on the first display panel 104R and the first display panel 104L, and the display images (second images) on the second display panel 106R and the second display panel 106L, respectively.

[0019] The first display panels 104R and 104L and the second display panels 106R and 106L have rectangular or square shapes of the same dimensions when viewed from the front. The first display panels 104R and 104L and the second display panels 106R and 106L are arranged so that their four sides coincide with each other when viewed from the front of the head mounted display device 1 as seen from the side of the second display panels 106R and 106L.

[0020] The first display panels 104R and 104L and the second display panels 106R and 106L may have the same pixel pitch. If the aperture sizes of the pixels of the first display panels 104R and 104L and the pixels of the second display panels 106R and 106L are the same, the same manufacturing process can be used for the first display panels 104R and 104L and the second display panels 106R and 106L, which has the advantage of reducing the manufacturing cost of the head mounted display device 1.

[0021] The first and second display panels 104R and 104L and 106R and 106L are arranged with a medium layer sandwiched between them. That is, the first and second display panels 104R and 104L and 106R and 106L are arranged parallel to each other at a distance corresponding to the thickness of the medium layer. A typical example of the material for the medium layer is air, but the material is not limited thereto. It may be a layer made of argon (refractive index: 1.000281), carbon dioxide (refractive index: 1.000449), helium (refractive index: 1.000036), hydrogen (refractive index: 1.000140), nitrogen (refractive index: 1.000297), oxygen (refractive index: 1.000276), or a mixture thereof, which have refractive indices similar to those of air (refractive index: 1.0002926). Furthermore, the medium layer may be a vacuum layer (refractive index: 1.0000). The use of a medium layer with a high refractive index has the advantage of enabling a smaller size by narrowing the distance between the first display panels 104R, 104L and the second display panels 106R, 106L. In addition, by eliminating oxygen and water vapor, oxidation of device components and the growth of mold can be suppressed.

[0022] The right eyepiece 108R is attached in front of the observer's right eye. The left eyepiece 108L is attached in front of the observer's left eye. The right eyepiece 108R and the left eyepiece 108L transmit illumination light incident from the backlight 102R and the backlight 102L via the first display panel 104R and the first display panel 104L and the second display panel 106R and the second display panel 106L, and guide the light to the right eye and the left eye, respectively. This allows the observer to view the image displayed in front of them.

[0023] The right eye detector 110R detects the position of the user's right eye, and the left eye detector 110L detects the position of the user's left eye.

[0024] In the head mounted display device 1 of the present embodiment, two display panels are arranged for one eye, but the present invention is not limited to this, and two display panels may be arranged for both eyes.

[0025] FIG. 2 is a block diagram showing an example of a functional configuration of the head mounted display device 1 according to the embodiment. The head mounted display device 1 includes, for example, a control unit 120 and a video signal output unit 130. The control unit 120 is realized by, for example, a processor such as a CPU (Central Processing Unit) executing a video display control program stored in a program memory. The control unit 120 moves at least one of the first image and the second image based on a change in the eye position detected by the right eye detection unit 110R and the left eye detection unit 110L. The control unit 120 controls a configuration for moving the first image and the second image, as will be described later. The video signal output unit 130 outputs a video signal for displaying on the first display panel 104R, the first display panel 104L, the second display panel 106R, and the second display panel 106L.

[0026] The control unit 120 may function as an image display control device that causes the first display panel 104 to display the first image and the second display panel 106 to display the second image, and controls the movement of at least one of the first image and the second image based on the change in the eye position detected by the detection unit 110. The image display control device is realized by, for example, an IC chip, and is incorporated into another head mounted display device 1 for use.

[0027] 3 is a diagram illustrating a virtual image and a reconstructed image displayed by the head mounted display device 1 according to the embodiment. Note that Fig. 3 shows the relationship between the reconstructed image and either the right or left eye of the observer. In the following description, when referring collectively to the first display panel 104R and the first display panel 104L, they will simply be referred to as the "first display panel 104", when referring collectively to the second display panel 106R and the second display panel 106L, they will simply be referred to as the "second display panel 106", and when referring collectively to the right eye detection unit 110R and the left eye detection unit 110L, they will simply be referred to as the "detection unit 110".

[0028] The head mounted display device 1 outputs a first video signal to the first display panel 104 and outputs a second video signal to the second display panel 106. As a result, the head mounted display device 1 displays a first virtual image on the first display panel 104 and a second virtual image on the second display panel 106. The head mounted display device 1 allows the viewer to view a reproduced image by superimposing the first virtual image and the second virtual image.

[0029] In the head-mounted display device 1, the depth and accuracy of the reproduced image (displayed image) deteriorate due to the movement of the observer's eyes. The head-mounted display device 1 is designed to obtain an image with depth and a sense of three-dimensionality by passing illumination light through two display panels, a first display panel 104 and a second display panel 106, and reproducing a light field (a group of rays) equivalent to the light rays incident on the observer's pupil within the observer's pupil. In particular, the range known as the central viewing angle is the angular range of the field of view in which the shape and color of an object can be clearly seen, and is generally said to be about 1 to 2 degrees. In addition, the size of the human pupil is about 5 mm, and the device is designed to reproduce a group of rays within this range of angles and positions, or a slightly wider range. This is because the wider the range, the lower the accuracy of the reproduction of the group of rays, so the device is designed to reproduce the group of rays within the minimum range.

[0030] Consider the effect on the image quality of the reproduced image that occurs when the position of the observer's eyes moves from the observation range of the head mounted display device 1.

[0031] The virtual image on the first display panel 104 and the virtual image on the second display panel 106 are located at different positions, and the reconstructed image is visible from the observation range by a group of light rays including light rays passing through the first display panel 104 and light rays passing through the second display panel 106. In other words, the correct reconstructed image can be recognized by placing the eye within the designed observation range.

[0032] When the observer's eye position P1 is within the observation range, the observer can view the reconstructed image. In contrast, when the observer's eye position moves rightward by m from P1 to P2 and goes outside the observation range, the observer has difficulty observing the reconstructed image. Possible reasons for the observer's eyes moving from P1 to P2 include improper attachment of the head-mounted display device 1, individual differences such as interocular distance between observers, and misalignment of the head-mounted display device 1 attached to the observer's head after attachment. Misalignment after attachment, in particular, is a major problem because it cannot be prevented by adjusting the device when attached.

[0033] In contrast, the head mounted display device 1 of the embodiment solves the problem that light rays from the first display panel 104 and the second display panel 106 do not properly enter the eyes of the observer due to movement of the observer's eyes. The first, second, and third methods will be described below. Note that the first, second, and third methods are explanations of the virtual images displayed by the first display panel 104 and the second display panel 106, and are different from the actual travel distance of the images displayed on the first display panel 104 and the second display panel 106.

[0034] FIG. 4 is a diagram for explaining the first method according to the embodiment. The first method is to move the virtual image on the second display panel 106 based on a change in the eye position, so that the reproduced image can be observed even if the observer's eyes move. The control unit 120 changes the position of the virtual image displayed on the second display panel 106 based on the change in the eye position detected by the detection unit 110.

[0035] Suppose the virtual image (display pattern) on the second display panel 106 is moved a distance n to the left. The reconstructed image reproduced by the virtual image on the first display panel 104 and the virtual image on the second display panel 106 shifts to the left, and the observation range of the reconstructed image also shifts to the left from R1 to R2.

[0036] If the shift distance of the observation range is m, the distance from the observation range to the first display panel 104 is L1, and the distance from the observation range to the second display panel 106 is L2, then the relationship shown in the following formula 1 holds. n=m·(L1-L2) / L1 (Equation 1) The movement distance n of the virtual image on the second display panel 106 may be a value calculated based on the movement distance m of the observation range. According to Equation 1, when the movement distance of the viewer's eyes is m, the viewer can view the reconstructed image by moving the virtual image on the second display panel 106 to the left by a distance n.

[0037] FIG. 5 is a diagram showing an example of the head mounted display device 1 for realizing the first method according to the embodiment. The head mounted display device 1 includes a movement mechanism 112R that moves the second display panel 106R in the left-right direction, and a movement mechanism 112L that moves the second display panel 106L in the left-right direction. The movement mechanism 112R and the movement mechanism 112L have mechanical structures such as motors and gear mechanisms for moving the second display panel 106R and the second display panel 106L in the left-right direction. The control unit 120 controls the movement mechanism 112R based on the movement distance m of the right eye detected by the detection unit 110, thereby moving the virtual image on the second display panel 106R to the left by a movement distance n.

[0038] The first method has the advantage of being simple, since it is sufficient to move only the virtual image on the second display panel 106. On the other hand, the first method has the disadvantage that the light rays are reconstructed at an angle different from the design, so that the distortion of the reconstructed image increases as the moving distance of the virtual image increases.

[0039] FIG. 6 is a diagram for explaining another example of the first method according to the embodiment. In the first method described above, the virtual image on the second display panel 106 is moved, but this is not limiting, and the virtual image on the first display panel 104 may be moved. That is, the first method moves the virtual image on the first display panel 104 based on a change in the eye position, thereby enabling the viewer to observe the reproduced image even if the viewer's eyes move. The control unit 120 changes the position of the virtual image displayed on the first display panel 104 based on the change in the eye position detected by the detection unit 110.

[0040] Suppose the virtual image (display pattern) on first display panel 104 is moved to the right (negative) by a distance O. In this embodiment, a movement to the left is represented as a positive value, and a movement to the right is represented as a negative value. The reconstructed image reproduced by the virtual image on first display panel 104 and the virtual image on second display panel 106 shifts to the left, and the observation range of the reconstructed image also shifts to the left, from R1 to R2.

[0041] If the shift distance of the observation range is m, the distance from the observation range to the first display panel 104 is L1, and the distance from the observation range to the second display panel 106 is L2, then the relationship shown in the following equation 2 holds. -O=m·(L1-L2) / L1 (Equation 2) The moving distance (-O) of the virtual image on the first display panel 104 may be a value calculated based on the moving distance m of the observation range. Equation 2 indicates that when the movement distance of the observer's eyes is m, the observer can view a reconstructed image by moving the virtual image on the first display panel 104 by −O.

[0042] FIG. 7 is a diagram showing another example of the head mounted display device 1 for realizing the first method according to the embodiment. The head mounted display device 1 includes a movement mechanism 114R that moves the first display panel 104R in the left-right direction, and a movement mechanism 114L that moves the first display panel 104L in the left-right direction. The movement mechanism 114R and the movement mechanism 114L have mechanical structures such as motors and gear mechanisms for moving the first display panel 104R and the first display panel 104L in the left-right direction. The control unit 120 controls the movement mechanism 114R based on the movement distance m of the right eye detected by the detection unit 110, thereby moving the virtual image on the first display panel 104R by a movement distance -O.

[0043] This first method has the advantage of being simple, since it is only necessary to move the virtual image on the first display panel 104. However, it has the disadvantage that the light rays are reproduced at an angle different from the design, and therefore the distortion of the reproduced image increases as the moving distance of the virtual image increases.

[0044] FIG. 8 is a diagram for explaining the second method according to the embodiment. The second method moves both the virtual image on the first display panel 104 and the virtual image on the second display panel 106 based on a change in the eye position, allowing the viewer to view the reproduced image even if the viewer's eyes move. The control unit 120 changes the positions of the virtual images displayed on the first display panel 104 and the second display panel 106 based on the change in the eye position detected by the detection unit 110.

[0045] Suppose the virtual image on the second display panel 106 is moved to the left by a distance n, and the virtual image on the first display panel 104 is moved to the left by a distance o. The reconstructed image reproduced by the virtual image on the first display panel 104 and the virtual image on the second display panel 106 shifts to the left, and the observation range of the reconstructed image also shifts to the left from R1 to R2.

[0046] If the shift distance of the observation range is m, the distance from the observation range to the first display panel 104 is L1, and the distance from the observation range to the second display panel 106 is L2, then the relationship shown in the following equation 3 holds. no=(mo)·(L1-L2) / L1 (Equation 3) The movement distance n of the virtual image on the second display panel 106 may be a value calculated based on the movement distance m of the observation range. The movement distance o of the virtual image on the first display panel 104 may be a value calculated based on the movement distance m of the observation range. Equation 3 indicates that when the movement distance of the observer's eye is m, the observer can observe the reconstructed image by moving the virtual image on the second display panel 106 to the left by a distance n and by moving the virtual image on the first display panel 104 to the left by a distance o.

[0047] FIG. 9 is a diagram showing an example of the head mounted display device 1 for realizing the second method according to the embodiment. The head mounted display device 1 includes a movement mechanism 112R that moves the second display panel 106R in the left-right direction, a movement mechanism 112L that moves the second display panel 106L in the left-right direction, a movement mechanism 114R that moves the first display panel 104R in the left-right direction, and a movement mechanism 114L that moves the first display panel 104L in the left-right direction. The movement mechanisms 112R, 112L, 114R, and 114L have mechanical structures such as motors and gear mechanisms for moving the second display panel 106R and the second display panel 106L, and the first display panel 104R and the first display panel 104L in the left-right direction. The control unit 120 controls the movement mechanisms 112R, 112L, 114R, and 114L based on the movement distance m of the right eye detected by the detection unit 110, thereby shifting the virtual image of the second display panel 106R, the virtual image of the second display panel 106L, the virtual image of the second display panel 106R, and the virtual image of the first display panel 104L to the left.

[0048] According to the second method, the change in angle from the design time can be made smaller than when only the second display panel 106 or only the first display panel 104 is moved, and therefore distortion of the reproduced image can be suppressed compared to the first method.

[0049] FIG. 10 is a diagram for explaining the third method according to the embodiment. The third method moves both the virtual image on the first display panel 104 and the virtual image on the second display panel 106 in the same direction and by the same distance based on a change in eye position, allowing the viewer to view the reproduced image even if the viewer's eyes move. The control unit 120 changes the positions of the virtual images displayed on the first display panel 104 and the second display panel 106 based on the change in eye position detected by the detection unit 110.

[0050] Suppose the virtual image on the second display panel 106 is moved to the left by a distance n, and the virtual image on the first display panel 104 is moved to the left by a distance O. The reconstructed image reproduced by the virtual image on the first display panel 104 and the virtual image on the second display panel 106 shifts to the left, and the observation range of the reconstructed image also shifts to the left from R1 to R2.

[0051] If the shift distance of the observation range is m, the distance from the observation range to the first display panel 104 is L1, and the distance from the observation range to the second display panel 106 is L2, then the relationship shown in the following equation 3 holds. no=(mo)·(L1-L2) / L1 (Equation 3) The movement distance n of the virtual image on the second display panel 106 is the same as the movement distance m of the observation range, and the movement distance O of the virtual image on the first display panel 104 is the same as the movement distance m of the observation range. According to Equation 3, when the movement distance of the observer's eye is m, the observer can observe the reconstructed image by moving the virtual image on the second display panel 106 to the left by a distance n that is the same as the distance m, and by moving the virtual image on the first display panel 104 to the left by a distance O that is the same as the distance m.

[0052] FIG. 11 is a diagram showing an example of the head mounted display device 1 for realizing the third method according to the embodiment. The head-mounted display device 1 includes movement mechanisms 116R and 116L that move the first display panel 104R and the first display panel 104L and the second display panel 106R and the second display panel 106L in the left-right direction in conjunction with each other. The movement mechanisms 116R and 116L have mechanical structures such as motors and gear mechanisms for moving the first display panel 104R and the first display panel 104L and the second display panel 106R and the second display panel 106L in the left-right direction. The control unit 120 controls the movement mechanisms 116R and 116L based on the movement distance m of the right eye detected by the detection unit 110, thereby shifting the virtual image of the second display panel 106R, the virtual image of the second display panel 106L, the virtual image of the first display panel 104R, and the virtual image of the first display panel 104L in the left-right direction.

[0053] According to the third method, by moving second display panel 106 and first display panel 104 by the same distance as eye movement distance m, the reproduced image can be viewed at the same angle as at the time of design even if the eye moves, and distortion of the reproduced image can be suppressed compared to the second method. On the other hand, the third method requires a longer movement distance of first display panel 104 and second display panel 106 than the first and second methods.

[0054] FIG. 12 is a diagram illustrating the observation range in the embodiment. In the explanations of the first, second, and third methods above, calculations were performed assuming that the observer's eye was positioned at the center of the observation range. However, if the observer's eye is within the observation range, the observer can observe the reconstructed image. Therefore, if the observation range is wider than the observer's pupil, there is an allowable range for the movement distance of the virtual image relative to the movement distance of the observer's eye.

[0055] If the width of the observation range is w and the diameter of the pupil is p, the allowable range within which the pupil can move relative to the observation range is ±(wp) / 2 from the observation center. For example, if the pupil size is 5 mm and the design value of the observation range is 10 mm, which is twice the pupil diameter, the tolerance is ±2.5 mm. The control unit 120 can allow the observer to view the reconstructed image by adjusting the movement distance of the virtual image using the first method, the second method, and the third method to keep the movement distance of the reconstructed image within an acceptable range.

[0056] [Configuration with moving virtual image] The movement mechanism for mechanically moving the first display panel 104 and the second display panel 106 is not particularly limited as long as it can mechanically move the first display panel 104 and the second display panel 106 linearly. For example, the movement mechanism may be configured to move the first display panel 104 and the second display panel 106 linearly in a parallel direction, such as a rack-and-pinion mechanism, a gear link mechanism, a slider crank mechanism, a slot crank mechanism, a heart cam mechanism, a Scotch yoke mechanism, a Bricard mechanism, a Porslier mechanism, a heart mechanism using crossed links, or a combination of these mechanisms. For example, an electromagnetic motor such as a DC motor, a brushless DC motor, a stepping motor, a servo motor, an induction motor, a PM motor, an AC motor, an in-wheel motor, or an ultrasonic motor may be used as a mechanism for automatically driving the movement mechanism. Furthermore, an internal combustion engine such as a reciprocating engine, a rotary engine, or a gas turbine engine may be used as a mechanism for automatically driving the movement mechanism.

[0057] Although a movement mechanism that mechanically moves first display panel 104 and second display panel 106 requires an additional mechanical mechanism, it has the advantage of being able to increase the movement distance of the virtual image. In the third method, because the movement distance of the virtual image on first display panel 104 is the same as the movement distance of the virtual image on second display panel 106, it is sufficient to move first display panel 104 and second display panel 106 together, and even though the movement is mechanical, it can be achieved with a relatively simple mechanism.

[0058] If the change in the position of the observer's eyes is due to individual differences such as the position of the observer's eyes, or if it occurs when wearing the head-mounted display device 1, the control unit 120 only needs to make adjustments to move the virtual images on each of the first display panel 104 and the second display panel 106 once after wearing the head-mounted display device 1. On the other hand, if the position of the observer's eyes changes after wearing the head mounted display device 1, it is necessary to measure the position of the observer's eyes using the detection unit 110 and move in real time the display positions of the virtual images on the first display panels 104 and the second display panels 106. There are various methods for measuring the position of the eyes, and some examples will be described later.

[0059] In the above-described embodiment, a mechanical movement mechanism that physically moves the positions of the first display panel 104 and the second display panel 106 has been described as a configuration for moving the virtual images displayed by the first display panel 104 and the second display panel 106, but this is not limited to this, and the virtual images displayed by the first display panel 104 and the second display panel 106 may be moved by an electronic method. The electronic method determines the position of the virtual image by changing the display position of the image displayed on each of the first display panel 104 and the second display panel 106. The electronic method has the advantage that it does not require a mechanical movement mechanism and can move the virtual image at high speed, but in order to move the virtual image by a large distance, the area of ​​the display panel needs to be large.

[0060] Unlike mechanical methods, electronic methods do not require a mechanical moving mechanism. The control unit 120 simply moves the display positions of the virtual images displayed on the first display panel 104 and the second display panel 106 in accordance with the change in the position of the observer's eye detected by the detection unit 110. The electronic method electronically moves the entire reconstructed image in accordance with the change in the position of the observer's eye, thereby allowing light rays from an appropriate group of pixels to be incident on the pupil, thereby avoiding deterioration of image quality such as loss of depth (three-dimensional effect) and image blurring. Note that the electronic method requires the entire reproduced image to be moved, which has the disadvantage in terms of display panel manufacturing that the image display areas of the first display panel 104 and the second display panel 106 must be designed to be one size larger by the maximum distance of translation. On the other hand, the electronic method has the advantage of being easier to assemble because it requires fewer mechanical parts than the mechanical method and the mechanical mechanism is simpler or even unnecessary.

[0061] Whether a mechanical or electronic method is adopted may vary depending on the use case, and either method may be adopted taking into consideration the advantages and disadvantages of the mechanical and electronic methods described above.Furthermore, a combination of mechanical and electronic methods may be used to move the virtual image.

[0062] In the embodiment, first display panel 104 and second display panel 106 are described as being liquid crystal displays, but the present invention is not limited to such displays, and other display types such as organic EL panels and DLP panels may also be used. Furthermore, it is not necessary to view the displays themselves, and a reproduced image may be observed using a display device that projects light rays.

[0063] In the embodiment, the first display panel 104 and the second display panel 106 are translated horizontally in a direction perpendicular to the direction in which the observer's eyes view the first display panel 104 and the second display panel 106, but this is not limiting. The observer's eyes move randomly left and right, up and down, and diagonally relative to the first display panel 104 and the second display panel 106, and the head-mounted display device 1 of the embodiment may move the virtual image of the first display panel 104 and the virtual image of the second display panel 106 in directions corresponding to the movement of the eyes in various directions.

[0064] [Operation procedure of head mounted display device 1] 13 is a diagram showing an example of an operation procedure of the head mounted display device 1 according to the embodiment. The following describes the process from capturing an image of an object to generating a light field image and finally providing a reconstructed image to the viewer.

[0065] First, multi-viewpoint images are created by synchronously capturing images of an object using multi-viewpoint cameras (step S100). In FIG. 14, (A) is a perspective view of a multi-viewpoint camera and a target object, and (B) is a diagram showing nine object images captured synchronously by nine multi-viewpoint cameras. The nine object images represent the target object as viewed from multiple viewpoints. It is desirable that the lens parameters of all camera lenses used in the multi-viewpoint camera be identical. Similarly, it is desirable to use the same model of camera body for all cameras. If the lens parameters of each lens are different or various types of cameras are mixed, the capturing range for each target image may differ, and various lens aberrations may cause various distortions in the captured images. This may complicate the subsequent process of reconstructing the object in a virtual space from the object images. The multi-viewpoint camera may be a single-lens reflex camera or a webcam built into a PC or similar device. It is preferable to select a multi-viewpoint camera appropriately based on, for example, image quality, weight, size, and price. The multi-view camera may be one that takes still images, one that takes video, or one that can switch between taking still images and video, but it is preferable that it be one that can save images as digital data.

[0066] Next, a 3D image (VR image) of the solid object is constructed from the multi-viewpoint images created in step S100 (step S102). The process of constructing a 3D image of the target object in virtual space may use the SfM (Structure from Motion) method, which reconstructs a 3D shape by combining multiple images of the target object. Examples of software that can use the SfM method include Metashape (registered trademark, manufactured by Agisoft LLC), Pix4Dmapper (manufactured by Pix4D), and TerraMapper (manufactured by Terra Drone). FIG. 15 is a diagram showing an example of a 3D image (VR image) of the target object constructed in virtual space in this embodiment.

[0067] Next, a light field image is generated using the 3D image constructed in step S102 (step S104). At this time, multiple virtual cameras (e.g., 5 vertical x 5 horizontal = 25 in total) are installed in the virtual space, and 25 synchronized images of the target object are acquired. The arrangement of 25 synchronized images is referred to as a light field image. FIG. 16 is a diagram showing a light field image generated from 25 simultaneously captured images in an embodiment. To install multiple cameras in the virtual space, game engine software is used. Examples of game engine software include Unity (manufactured by Unity Technologies), Unreal Engine (manufactured by Epic Games), and PhyreEngine (manufactured by Sony Interactive Entertainment).

[0068] Next, the 25 light field images generated in step S104 are dimensionally compressed to generate two images (step S106). At this time, the 25 light field images are dimensionally compressed into two images to be allocated to the first display panel 104 and the second display panel 106. The process of dimensionally compressing a plurality of images into two images may use, for example, the Non-negative Matrix Factorization (NMF) method. The NMF method divides matrix V into matrices W and H to obtain matrix V. ~ is a mathematical method for approximating V, W, and V ~ The NMF method is characterized by the fact that each factor of the matrix does not take a negative value. 2 sheets (A×A sheets, A 2 A 2D (three-dimensional) light field image exhibits A levels of depth in the vertical direction and A levels of depth in the horizontal direction. Figure 17 shows images two-dimensionally compressed using the NMF method in an embodiment, where (A) shows the image to be displayed on the first display panel 104 and (B) shows the image to be displayed on the second display panel 106. The superimposed image of the two images after two-dimensional compression can present five levels of depth.

[0069] Next, with the head mounted display device 1 attached to the head of the observer, the position of the observer's eyes is detected by the detection unit 110 (step S108). Note that the detection unit 110 detects the eye positions using various sensors, but is not limited to this, and the position information may be detected by manual processing using the observer's own sensor markers or the like.

[0070] Next, the control unit 120 calculates the direction and amount of movement of the virtual image on at least one of the first display panel 104 and the second display panel 106 based on the eye position detected in step S108 (step S110).

[0071] Next, the control unit 120 moves the virtual image on at least one of the first display panel 104 and the second display panel 106 based on the direction and amount of movement calculated in step S110 (step S112).

[0072] Next, the video signal output unit 130 outputs video signals based on the two dimensionally compressed images to the first display panel 104 and the second display panel 106, respectively, and displays a virtual image on the first display panel 104 and a virtual image on the second display panel 106, thereby allowing the reproduced image to be viewed (step S114).

[0073] Next, video signal output unit 130 determines whether the video to be displayed on first display panel 104 and second display panel 106 has finished, and if the video has not finished (step S116: NO), returns to step S102, and if the video has finished (step S116: YES), ends the processing of this flowchart. The processing of this flowchart enables the viewer to view a stereoscopic image in real time.

[0074] Note that this flowchart shows an example of detecting the eye position in real time and adjusting the display position of the virtual image, but if the positions of the observer's eyes and the head-mounted display device 1 are unlikely to change after wearing the head-mounted display device 1, the process of detecting the eye position and adjusting the display position of the virtual image only needs to be performed once after wearing the head-mounted display device 1.

[0075] The above-described head mounted display device 1 may detect the line of sight of the viewer. The detection unit 110 may include, for example, an eye-tracking device equipped with a light source that irradiates near-infrared light onto the observer's pupil and a small camera that detects the light reflected by the cornea of ​​the irradiated near-infrared light. The eye-tracking device tracks the gaze based on the positional relationship between the pupil and the Purkinje reflex image, which is generated when the near-infrared light is reflected by the cornea. There are two types of eye-tracking devices: one that is built into the head-mounted display device 1 and one that is glasses-type. The latter is preferable because the glasses lens may collide with the eyepiece lens 108 of the head-mounted display device 1. As an eye-tracking device that is built into the head-mounted display device 1, the Binocular Add-on (manufactured by PupilLabs) or the like can be used. As a glasses-type device, the Tobii Pro Glasses 3 (manufactured by Tobii) or Neon (manufactured by PupilLabs) or the like can be used.

[0076] In this embodiment, the gaze tracking device can be used to track the gaze of the observer's pupil. However, as described above, if the eye moves, the gaze tracking device cannot measure the distance the eyeball moves. The detection unit 110 may calculate the distance the eye moves by analyzing images captured by a small camera provided in the gaze tracking device. The head-mounted display device 1 calculates the distance the eye moves in real time and moves the virtual image based on the calculated eye movement distance, thereby realizing movement of the reproduced image without delay. To develop an image analysis program, a program library such as OpenCV (manufactured by Intel) or MATLAB (registered trademark) (manufactured by MathWorks) can be selected.

[0077] In this embodiment, for the sake of simplicity, the number of stacked display panels is set to two, but this is not limiting. For example, in the case of a three-layer stacked light field head mounted display device, the head mounted display device 1 of the embodiment may have a third display panel positioned closest to the front of the viewer in addition to the first display panel 104 and the second display panel 106. The head mounted display device 1 may move the virtual image displayed by the third display panel in addition to the virtual image displayed by the first display panel 104 and the virtual image displayed by the second display panel 106 in accordance with the movement of the viewer's eyes.

[0078] The stacked light field head mounted display device according to the present invention will be further described using examples and comparative examples. The technical scope of the present invention is not limited solely by the specific content of the examples.

[0079] Example 1 First, the fabrication of the head mounted display device 1 for the right eye will be described. Two IPS (In-Plane Switching) liquid crystal display panels of the same size and rectangular shape when viewed from the front were prepared. A backlight and a color filter were attached to one of the display panels to form the first display panel 104, and a color filter was attached to the other to form the second display panel 106. The first display panel 104 and the second display panel 106 were completely overlapped when viewed from the front, with an air gap of 6 mm between them. Next, rack-and-pinion mechanisms were attached to the right and bottom ends of the first display panel 104 and the second display panel 106, respectively, and the rack-and-pinion mechanisms were driven via DC motors, allowing the first display panel 104 and the second display panel 106 to move simultaneously and in the same direction, up and down, left and right, and diagonally, in a direction perpendicular to the viewer's viewing direction. Next, the eye-tracking device Binocular Add-on (manufactured by PupilLabs) was incorporated into the edge of the frame that holds the right eye eyepiece 108R, and then the right eye eyepiece 108R was attached to complete the head-mounted display device for the right eye. A head mounted display device for the left eye portion was fabricated in the same manner as the right eye portion, and then the right eye portion and the left eye portion were combined to finally complete a stacked head mounted display device 1 for both eyes.

[0080] Using the stacked light field head mounted display device of Example 1, the image quality was evaluated by visual observation when the observer's eyes moved. First, the observer calibrates the eye position using the stacked light field head-mounted display device of Example 1. Next, the entire stacked light field head-mounted display device of Example 1 is rotated clockwise by 10° around the top of the observer's head. This is equivalent to rotating the observer's head by 10° counterclockwise relative to the head-mounted display device. This state is photographed with a small camera of an eye-tracking device attached to the head-mounted display device, and the photographed image is analyzed using OpenCV, revealing that the movement distance m is 17.5 mm.

[0081] Next, using the head-mounted display device of Example 1, an observer visually evaluated the quality of an image that was mechanically translated by m = 17.5 mm simultaneously to the left on the first display panel 104 and the second display panel 106 in real time at the moment the eyes moved, and the observer was able to see a good depth and stereoscopic image.

[0082] Example 2 First, the fabrication of the head mounted display device for the right eye will be described. Two IPS (In-Plane Switching) liquid crystal display panels of the same size and rectangular shape when viewed from the front were prepared. A backlight and a color filter were attached to one display panel to form the first display panel 104, and a color filter was attached to the other to form the second display panel 106. The first display panel 104 and the second display panel 106 were completely overlapped when viewed from the front, with an air gap of 6 mm between them. Next, the eye-tracking device Binocular Add-on (manufactured by PupilLabs) was attached to the edge of the frame that holds the eyepiece, and then an eyepiece was attached to complete the head-mounted display device for the right eye. A head-mounted display device for the left eye was fabricated in the same way as the right eye, and then the right and left eye portions were combined to finally complete a stacked light field head-mounted display device for both eyes.

[0083] Using the stacked light field head mounted display device of Example 2, the image quality was evaluated by visual observation when the eyes were moved. First, the observer calibrated the eye position using the stacked light field head-mounted display device of Example 2. Next, the entire stacked light field head-mounted display device of Example 2 was rotated clockwise by 5° around the top of the observer's head. This was the same as rotating the observer's head by 5° to the left relative to the head-mounted display device. This state was photographed with a small camera of an eye-tracking device attached to the head-mounted display device, and the photographed image was analyzed using MATLAB, revealing that the movement distance m was 8.7 mm. Next, using the head-mounted display device of Example 2, an image was displayed on the first display panel 104 and the second display panel 106 simultaneously to the left, and electronically shifted by m = 8.7 mm in real time at the moment the eyes moved. The image was then visually evaluated by an observer to assess its quality. The observer was able to see a good depth / 3D image. The maximum angle of the head-mounted display device that rotates around the viewer's head was set to 7°, and the first display panel 104 and the second display panel 106 were designed to be slightly larger. Specifically, for a maximum angle of 7°, the first display panel 104 and the second display panel 106 were made to be slightly larger by 13 mm in both length and width.

[0084] (Comparative Example) First, the fabrication of the head mounted display device for the right eye will be described. Two IPS (In-Plane Switching) liquid crystal display panels of the same size and rectangular shape when viewed from the front were prepared. One display panel was equipped with a backlight and a color filter to make it the first display panel, and the other was equipped with a color filter to make it the second display panel. The first and second display panels were completely overlapped in front view with an air gap of 6 mm between them. Next, the eye-tracking device Binocular Add-on (manufactured by PupilLabs) was attached to the edge of the frame that holds the eyepiece, and then an eyepiece was attached to complete the head-mounted display device for the right eye. A head-mounted display device for the left eye was fabricated in the same way as the right eye, and then the right and left eye portions were combined to finally complete a stacked light field head-mounted display device for both eyes.

[0085] Using the stacked light field head mounted display device of the comparative example, the image quality was evaluated by visual observation when the eyes were moved. First, the observer calibrated the eye position using the stacked light field head-mounted display device of the comparative example. Next, the entire stacked light field head-mounted display device of the comparative example was rotated clockwise by 10° around the top of the observer's head. In other words, this was the same as rotating the observer's head counterclockwise by 10° relative to the head-mounted display device. In this state, the pixel areas of the first display panel and the second display panel were inappropriately overlapped, causing unintended light rays to enter the pupils of the observer's eyes, causing the observer to view a poor depth and stereoscopic image.

[0086] The functions of the control unit 120 in the above-described embodiment may be implemented by a computer. In this case, a program for implementing the functions may be recorded on a computer-readable recording medium, and the program may be loaded into a computer system and executed. The term "computer system" as used herein includes hardware such as an OS and peripheral devices. The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into a computer system. The term "computer-readable recording medium" may also include media that dynamically store programs for a short period of time, such as communication lines used when transmitting programs over a network such as the Internet or a telephone line, or media that store programs for a fixed period of time, such as volatile memory within a computer system serving as a server or client. The program may be a program that implements only a portion of the above-described functions, or may be a program that can implement the above-described functions in combination with a program already stored in the computer system, or may be implemented using a programmable logic device such as an FPGA (Field Programmable Gate Array).

[0087] Each embodiment and variant has been described, but these are merely examples and are not intended to be limiting. For example, one aspect of the present invention may be realized by combining any of the embodiments or variants, or a part of each embodiment or a part of each variant, with one or more other embodiments or one or more other variants. [Explanation of symbols]

[0088] 1. Head-mounted display device 100 cabinets 102L, 102R backlight 104, 104L, 104R First display panel 106, 106L, 106R Secondary Display Panel 108 eyepiece 108L left eyepiece 108R Right eyepiece 110 Detector 110L Left eye detector 110R Right eye detection unit 112R, 112L, 114R, 114L, 116R, 116L Moving mechanism 120 control section 130 Video signal output section

Claims

1. a detection unit that detects the position of the user's eyes; An eyepiece and a first display panel that displays a first image; a second display panel that is disposed closer to the eyepiece than the first display panel and that displays a second image; a control unit that moves at least one of the first image and the second image based on a change in the eye position detected by the detection unit; A head-mounted display device comprising:

2. The head-mounted display device according to claim 1 , wherein the control unit moves the first image based on the position of the eye detected by the detection unit.

3. The head-mounted display device according to claim 1 , wherein the control unit moves both the first image and the second image based on the position of an eye detected by the detection unit.

4. 2. The head-mounted display device according to claim 1, wherein the control unit moves both the first image and the second image in the direction of movement of the eye position detected by the detection unit by the same distance based on the movement distance of the eye.

5. 2. The head-mounted display device of claim 1, wherein the control unit moves both the first image and the second image in the direction of movement of the eye position detected by the detection unit, and moves the second image by a distance greater than the distance of movement of the first image.

6. The head-mounted display device of claim 1, wherein the control unit moves at least one of the first image and the second image so that the first image and the second image can be viewed within an allowable range set based on the size of the pupil and the observation range.

7. An image display control device for a head mounted display device including: a detection unit that detects the position of a user's eye; an eyepiece; a first display panel that displays a first image; and a second display panel that is disposed closer to the eyepiece than the first display panel and that displays a second image, An image display control device comprising a control unit that displays a first image on the first display panel and a second image on the second display panel, and controls the movement of at least one of the first image and the second image based on a change in eye position detected by the detection unit.

8. a computer for a head mounted display device including a detection unit for detecting the position of a user's eye, an eyepiece, a first display panel for displaying a first image, and a second display panel disposed closer to the eyepiece than the first display panel and for displaying a second image; detecting the position of the user's eyes; displaying a first image on the first display panel; displaying a second image on the second display panel; moving at least one of the first image and the second image based on a change in the position of the eye detected by the detection unit; A video display control program that executes the above.

Images