Display device, display control method, and display control program

Abstract

Provide users with appropriate images. [Solution] The display device 1a provides the user with a stereoscopic image and comprises multiple pixels P A display unit 10 that provides an image to the user by irradiating light, and a display unit 10 that is more user-friendly than the display unit 10. It is provided on the side and has multiple lenses 32a, and the focus of each lens 32a can be changed individually. Based on the microlens array 30a and depth information indicating the position in the depth direction of the image, It includes a focal position setting unit that sets the position of a point for each lens 32a.

Description

Technical Field

[0001] The present invention relates to a display device, a display control method, and a display control program.

Background Art

[0002] A display device that allows a user to visually recognize images with different parallaxes for the right and left eyes and displays a stereoscopic image using the difference in convergence is known. As such a display device, there is a so-called head-mounted display (HMD) mounted on the user's head. For example, Patent Document 1 describes a head-mounted display in which a microlens array is arranged between a display and an optical system.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Here, in a display device that displays a stereoscopic image, it is required to appropriately provide an image to the user.

[0005] In view of the above problems, an object of the present invention is to provide a display device, a display control method, and a display control program that can appropriately provide an image to a user.

Means for Solving the Problems

[0006] A display device according to an aspect of the present invention includes a plurality of pixels and irradiates light to the user. ​A display unit that provides images to the user, and a plurality of lenses provided on the user side of the said unit. A microlens array having, the focus of the lenses can be individually changed, and the depth of the image Based on depth information indicating the position in the direction, the focal position is set for each lens. Includes a position setting unit.

[0007] A display control method according to one aspect of the present invention is a display unit that includes multiple pixels and emits light, A microlens array provided on the user side, allowing the focus of multiple lenses to be changed individually. The position of the point is determined for each lens based on depth information indicating the position in the depth direction of the image. Includes a step of setting the focal position.

[0008] A display control program according to one aspect of the present invention includes a display unit that includes multiple pixels and emits light. A microlens array, located on the user's side, allows for individual adjustment of the focus of multiple lenses. The focal position of (i) is determined based on depth information indicating the position in the depth direction of the image, and the lens The computer executes the focus position setting step that is set each time. [Effects of the Invention]

[0009] According to the present invention, images can be provided appropriately to the user. [Brief explanation of the drawing]

[0010] [Figure 1] Figure 1 is a schematic diagram illustrating the vergence-accommodation contradiction. [Figure 2] Figure 2 is a schematic diagram of the display device according to the first embodiment. [Figure 3] Figure 3 is a schematic diagram of the various components of the display device according to the first embodiment. [Figure 4] Figure 4 is a schematic block diagram of the control device according to the first embodiment. [Figure 5] FIG. 5 is a schematic diagram for explaining the setting of irradiation timing. [Figure 6] FIG. 6 is a schematic diagram for explaining the setting of irradiation timing. [Figure 7] FIG. 7 is a schematic diagram for explaining the setting of irradiation timing. [Figure 8] FIG. 8 is a flowchart for explaining the processing flow of the control device according to the present embodiment. [Figure 9] FIG. 9 is a schematic diagram of each component of the display device according to the second embodiment. [Figure 10] FIG. 10 is a schematic block diagram of the control device according to the second embodiment. [Figure 11] FIG. 11 is a flowchart for explaining the processing flow of the control device according to the second embodiment. [Figure 12] FIG. 12 is a schematic diagram of the display device according to the modification example.

Mode for Carrying Out the Invention

[0011] Hereinafter, embodiments of the present invention will be described in detail based on the drawings. Note that the present invention is not limited by the embodiments described below. The present invention is not limited by the embodiments described below.

[0012] (Convergence adjustment contradiction) FIG. 1 is a schematic diagram for explaining convergence adjustment contradiction. A display device that displays a stereoscopic image causes the user to visually recognize images with different parallaxes for the right and left eyes, and displays a stereoscopic image by utilizing the difference in convergence. When displaying a stereoscopic image, the display surface on which the image is actually displayed becomes the focal position of the user's eyes, and the position where the lines of sight of the left and right eyes intersect becomes the convergence position. However, as shown in the example of FIG. 1, in a stereoscopic image, the positions of the focal position PO1 and the convergence position PO2 in the Z direction, which is the depth direction of the stereoscopic image, may be shifted. The focal position PO1 and the convergence position P causes the user to visually recognize images with different parallaxes for the right and left eyes, and displays a stereoscopic image by utilizing the difference in convergence. When displaying a stereoscopic image, the display surface on which the image is actually displayed becomes the focal position of the user's eyes, and the position where the lines of sight of the left and right eyes intersect becomes the convergence position. However, as shown in the example of FIG. 1, in a stereoscopic image, the positions of the focal position PO1 and the convergence position PO2 in the Z direction, which is the depth direction of the stereoscopic image, may be shifted. The focal position PO1 and the convergence position P causes the user to visually recognize images with different parallaxes for the right and left eyes, and displays a stereoscopic image by utilizing the difference in convergence. When displaying a stereoscopic image, the display surface on which the image is actually displayed becomes the focal position of the user's eyes, and the position where the lines of sight of the left and right eyes intersect becomes the convergence position. However, as shown in the example of FIG. 1, in a stereoscopic image, the positions of the focal position PO1 and the convergence position PO2 in the Z direction, which is the depth direction of the stereoscopic image, may be shifted. The focal position PO1 and the convergence position P causes the user to visually recognize images with different parallaxes for the right and left eyes, and displays a stereoscopic image by utilizing the difference in convergence. When displaying a stereoscopic image, the display surface on which the image is actually displayed becomes the focal position of the user's eyes, and the position where the lines of sight of the left and right eyes intersect becomes the convergence position. However, as shown in the example of FIG. 1, in a stereoscopic image, the positions of the focal position PO1 and the convergence position PO2 in the Z direction, which is the depth direction of the stereoscopic image, may be shifted. The focal position PO1 and the convergence position P In a stereoscopic image, as shown in the example of FIG. 1, the positions of the focal position PO1 and the convergence position PO2 in the Z direction, which is the depth direction of the stereoscopic image, may be shifted. The focal position PO1 and the convergence position P In a stereoscopic image, as shown in the example of FIG. 1, the positions of the focal position PO1 and the convergence position PO2 in the Z direction, which is the depth direction of the stereoscopic image, may be shifted. The focal position PO1 and the convergence position P When the position of O2 is misaligned, a so-called convergence-accommodation discrepancy occurs, leading to eye strain and so-called 3D motion sickness. This can lead to various problems. Therefore, it is necessary to suppress the vergence-accommodative contradiction. 1(A) shows that the focal point PO1 is closer to the user's eye EY than the convergence point PO2, as shown in Figure 1. (B) shows an example where the convergence point PO2 is closer to the user's eye EY than the focal point PO1. ru.

[0013] (First Embodiment) (Overall configuration of the display device) Figure 2 is a schematic diagram of the display device according to the first embodiment. Display device 1 according to the first embodiment This is a display device that displays stereoscopic images. As shown in Figure 2, the display device 1 is for user U This is a so-called HMD (Head Mount Display) that is worn on the head. For example, the display device 1 has a display unit 10 mounted in a position facing the user U's eye EY. The display device 1 then displays an image on the display unit 10 and provides content to the user U. Note that the configuration of the display device 1 shown in Figure 2 is just one example. For example, the display device 1 is for user U It may also be equipped with an audio output unit (speaker) that is worn on the ear.

[0014] Since the display device 1 is attached to the user U in this manner, the position of the user U's eye EY is The display device 1 is not limited to being an HMD worn by user U, but also includes equipment. It may also be a display device that is fixed to the surface. In such cases, the display device 1 may be, for example, User U's position relative to the seat is fixed, and so is User U's position relative to the eye EY. It is preferable that this be done.

[0015] Figure 3 is a schematic diagram of each component of the display device according to the first embodiment. As shown in Figure 3, The display device 1 includes a display unit 10, an eyepiece lens 20, a microlens array 30, and a control device. It has 40.

[0016] (Display) The display unit 10 is a device that displays a three-dimensional image. The display unit 10 is arranged in a matrix and This is a display having multiple light-emitting pixels P (display elements). Each of the display units 10 Since each pixel P emits light on its own, the display unit 10 can individually control the emission (light irradiation) of each pixel P. It is possible. Each pixel P of the display unit 10 is, for example, an organic light-emitting diode (OLED). It could also be an inorganic light-emitting diode (nic Light Emitting Diode), or Odor (may also be a micro LED). The image is displayed on the display unit 10. The indicated surface will be referred to as the display surface 10A. The following describes the direction from the display surface 10A towards the user U's eye EY. Let the direction be Z1, and the direction opposite to Z1, i.e., from the user U's eye EY to the display surface 10 Let Z2 be the direction towards A. If directions Z1 and Z2 are not distinguished, write it as direction Z. In Figure 3, the surface of the display unit 10 on the side of the user U's eye EY is referred to as the display surface 10A. However, the display surface 10A is not limited to being the surface on the eye side EY of the user U, but also the surface on the eye side EY of the user U It may be located inside the surface of the EY side. The display unit 10 may be located inside the control device 40 which will be described later. The system then receives a control signal to control the pixels P of the display unit 10.

[0017] The display unit 10 displays the image light L, which is light emitted from each pixel P, to the user U's eye EY. By reaching this point, a three-dimensional image is provided to the user U. More specifically, the display unit 10 is on the left The emission of light from each pixel P is controlled so that an image for the right eye and an image for the right eye are provided. Of these image rays L, the image rays L from the pixel P corresponding to the left eye image enter the user U's left eye. When the image light L from the pixel P corresponding to the right eye image is incident on the user U's left eye, A 3D image is provided to user U.

[0018] (Eyepiece) The eyepiece 20 is located on the Z1 side of the display unit 10. The eyepiece 20 is designed to capture light (image). It is an optical element that transmits image light. Furthermore, the eyepiece 20 is the most... This is an optical element (lens) located on the eye side (EY) of the user U. Image light emitted from the display unit 10. L passes through the eyepiece 20 and enters the user U's eye EY. In this embodiment, Optical axis in the optical path of the image light L from eyepiece 20 (eyepiece) to user U's eye EY The direction can also be described as direction Z.

[0019] In the example shown in Figure 3, the optical elements on the Z1 direction side of the display unit 10 are the eyepiece lens 20 and a Although only the microlens array 30 is shown, it is not limited to that, and other optical elements are also shown. It may be provided.

[0020] (Microlens array) The microlens array 30 is positioned in the optical axis direction of the image light L, and is more user-friendly than the display unit 10. It is located on the U side. Furthermore, the microlens array 30 is located on the optical axis side of the image light L. In this direction, it is provided between the display unit 10 and the eyepiece lens 20. I30 is a light source in which multiple lenses 32 are arranged in a matrix on a plane parallel to the display surface 10A. It is a scientific element. In this embodiment, the pitch of each lens 32 of the microlens array 30 is In other words, the distance between the centers of adjacent lenses 32 is, for example, the pitch of the pixels P of the display unit 10 ( It is approximately the same as the distance between the centers of adjacent pixels P. The 32 is provided in the optical axis direction of the image light L, at a position opposite each pixel P of the display unit 10. The microlens array 30 is connected to the control device 40, which will be described later. The system receives a control signal to control the array 30.

[0021] The microlens array 30 is in the Z direction (optical axis direction of the image light L) of each lens 32 A so-called variable-focus microscope, in which the focal point (the distance from lens 32 to the focal point) can be changed. It is a lens array. In the example of the first embodiment, the microlens array 30 is each This uniformly changes the focal position of lens 32, and each lens at the same time The focal position of Z32 in the Z direction is the same. The microlens array 30 is arbitrary It can be a structure, but for example, it can be composed by combining a liquid crystal and a Fresnel lens. For example, as each lens 32 of the microlens array 30, a "liquid crystal variable focus lens" "Volumetric Three-Dimensional Display Utilized" (Shiro Toyama, Tokushima University, Optics, Vol. 40, No. 12 (2011)) Variable focus lenses disclosed in the literature may be used.

[0022] (Control device) The control device 40 is a device that controls each part of the display device 1. Figure 4 shows the first embodiment. This is a schematic block diagram of the control device. In this embodiment, the control device 40 is a computer It has a storage unit 42 and a control unit 44. The storage unit 42 stores the calculation contents of the control unit 44 and It is a memory that stores various types of information such as programs, for example, RAM (Random Access Memory). There are primary memory types such as ROM (Read Only Memory) and ROM (Read Only Memory). Of the storage devices and external storage devices such as HDDs (Hard Disk Drives), It includes at least one. The program for the control unit 44 stored in the memory unit 42 is the control device 40 It may be stored on a readable recording medium.

[0023] The control unit 44 is an arithmetic unit, for example, a CPU (Central Processor). It includes calculation circuits such as g Unit. The control unit 44 includes an image information acquisition unit 50 and a drive control It includes a unit 52, a timing setting unit 54, and an irradiation control unit 56. The control unit 44 is a storage unit 4 By reading and executing the program (software) from 2, the image information acquisition unit 50 and The drive control unit 52, timing setting unit 54, and irradiation control unit 56 are implemented, and their processing is performed Execute. Note that the control unit 44 may execute these processes using a single CPU. It may also have multiple CPUs, and the processing may be performed by those multiple CPUs. The information acquisition unit 50, the drive control unit 52, the timing setting unit 54, and the irradiation control unit 56 are at least Another method could be implemented using hardware circuits.

[0024] (Image information acquisition unit) The image information acquisition unit 50 acquires image data of the 3D image displayed by the display unit 10. The image information acquisition unit 50 then processes the image data for the left eye and the image data for the right eye. It acquires the image information. The image information acquisition unit 50 also acquires the depth of the stereoscopic image, which indicates the position in the depth direction. Information is obtained. The position in the depth direction of the 3D image is the position of the image displayed on the display surface 10A. This refers to the position in the depth direction of the virtual image that is visible to the user U when this occurs. It can also be said that this is a direction perpendicular to the display surface 10A of the display unit 10, and in this embodiment, it is the Z direction. Depth information is associated with image data. Furthermore, a stereoscopic image is 1F Each image included in the frame has a position set in the depth direction, or in other words Therefore, a position in the depth direction is set for each position on the display surface 10A. The image information acquisition unit 50 acquires depth information for each position on the display surface 10A of the stereoscopic image. It can be said that it is acquired. Furthermore, in a stereoscopic image, the position in the depth direction is set for each pixel P. However, for multiple pixels P that make up a single image, their positions in the depth direction are the same. It may be set to be such. The image information acquisition unit 50 acquires image data and The depth information may be acquired, and the image data and depth information that were previously stored in the memory unit 42 may be acquired. The report may be read, or image data and depth information may be received via a communication unit (not shown). It is also possible. Furthermore, the image information acquisition unit 50 calculates the position in the depth direction based on the image data. Depth information may be obtained by doing so.

[0025] (Drive control unit) The drive control unit 52 controls the microlens array 30 The focal position of each lens 32 is moved in the Z direction. The drive control unit 52 controls, for example, a microphone By controlling the voltage applied to the liquid crystal elements included in the lens array 30, lens 3 The focal position of lens 2 is moved in the Z direction. The drive control unit 52 controls the focal position of lens 32 when it moves in the Z direction. A reciprocating motion (vibration) in the Z direction involves moving a predetermined distance in one direction and then moving a predetermined distance in the Z2 direction. The focal position of lens 32 is moved repeatedly. The drive control unit 52 controls the lens The position of the 32 focal points is moved along the Z direction at a predetermined period. In other words, drive control. The unit 52 causes the focal point of the lens 32 to move back and forth in the Z direction at a predetermined period. So, what is the period of reciprocating movement in the Z direction (until the display unit 10 returns to its original position in the Z direction)? The time (of the period) is constant, but it is not limited to being constant; the period may be changed.

[0026] (Timing setting unit and irradiation control unit) The timing setting unit 54 sets the illumination timing of the image light L for each pixel P of the display unit 10. The irradiation control unit 56 controls the pixels P of the display unit 10 based on the image data to illuminate the pixels P. The image light L is irradiated. The irradiation control unit 56 sets the timing for each pixel P set by the timing setting unit 54. At the irradiation timing, image light L is irradiated to each pixel P. That is, the irradiation control unit 56, Image light L is directed to a pixel P of the display unit 10 at an irradiation timing set for that pixel P. Irradiate. The timing setting unit 54 controls the micro in the Z direction (optical axis direction of the image light L). The irradiation timing is set based on the focal position of the lens array 30 (lens 32). Furthermore, the timing setting unit 54 uses the depth information of the 3D image and the microphone in the Z direction. The irradiation timing is set based on the focal position of the Lorenz array 30. The timing setting unit 54 sets the irradiation timing for each pixel P, but the irradiation timing for each pixel P This is not limited to the fact that they are different, but for example, a group of pixels P that make up one image (for example, the house in Figure 5) The illumination timing of a group of pixels (P, etc.) that display an image may be set to be the same. The settings of the timing setting unit 54 will be explained in more detail.

[0027] Figures 5 to 7 are schematic diagrams illustrating the setting of irradiation timing. For example, in Figure 5... As shown, the image light L emitted from the pixels P of the display unit 10 is a light beam with a predetermined opening angle and Then, it enters the user U's eye EY. In this case, user U will be at the angle of opening of this light beam. Convergence: The eye unconsciously changes the thickness of its lens to adjust the focus to the retina. The convergence-accommodation discrepancy depends on the degree of convergence between the left and right eyes (how closely the two eyes converge) and the angle of light beam divergence. This refers to a state where the combined state does not match. In contrast, the display device of this embodiment 1 is the aperture angle assuming that light is emitted from the virtual image (convergence position) and enters the eye's eye (EY). A certain virtual image opening angle (angle θ1A in the example in Figure 5) and the actual image light L emitted from pixel P The image light L is emitted such that the difference between the angle of opening of the image light L when it enters the eye EY is minimized. Here, the image light L, as it passes through the lens 32 of the microlens array 30, The aperture angle changes (refracts), and the degree of change in the aperture angle of the light beam is determined at the focal point of lens 32. The timing setting unit 54 determines the virtual image aperture angle based on the focal position of the lens 32. The timing at which the difference between the image light angle L and the position where the difference becomes small is called the irradiation timing. Set it as follows.

[0028] More specifically, the timing setting unit 54 acquires depth information for each pixel P. The timing setting unit 54 acquires position information in the depth direction (Z direction) for each pixel P. Then, the timing setting unit 54 sets the depth information of the pixel P and sets the image of the pixel P. The illumination position, which is the focal point of lens 32 when the illumination of image light L is started, is set. The setting unit 54 sets the depth direction position of the portion of the stereoscopic image displayed by the pixel P. The luminous beam divergence when light is shone onto the eye EY from the position of the virtual image formed by the pixel P. At the angle (virtual image opening angle), the image light that actually enters the eye EY from the pixel P through the lens 32 is directed. The focal position of lens 32 in the Z direction when the opening angle of L matches is determined by its pixel P. The irradiation position is set to the distance from the irradiation position. The timing setting unit 54 then sets the distance from the irradiation position. The timing at which the focal point of the lens 32 reaches a position within a predetermined distance range is determined for that pixel P. The irradiation timing is set for each pixel P. The timing setting unit 54 sets the irradiation position for each pixel P. The settings are configured, and the irradiation timing is set for each pixel P. The irradiation position and timing are set for each individual pixel P, but the irradiation position and timing differ for each pixel P. It is not limited to becoming; for example, a group of pixels P that make up a single image (for example, the house image in Figure 5) The illumination position and illumination timing of a group of pixels (such as pixels P) that display the image may be set to be the same.

[0029] In this embodiment, the timing setting unit 54 determines the focal position of the lens 32 in the Z direction. Information is acquired sequentially, and the focal position of lens 32 in the Z direction is at a predetermined distance from the illumination position. When it gets close enough, it is determined that it is time to irradiate. The timing setting unit 54 is in the Z direction. Information about the focal position of lens 32 in can be obtained by any method. For example, lens 3 When the focal point of lens 2 is moved back and forth in the Z direction at a predetermined period, the focal point of lens 32 at each time interval The position of the point in the Z direction (predicted position) can be determined. Therefore, the timing setting unit 54, The position information of the focal point of lens 32 in the Z direction may be obtained from the time information. The timing setting unit 54 receives information on the predicted focal position of the lens 32 and the illumination position for each time period. Based on the report, the time at which the focal point of lens 32 reaches a predetermined distance from the illumination position is determined as the illumination timing. Set as the target, and when the current time reaches the illumination timing, the focus of lens 32 will move to the illumination position. If it is determined that the required distance has been reached, the image light L may be shone onto the pixel P. These can be set arbitrarily, but in order to minimize the vergence adjustment discrepancy, the virtual image opening angle and the opening of the image light L It is preferable to set the distance such that the difference with the angle is small. Also, timing The setting unit 54 sets the depth direction value to determine the depth of the pixel P used for setting the illumination position. It can be used as a position in a direction. That is, for example, the depth direction can be divided into multiple numerical ranges. Then, for each numerical range, a predetermined value within that numerical range is set as the reference position. Then, the timing setting unit 54 sets the depth of the pixel P acquired by the image information acquisition unit 50. Extract the numerical range that includes the position in the direction, and set the reference position for that numerical range. This is treated as the position in the depth direction of the pixel P used to set the firing position.

[0030] The timing setting unit 54 sets the illumination timing for each pixel P in this manner. The timing setting unit 54 sets the irradiation stop timing to a timing later than the irradiation timing. The irradiation control unit 56 determines that the irradiation timing has been reached for a certain pixel P. If interrupted, the irradiation of the image light L to that pixel P is started. The irradiation control unit 56 determines the irradiation timing. From the start of irradiation until the irradiation stop timing, the image light L is irradiated onto the pixel P, and the irradiation stop timing When the ming point is reached, the illumination of the image light L to that pixel P is stopped. Note that the illumination stop timing The timing can be set arbitrarily; for example, the irradiation stop timing can be set to a predetermined time after the irradiation timing. Alternatively, you can set it as follows: immediately after the irradiation timing, the focal position of lens 32 and the irradiation position The timing at which the distance to the device falls outside a predetermined distance range may be set as the irradiation stop timing. stomach.

[0031] The display device 1, in this manner, irradiates the pixel P with image light L when the irradiation timing is reached. When the irradiation stop timing is reached, the irradiation of image light L is stopped. Between the start and the timing of irradiation stop, the light is incident on the user U's eye EY. Therefore, in the eye EY The aperture angle of the incident image light beam L is close to the virtual image aperture angle from the virtual image formed by that pixel P. This makes it possible to reduce the convergence-accommodation discrepancy. The focal position of lens 32 is Z-direction Because it moves back and forth in that direction, the distance to the irradiation position will be within a predetermined distance range, and outside the predetermined distance range The process repeats. The control device 40 controls the distance between the focal position of the lens 32 and the illumination position. Each time the distance reaches a predetermined range, that is, each time the irradiation timing is reached, the image light L is emitted to the pixel P. The light is shone onto the device. As a result, the 3D image is perceived by user U as a moving image. The focal point of Lens 32 moves back and forth in the Z direction, so in one cycle of this back and forth movement, the position of the irradiation position is different from the focal point of Lens 32. There are two instances where the distance becomes a predetermined distance. Therefore, the focal position of lens 32 moves back and forth. The vibration frequency for movement should ideally be at least half the frame rate of the 3D image. Furthermore, the frequency (period) of the reciprocating movement of the focal point of lens 32 can be set arbitrarily.

[0032] The example of setting the irradiation timing described above will be explained using Figures 5 to 7. Images of a house, a car, and a helicopter are displayed as 3D images, and the images of the car and the house In the order of the image and the helicopter image, the position in the depth direction (Z direction) is farther from the user U's eye EY. Let's take the case where it becomes more difficult. That is, the virtual image P2 of the car is more difficult than the virtual image P1 of the house. Located on one side, the virtual image P3 of the helicopter is in the Z2 direction compared to the virtual image P1 of the house. It is located to the side.

[0033] Figure 5 shows an example of a case where a virtual image P1 of a house is made visible to user U. In the example in Figure 5, Light was shone onto the eye EY from the virtual image P1 (the position in the depth direction of the pixel P that makes up the image of the house). Let the angle of light beam opening (virtual image opening angle) in this case be angle θ1A. Then, the lens in the Z direction When the focal position of Z32 is the first position, the image light L from the pixels P that make up the image of the house Let's assume the angle of the light beam opening is angle θ1A. In this case, the first position forms the image of the house. This is the illumination position for pixel P, and the timing setting unit 54 determines the distance from the first position to a predetermined distance. The timing at which the focus of lens 32 reaches a position within the separation range is determined by the image that makes up the picture of the house. This is set as the irradiation timing for element P. The irradiation control unit 56 sets the irradiation timing. Then, the image light L is shone onto the pixels P that make up the image of the house. This creates a virtual image of the house. The virtual image opening angle from image P1 is close to the opening angle of the image light L that actually enters the user U's eye EY. This reduces the convergence-accommodation discrepancy. The image light L is refracted by the eyepiece 20. Because it is bent and enters the eye EY, the opening angle of the light beam of the image light L here is the eyepiece 20 This refers to the angle of opening of the luminous beam L of the image light after it has passed through the light source.

[0034] Figure 6 shows an example of how a virtual image P2 of a car is made visible to user U. In the example in Figure 6, Light was shone onto the eye EY from the virtual image P2 (the position in the depth direction of the pixel P that makes up the car image). Let the angle of light beam opening (virtual image opening angle) be angle θ²A. Then, the lens in the Z direction When the focal position of Z32 is the second position, the image light L from the pixels P that make up the car image Assume the angle of the luminous beam is angle θ2A. Note that the virtual image P2 is in the Z1 direction relative to the virtual image P1. Since it is visible, angle θ2A is larger than angle θ1A in Figure 5, and the first position is the focal point. The degree to which the angle of light beam is expanded by lens 32 is greater when the second position is the focal point than when the first position is the focal point. The size increases (or the degree of reduction decreases). In this case, the second position constitutes the image of the car. This becomes the illumination position for the pixel P, and the timing setting unit 54 determines the distance from the second position. The timing at which the focal point of lens 32 reaches a position within a fixed distance range is used to compose the image of the car. This is set as the irradiation timing for pixel P. The irradiation control unit 56 sets the irradiation timing Then, light is shone onto the pixels P that make up the car image. This creates a virtual image of the car. The virtual image opening angle from P2 is close to the opening angle of the image light L that actually enters the user U's eye EY. This allows us to reduce the vergence-accommodation contradiction.

[0035] Figure 7 shows an example of how to make user U view a virtual image P3 of a helicopter. In the example, the virtual image P3 (the position in the depth direction of the pixel P that makes up the helicopter image) is used to view the eye EY. Let the angle of the light beam (virtual image angle) when light is shone on Z be angle θ3A. When the focal position of lens 32 in the direction is the third position, the image of the helicopter is constructed. Assume that the angle of the light beam L from the pixel P is angle θ3A. Note that the virtual image P3 Since it is visible in the Z2 direction more than the virtual image P1, angle θ3A is smaller than angle θ1A in Figure 5. Furthermore, when the third position is the focal point, the lens 32 performs better than when the first position is the focal point. The degree to which the angle of the light beam expands decreases (or the degree to which it contracts increases). In this case, Position 3 is the illumination position for pixel P that makes up the helicopter image, and timing setting The fixed part 54 is determined when the focal point of the lens 32 reaches a position where the distance from the third position is within a predetermined distance range. The timing of this is set as the illumination timing for pixels P that make up the helicopter image. The irradiation control unit 56 determines the image that constitutes the helicopter image when the irradiation timing is reached. The element P is illuminated with image light L. This causes the virtual image of the helicopter to unfold from the virtual image P3. The angle and the angle of opening of the image light L that actually enters the user U's eye EY become close, resulting in a vergence-accommodation discrepancy. It can be made smaller.

[0036] Note that the display device 1 sets the illumination timing for each pixel P, so only a portion of the image is visible in one frame. In some cases, only pixel P may light up, for example, at the timing shown in Figure 5, only the image of the house is displayed. At the timing shown in Figure 6, only the car image is displayed, and at the timing shown in Figure 7, the helicopter image is displayed. Only the image is displayed. However, due to the afterimage effect created by playing multiple frames in sequence, The U recognizes a house, a car, and a helicopter in a single image. Also, one frame Even if you configure it to display the entire image (in this case, the house, car, and helicopter) within the display time, Good. In this case, the drive control unit 52 determines the focal position of the lens 32 within the display time of one frame. It is sufficient to move it for at least half a cycle of round-trip movement. This reduces the display time of one frame. Within this context, it is possible to cover all focal points within the reciprocal movement and display the entire image. This becomes possible.

[0037] (Processing flow) Next, the processing flow of the control device 40 described above will be explained. Figure 8 shows the embodiment of this product. This is a flowchart illustrating the processing flow of the control device. The control device 40 is the drive control unit 5 By step 2, the focal position of the microlens array 30 is moved back and forth in the Z direction at a predetermined period. The control device 40, using the timing setting unit 54, uses the depth information for each pixel P to determine the depth for each pixel P The irradiation position is set (step S10). The control device 40 controls the microlens array 30 The position of the focal point in the Z direction is acquired sequentially, and the focal point of the microlens array 30 is the irradiation position. It is determined for each pixel P whether it has reached within a predetermined distance (step S12). When the focal point of the array 30 reaches within a predetermined distance from the irradiation position (Step S12; Yes) In other words, it was determined that the focal point of the microlens array 30 had reached a predetermined distance from the irradiation position. If there is a pixel P that is to be illuminated, the timing setting unit 54 sets the illumination timing for that pixel P. Upon determining that the target has been reached, the irradiation control unit 56 illuminates the pixel P with image light L based on the image data. The light is emitted (step S14). Then, at the irradiation stop timing, the irradiation of the image light L is stopped. If the process does not terminate (step S16; No), return to step S10 and continue the process. On the other hand, the focal point of the microlens array 30 does not reach within a predetermined distance from the irradiation position. In the case of (step S12; No), that is, when the focal point of the microlens array 30 is at the illumination position If no pixel P is determined to have reached a predetermined distance from the location, the process returns to step S12. Until the focal point of the microlens array 30 reaches within a predetermined distance from the illumination position, light is directed to the pixel P. Do not irradiate. If the process is terminated in step S16 (step S16; Yes), Processing will now be terminated.

[0038] (effect) As described above, the display device 1 according to this embodiment provides a stereoscopic image to the user U. It consists of a display unit 10, a microlens array 30, a drive control unit 52, and a timing It includes a setting unit 54 and an illumination control unit 56. The display unit 10 displays a plurality of self-illuminating pixels P By including and directing image light L from pixel P to user U, a stereoscopic image is provided to user U. The microlens array 30 is positioned to be more elongated than the display unit 10 in the optical axis direction of the image light L. It is located on the U side and its focus can be changed. The drive control unit 52 controls the microlens array 3 The focal position of 0 is set along the optical axis direction of the image light L (Z direction in this embodiment) at a predetermined period. Move. The timing setting unit 54 adjusts the optical axis direction of the focal point of the microlens array 30 (main In this embodiment, the irradiation timing of the image light L is set for each pixel P based on its position in the Z direction. The irradiation control unit 56 irradiates the pixel P with image light L at the irradiation timing.

[0039] In this case, when displaying stereoscopic images, it is necessary to provide the user with stereoscopic images appropriately. In contrast, in this embodiment, the focal point of the microlens array 30 is made of light. Move it along the axis, and set the irradiation timing of the image light L based on the position of the focal point. Therefore, according to this embodiment, the appropriate timing based on the position of the optical axis of the focal point is used This makes it possible to deliver image light L to the U, and to appropriately provide a stereoscopic image to the user U. Furthermore, as mentioned above, when displaying stereoscopic images, a vergence-accommodation discrepancy can occur. In contrast, in this embodiment, the focal point of the microlens array 30 is the optical axis. While moving in the direction, the irradiation timing of the image light L is set based on the position of the optical axis of the focal point. By doing so, the angle of the luminous beam of the image light L is appropriately adjusted to reduce the vergence adjustment discrepancy. It is possible.

[0040] Furthermore, the timing setting unit 54 controls the position of the stereoscopic image in the depth direction (Z direction in this embodiment). The irradiation timing is set based on depth information indicating the position. According to this embodiment, the depth The irradiation timing is set using the information, so the image light L is adjusted in accordance with the displayed 3D image. This makes it possible to appropriately adjust the luminous beam angle, thereby reducing the convergence adjustment discrepancy. Cut.

[0041] Furthermore, the timing setting unit 54 determines that the opening angle of the light beam L of the image light from the pixel P is equal to the opening angle of the pixel P Image light L is directed towards the user U from the position in the depth direction of the portion of the 3D image displayed by the system. A microlens array that corresponds to the aperture angle of the light beam (virtual image aperture angle) when irradiated with Acquire information on the irradiation position, which is the position of the 30 focal points along the optical axis. Timing setting The part 54 is such that the focal point of the microlens array 30 is within a predetermined distance range from the irradiation position. The timing is set as the illumination timing for that pixel P. According to this embodiment By emitting light from pixels P where the light beam angle and the virtual image angle are close, the light beam angle and the virtual image angle are... This makes it possible to prevent light from being emitted from distant pixels P, thereby appropriately reducing the vergence adjustment discrepancy. It is possible.

[0042] Furthermore, the drive control unit 52 controls the focus of the multiple lenses 32 of the microlens array 30. The position in the Z direction is uniformly changed. According to this embodiment, the light beam in lens 32 By adjusting the opening angle, a stereoscopic image can be appropriately provided to user U.

[0043] Furthermore, the display device 1 is a head-mounted display. The display can appropriately provide stereoscopic images.

[0044] (Other examples) In the above explanation, the display unit 10 was described as a self-emissive display, but it is not limited to that. It is not possible. For example, as another example of the first embodiment, the display unit 10 is a display panel including multiple pixels. The configuration may include a panel and a light source that illuminates the display panel with light. In this case, for example, The display panel consists of multiple pixel electrodes arranged in a matrix and a liquid crystal layer filled with liquid crystal elements. It may be a liquid crystal display panel. The light source is provided, for example, on the back of the display panel. This may be a backlight or a side light provided on the side of the display panel. In such a configuration, the light source uniformly illuminates all pixels of the display panel. In other words, it does not control the illumination of each pixel individually. The light source is not limited to one that uniformly illuminates all pixels of the display panel; for example, multiple pixels It is possible to adjust the light intensity for each element, for example, by dividing the entire screen into several sections and adjusting the light intensity for each section. It may also be a system with adjustable dimming, known as local dimming.

[0045] In this example, since light emission cannot be controlled for each pixel, the user U's eye EY is facing it. It is preferable to provide a sensor (gaze detection unit) that detects the direction, that is, the gaze of user U. And the timing setting unit 54 adjusts the timing based on the eye gaze detection result of the eye gaze detection unit for user U. Based on this, the gaze position of user U is detected. The gaze position is the part of the 3D image that user U is looking at. This refers to the position on the display surface 10A of the image that user U is looking at, or in other words, the display surface 1 This refers to the position within the entire area of ​​0A that user U is focusing on. The timing setting unit 54 is U The results of gaze detection from the U, depth information of the stereoscopic image, and the focal point of the microlens array 30 The irradiation timing is set based on the position in the Z direction. Specifically, the timing setting The fixing unit 54 is based on depth information at the gaze position (position in the depth direction of the gaze position) Next, set the irradiation position. The timing setting unit 54 sets the position in the depth direction at the gaze position ( The angle of light beam (imaginary) when light is shone from the position of the virtual image that the U is fixated on to the eye EY. The Z of the focal point of the microlens array 30 when the aperture angle of the image light L matches the image aperture angle. The position in the direction is set as the irradiation position. The irradiation control unit 56 then controls this irradiation position At the irradiation timing when the focal point reaches within a predetermined distance range from the source, light is irradiated onto the light source. This causes image light L to be emitted from all pixels. In this example, user U is looking at Because the virtual image opening angle at the gaze position matches the opening angle of the image light L, the vergence-accommodation contradiction is resolved. It can be suppressed. Furthermore, at locations away from the gaze position, the virtual image opening angle and the image light L opening angle coincide. However, since it is outside the scope of what user U is focusing on, the impact on user U is small.

[0046] (Second Embodiment) Next, a second embodiment will be described. In the second embodiment, a microlens array This differs from the first embodiment in that the focal position of each lens can be adjusted individually. In the second embodiment, the parts that have the same configuration as the first embodiment will not be described.

[0047] Figure 9 is a schematic diagram of the configurations of the display device according to the second embodiment. As shown in Figure 9, 2. The display device 1a according to this embodiment includes a display unit 10, an eyepiece lens 20, and a microlens. It has a ray 30a and a control device 40a. In the second embodiment, the display unit 10 is Hereafter, the self-emissive display will be described as in the first embodiment, but the first The display may include a light source and a display panel as described in other examples of the embodiment. In the second embodiment, even if the display includes a light source and a display panel (i.e., the image Even if individual light emission control is not possible, user U gaze detection is unnecessary.

[0048] The microlens array 30a has the advantage of allowing the focus of multiple lenses 32a to be changed individually. Therefore, it differs from the first embodiment. That is, the microlens array 30a of the second embodiment is It can be described as an active-matrix type variable-focus microlens array. The microlens array 30a receives control signals from the control device 40a to control the microlens array 30a. Received the number.

[0049] Figure 10 is a schematic block diagram of the control device according to the second embodiment. As shown in Figure 10... In the second embodiment, the control unit 44 of the control device 40a includes an image information acquisition unit 50 and a drive unit. It includes a control unit 52a, a focal position setting unit 54a, and an irradiation control unit 56a. The control unit 44 is Image information is acquired by reading and executing a program (software) from the memory unit 42. The unit 50, the drive control unit 52a, the focal position setting unit 54a, and the irradiation control unit 56a are realized, These processes are executed. Note that the control unit 44 executes these processes using a single CPU. Alternatively, the system may be equipped with multiple CPUs, and the processing may be performed using those multiple CPUs. Furthermore, the image information acquisition unit 50, the drive control unit 52a, the focus position setting unit 54a, and the irradiation control unit 5 At least one of 6a may be implemented in hardware circuitry.

[0050] In the second embodiment, the irradiation control unit 56a, based on the image data, applies an image to each pixel P. Light L is irradiated. In the second embodiment, according to the focal position of the microlens array Without controlling the timing of the illumination of the image light L, the image light L is directed to each pixel P according to the image data. It is OK to irradiate it.

[0051] In the second embodiment, the focus position setting unit 54a acquires depth information for each pixel P. In other words, the focus position setting unit 54a sets the position of each pixel P in the depth direction (Z direction) Information is acquired. Then, the focus position setting unit 54a, based on the depth information of the pixel P, When illuminating a pixel P with image light L, at the focal position of the lens 32a facing the pixel P: A certain focal position is set. Note that the lens 32a opposite pixel P is the same as the pixel P. This lens 32a is the lens into which the image light L is incident, and will be referred to as the opposing lens as appropriate below.

[0052] The focus position setting unit 54a sets the depth position of the portion of the stereoscopic image displayed by the pixel P. The luminous beam when light is shone onto the eye EY from the position (the position of the virtual image formed by that pixel P) At the opening angle (virtual image opening angle), the actual light projected from that pixel P through the opposing lens to the eye EY is When the aperture angles of the image light L match, the focal positions of the opposing lenses in the Z direction are given to the opposing lenses. Set the focal position of the target lens. However, the virtual image aperture angle and the actual image light aperture L are set. The set focal position is not limited to the position where the virtual image aperture and the virtual image aperture are precisely aligned. The focal position of the opposing lens is set such that the difference between the actual image light L and the aperture angle is within a predetermined range. This can be used as the focal point. The predetermined range here can be set arbitrarily, but the convergence-accommodation discrepancy should be kept small. Therefore, the value is set such that the difference between the virtual image aperture angle and the aperture angle of the image light L becomes small. It is preferable to do so. Furthermore, the focus position setting unit 54a sets the quantized value in the depth direction. It may be used as the position in the depth direction of the pixel P used to set the focal position. That is, for example... For example, the depth direction is divided into multiple numerical ranges, and for each numerical range, the numerical range A predetermined value within the enclosure is set as the reference position. Then, the focus position setting unit 54a is The image information acquisition unit 50 extracts a numerical range that includes the position of the pixel P in the depth direction. The reference position for that numerical range is set in the depth direction of the pixel P used to set the focal position. Treat it as a position within the context.

[0053] The focus position setting unit 54a sets the target focus position for each opposing lens (each lens 32a). However, this is not limited to the set focal position being different for each lens 32a; for example, one image Opposite to a group of pixels P that make up the whole (for example, a group of pixels P that display the house image in Figure 5) The focal positions of the group of lenses 32a may be set to be the same.

[0054] The drive control unit 52a moves the position of the focal point of the lens 32a in the Z direction to the set focal point position. The drive control unit 52a individually controls the Z-direction position of the focal point of each lens 32a. By doing so, the focus of each lens 32a is set for each lens 32a. The focal position is set to the specified focal position. In the second embodiment, the focal position is set for each lens 32a. By doing so, the virtual image opening angle and the image light L opening angle are always kept the same across the entire display surface 10A. It becomes possible to make them do it.

[0055] Next, the processing flow of the control device 40a of the second embodiment will be described. Figure 11 shows the second embodiment. This is a flowchart illustrating the processing flow of the control device related to the state. The control device 40a is The point position setting unit 54a sets the focal position for each lens 32a based on the depth information for each pixel P. Set (step S20). The control device 40 is controlled by the drive control unit 52a, each The focal position of lens 32a is set to the set focal position for each lens 32a. Move to (step S22). In conjunction with this, the control device 40 moves the image data Based on this, image light L is irradiated onto each pixel P. This allows each lens 3 Position 2a at the set focal position. Note that if you do not want to terminate the process, (S If step S24 (No), return to step S20 and continue processing, and if processing is terminated (step S24; Yes), this process will be terminated.

[0056] As described above, the display device 1a according to the second embodiment provides a stereoscopic image to the user U. The system consists of a display unit 10, a microlens array 30a, and a focus position setting unit 54. a and drive control unit 52a are included. The display unit 10 includes a plurality of pixels and displays image light L to the user By reaching U, it provides a stereoscopic image to user U. The microlens array 30a is Multiple lenses 32 are provided on the user U side of the display unit 10 in the optical axis direction of the image light L. It has a and the focus of lens 32a can be changed individually. The focus position setting unit 54a is three-dimensional Based on depth information indicating the position in the depth direction of the image, the focal point in the optical axis direction of the image light L is The set focal position, which is the position, is set for each lens 32a. The drive control unit 52a controls lens 3 Move the focal point of 2a to the set focal point.

[0057] In this case, when displaying stereoscopic images, it is necessary to provide the user with stereoscopic images appropriately. In contrast, in this embodiment, the position in the depth direction of the stereoscopic image is determined based on the position in the depth direction of the stereoscopic image. The focal position of lens 32a is moved to the set focal position. Therefore, in this embodiment According to the system, the image light is directed to the user U at an appropriate beam angle based on the depth position of the stereoscopic image. It becomes possible to reach L, and to provide the user U with a proper 3D image. Furthermore, However, as mentioned above, when displaying stereoscopic images, a vergence-accommodation discrepancy may occur. In contrast, in this embodiment, the opening angle of the light beam of the image light L is appropriately adjusted to control the convergence. The inconsistency can be minimized. Furthermore, in the second embodiment, each lens 32a is individually By setting a separate focal position, the virtual image opening angle and image light L can be set across the entire display surface 10A. This makes it possible to always match the opening angle.

[0058] Furthermore, the focus position setting unit 54a determines the opening angle of the light beam of the image light L passing through the lens 32a, and its From the position in the depth direction of the portion of the 3D image displayed by the image light L toward the user U The difference between the luminous beam angle when image light L is irradiated (virtual image luminous beam angle) and the luminous beam angle when image light L is irradiated is within a predetermined range. The focal position of lens 32a in the optical axis direction of image light L is set as the set focal position. According to this embodiment, by reducing the difference between the luminous beam aperture angle and the virtual image aperture angle, The congestion-regulation contradiction can be appropriately minimized.

[0059] The display unit 10 of the second embodiment is a display panel including multiple pixels, as described in other examples. The configuration may also include a light source that illuminates the display panel, or multiple self-emitting elements. The configuration may include pixels P. In any case, the stereoscopic image is provided appropriately to the user U. can.

[0060] (modified version) Next, a modified example will be described. The display device 1b in the modified example replaces the eyepiece lens 20. This differs from the second embodiment in that it magnifies the display using a concave mirror 20C. Parts that are common to the second embodiment and its configuration will be omitted from the explanation.

[0061] Figure 12 is a schematic diagram of a modified display device. As shown in Figure 12, The display device 1b does not have an eyepiece lens 20 and displays in the optical axis direction of the image light L. A half-mirror 20B and a concave mirror 20C are provided on the user U's eye EY side of section 10. Yes. The half-mirror 20B and the concave mirror 20C can also be considered optical elements. In the modified example... The image light L emitted from the display unit 10 is reflected by the half mirror 20B and then reflected by the concave mirror 20 The light enters C. The image light L that enters the concave mirror 20C has a slight divergence angle at the concave mirror 20C. The light then becomes nearly parallel, passes through the half-mirror 20B, and enters the user U's eye EY. .

[0062] In the modified example, similar to the second embodiment, each lens 32 of the microlens array 30a The focal position of a is controlled individually. Therefore, even with a modified configuration, the second implementation is Similar to the state, it is possible to appropriately provide a stereoscopic image to the user U and appropriately adjust the opening angle of the light beam L of the image light. By optimizing the system, the vergence-accommodation contradiction can be reduced.

[0063] Furthermore, the modified form can also be applied to the first embodiment. In addition, the configuration of the display device is different for each embodiment. The form may be different from the variations shown in Figure 12.

[0064] Although embodiments of the present invention have been described above, the embodiments are not limited by the content of these embodiments. It is not something that can be easily imagined by a person skilled in the art. Furthermore, the aforementioned components include those that can be easily imagined by a person skilled in the art, and those that are substantial. This includes things that are essentially the same, or within the so-called equivalent range. Furthermore, the aforementioned components are appropriate They can be combined as desired, and the configurations of each embodiment can also be combined. Furthermore, various omissions, substitutions, or modifications of the components may be made without departing from the gist of the embodiments described above. Further actions can be taken. [Explanation of symbols]

[0065] 1 Display device 10 Display 20 eyepieces 30, 30a Microlens Array 32, 32a lenses 40, 40a Control device 50 Image Information Acquisition Unit 52, 52a Drive control unit 54 Timing setting section 54a Focus position setting section 56, 56a Irradiation control unit L Image Light P pixels U User

Claims

1. A display unit containing multiple pixels that provides an image to the user by illuminating it with light, It is located on the user side of the display unit and has multiple lenses, and the focus of the lenses A changeable microlens array, A drive control unit that moves the focal position of the lens back and forth at a predetermined period, A timing setting unit sets the timing for irradiating the light based on the focal position of the lens and depth information indicating the position in the depth direction of the image. including, Display device.

2. The system further includes an image information acquisition unit that acquires the aforementioned depth information, The timing setting unit sets the timing for irradiating the light to occur when the focal position of the lens reaches within a predetermined distance from the position of the depth information. The display device according to claim 1.

3. The drive control unit moves the focal position of the lens back and forth so that the display unit emits the light at least once within the display time of one frame. The display device according to claim 1 or 2.

4. A drive control step that causes the focal position of a lens in a microlens array, which includes multiple pixels, is located on the user side of a display unit that emits light, has multiple lenses, and whose focal point can be changed, to reciprocate at a predetermined period; A timing setting step to set the timing for irradiating the light based on the focal position of the lens and depth information indicating the position in the depth direction of the image, Having, Display method.

5. A drive control step that causes the focal position of a lens in a microlens array, which includes multiple pixels, is located on the user side of a display unit that emits light, has multiple lenses, and whose focal point can be changed, to reciprocate at a predetermined period; A timing setting step to set the timing for irradiating the light based on the focal position of the lens and depth information indicating the position in the depth direction of the image, Make the computer execute it. Display program.

Images