Display device, display control method, and display control program

Abstract

To provide the user with appropriate images.SOLUTION: A display 1 includes a display unit 10 that includes a plurality of pixels P and provides images to the user by irradiating light from the pixels P, a microlens array 30 that is provided on the user side of the display unit 10 and can change its focus, a drive control unit that changes the position of the focus of the microlens array 30 at a predetermined period, and a timing setting unit that sets the irradiation timing of the image light L for each pixel P, based on the position of the focus of the microlens array 30.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a display device, a display control method, and a display control program. [Background technology]

[0002] There is known a display device that displays a stereoscopic image by making the right and left eyes of a user view images with different parallax and utilizing the difference in convergence. One such display device is a so-called head-mounted display (HMD) that is mounted on the user's head. For example, Patent Document 1 describes a head-mounted display in which a microlens array is disposed between a display and an optical system. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2019 / 044501 Summary of the Invention [Problem to be solved by the invention]

[0004] Here, in a display device that displays a stereoscopic image, it is required to provide the image appropriately to the user.

[0005] In view of the above-mentioned problems, an object of the present invention is to provide a display device, a display method, and a program capable of providing an appropriate image to a user. [Means for solving the problem]

[0006] A display device according to one aspect of the present invention includes a display unit including a plurality of pixels and providing an image to the user by irradiating light from the pixels; a microlens array that is provided closer to the user than the display unit and has a changeable focus; a drive control unit that changes the focus position of the microlens array at a predetermined cycle; and a timing setting unit that sets the irradiation timing of the image light for each pixel based on the focus position of the microlens array. The timing setting unit sets the irradiation timing based also on depth information indicating a position in the depth direction of the image, and the timing setting unit acquires information on the irradiation position, which is the focal position of the microlens array corresponding to the opening angle of the light beam from the pixel when the light is irradiated toward the user from a depth direction position in the part of the image displayed by that pixel, and sets the timing at which the focal position of the microlens array is within a predetermined distance range of the irradiation position as the irradiation timing for that pixel.

[0007] A display control method according to one aspect of the present invention includes a drive control step of changing a focal position of a microlens array, which is provided on a user side of a display unit including a plurality of pixels and irradiating light from the pixels and has a variable focus, at a predetermined period, and a timing setting step of setting a timing of irradiating the light for each of the pixels based on the focal position of the microlens array. In the timing setting step, the irradiation timing is set based also on depth information indicating the depth position of the image, and in the timing setting step, information on the irradiation position is obtained, which is the focal position of the microlens array corresponding to the opening angle of the light beam from the pixel when the light is irradiated toward the user from a depth position in the part of the image displayed by that pixel, and the timing at which the focal position of the microlens array is within a predetermined distance range of the irradiation position is set as the irradiation timing for that pixel.

[0008] A display control program according to one aspect of the present invention causes a computer to execute a drive control step of changing a focal position of a focus-changing microlens array, which is provided on a user side of a display unit that includes a plurality of pixels and irradiates light from the pixels, at a predetermined period, and a timing setting step of setting a timing of irradiating the light for each of the pixels based on the focal position of the microlens array. In the timing setting step, the irradiation timing is set based also on depth information indicating the depth position of the image, and in the timing setting step, information on the irradiation position is obtained, which is the focal position of the microlens array corresponding to the opening angle of the light beam from the pixel when the light is irradiated toward the user from a depth position in the part of the image displayed by that pixel, and the timing at which the focal position of the microlens array is within a predetermined distance range of the irradiation position is set as the irradiation timing for that pixel. Effect of the Invention

[0009] According to the present invention, an image can be appropriately provided to a user. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic diagram for explaining the convergence accommodation conflict. [Diagram 2] FIG. 2 is a schematic diagram of the display device according to the first embodiment. [Diagram 3] FIG. 3 is a schematic diagram of each component of the display device according to the first embodiment. [Figure 4]FIG. 4 is a schematic block diagram of the control device according to the first embodiment. [Diagram 5] FIG. 5 is a schematic diagram for explaining the setting of the irradiation timing. [Figure 6] FIG. 6 is a schematic diagram for explaining the setting of the irradiation timing. [Figure 7] FIG. 7 is a schematic diagram for explaining the setting of the irradiation timing. [Figure 8] FIG. 8 is a flowchart illustrating a process flow of the control device according to the present embodiment. [Figure 9] FIG. 9 is a schematic diagram of each component of the display device according to the second embodiment. [Figure 10] FIG. 10 is a schematic block diagram of a control device according to the second embodiment. [Figure 11] FIG. 11 is a flowchart illustrating a process flow of the control device according to the second embodiment. [Figure 12] FIG. 12 is a schematic diagram of a display device according to a modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to the embodiment described below.

[0012] (Convergence regulation conflict) FIG. 1 is a schematic diagram for explaining the convergence accommodation conflict. A display device for displaying a stereoscopic image displays a stereoscopic image by making the right eye and the left eye of a user view images with different parallax and utilizing the difference in convergence. When displaying a stereoscopic image, the display surface on which the image is actually displayed becomes the focal position of the user's eyes, and the position where the lines of sight of the left and right eyes intersect becomes the convergence position. However, as shown in the example of FIG. 1, in a stereoscopic image, the positions of the focal position PO1 and the convergence position PO2 in the Z direction, which is the depth direction of the stereoscopic image, may shift. When the positions of the focal position PO1 and the convergence position PO2 shift, a so-called convergence accommodation conflict occurs, which causes eye strain and so-called 3D sickness. For this reason, it is required to suppress the convergence accommodation conflict. Note that FIG. 1 (A) shows an example in which the focal position PO1 is closer to the user's eye EY than the convergence position PO2, and FIG. 1 (B) shows an example in which the convergence position PO2 is closer to the user's eye EY than the focal position PO1.

[0013] (First embodiment) (Overall configuration of the display device) FIG. 2 is a schematic diagram of a display device according to the first embodiment. The display device 1 according to the first embodiment is a display device that displays a stereoscopic image. As shown in FIG. 2, the display device 1 is a so-called HMD (Head Mount Display) that is worn on the head of a user U. For example, the display device 1 has a display unit 10 worn at a position facing the eye EY of the user U. The display device 1 displays an image on the display unit 10 to provide content to the user U. Note that the configuration of the display device 1 shown in FIG. 2 is an example. For example, the display device 1 may include an audio output unit (speaker) that is worn on the ear of the user U.

[0014] Since the display device 1 is worn by the user U in this manner, the position of the display device 1 with respect to the eyes EY of the user U is fixed. The display device 1 is not limited to being an HMD worn by the user U, and may be a display device fixed to equipment, etc. Even in such a case, it is preferable that the position of the display device 1 with respect to the eyes EY of the user U is fixed, for example, by fixing the position of the display device 1 with respect to the seat of the user U.

[0015] 3 is a schematic diagram of each component of the display device according to the first embodiment. As shown in FIG. 3, the display device 1 has a display unit 10, an eyepiece lens 20, a microlens array 30, and a control device 40.

[0016] (Display) The display unit 10 is a device that displays a stereoscopic image. The display unit 10 is a display having a plurality of self-emitting pixels P (display elements) arranged in a matrix. Since each pixel P of the display unit 10 is self-emitting, the display unit 10 can control the emission (light irradiation) of each pixel P individually. Each pixel P of the display unit 10 may be, for example, an organic light emitting diode (OLED) or an inorganic light emitting diode (micro LED). The surface on which an image is displayed on the display unit 10 is referred to as a display surface 10A. Hereinafter, the direction from the display surface 10A toward the eye EY of the user U is referred to as a direction Z1, and the opposite direction to the direction Z1, i.e., the direction from the eye EY of the user U toward the display surface 10A is referred to as a direction Z2. When there is no need to distinguish between the directions Z1 and Z2, the direction will be referred to as a direction Z. Note that in FIG. 3, the surface of the display unit 10 on the side of the eye EY of the user U is referred to as the display surface 10A, but the display surface 10A is not limited to being the surface on the side of the eye EY of the user U, and may be located inside the surface on the side of the eye EY of the user U. Note that the display unit 10 receives a control signal for controlling the pixels P of the display unit 10 from a control device 40 described later.

[0017] The display unit 10 provides a stereoscopic image to the user U by making the image light L, which is light irradiated (emitted) from each pixel P, reach the eye EY of the user U. More specifically, the display unit 10 controls the emission of each pixel P so that an image for the left eye and an image for the right eye are provided. Of the image light L from each pixel P, the image light L from the pixel P corresponding to the image for the left eye is incident on the left eye of the user U, and the image light L from the pixel P corresponding to the image for the right eye is incident on the left eye of the user U, thereby providing a stereoscopic image to the user U.

[0018] (Eyepiece) The eyepiece 20 is provided on the Z1 direction side of the display unit 10. The eyepiece 20 is an optical element that transmits light (image light). More specifically, the eyepiece 20 is the optical element (lens) that is closest to the eye EY of the user U in the display device 1. The image light L emitted from the display unit 10 passes through the eyepiece 20 and enters the eye EY of the user U. Note that in this embodiment, the optical axis direction in the optical path of the image light L from the eyepiece 20 (eyepiece) to the eye EY of the user U can also be referred to as the Z direction.

[0019] In the example of FIG. 3, only the eyepiece lens 20 and the microlens array 30 are shown as optical elements on the Z1 direction side of the display unit 10, but the present invention is not limited to this and other optical elements may be provided.

[0020] (Microlens Array) The microlens array 30 is provided on the user U side of the display unit 10 in the optical axis direction of the image light L. More specifically, the microlens array 30 is provided between the display unit 10 and the eyepiece lens 20 in the optical axis direction of the image light L. The microlens array 30 is an optical element in which a plurality of lenses 32 are arranged in a matrix on a plane parallel to the display surface 10A. In this embodiment, the pitch of each lens 32 of the microlens array 30, that is, the distance between the centers of adjacent lenses 32, is approximately the same as the pitch of the pixels P of the display unit 10 (the distance between the centers of adjacent pixels P), for example. Each lens 32 of the microlens array 30 is provided at a position facing each pixel P of the display unit 10 in the optical axis direction of the image light L. The microlens array 30 receives a control signal for controlling the microlens array 30 from a control device 40 described later.

[0021] The microlens array 30 is a so-called variable-focus microlens array that can change the focal position (distance from the lens 32 to the focal point) of each lens 32 in the Z direction (optical axis direction of the image light L). In the example of the first embodiment, the microlens array 30 changes the focal position of each lens 32 uniformly, and the focal position of each lens 32 in the Z direction at the same timing is the same. The microlens array 30 may have any structure, and may be configured by combining, for example, liquid crystal and a Fresnel lens. For example, a variable-focus lens disclosed in the document "Volumetric three-dimensional display using a liquid crystal variable-focus lens" (Shiro Suyama, Tokushima University, Optics, Vol. 40, No. 12 (2011)) may be used as each lens 32 of the microlens array 30.

[0022] (Control device) The control device 40 is a device that controls each part of the display device 1. FIG. 4 is a schematic block diagram of the control device according to the first embodiment. In this embodiment, the control device 40 is a computer, and has a storage unit 42 and a control unit 44. The storage unit 42 is a memory that stores various information such as the calculation contents and programs of the control unit 44, and includes at least one of a main storage device such as a RAM (Random Access Memory), a ROM (Read Only Memory), and an external storage device such as an HDD (Hard Disk Drive). The program for the control unit 44 stored in the storage unit 42 may be stored in a recording medium that can be read by the control device 40.

[0023] The control unit 44 is a calculation device and includes a calculation circuit such as a CPU (Central Processing Unit). The control unit 44 includes an image information acquisition unit 50, a drive control unit 52, a timing setting unit 54, and an irradiation control unit 56. The control unit 44 realizes the image information acquisition unit 50, the drive control unit 52, the timing setting unit 54, and the irradiation control unit 56 by reading and executing a program (software) from the storage unit 42, and executes the processes. The control unit 44 may execute these processes by one CPU, or may include multiple CPUs and execute the processes by the multiple CPUs. At least one of the image information acquisition unit 50, the drive control unit 52, the timing setting unit 54, and the irradiation control unit 56 may be realized by a hardware circuit.

[0024] (Image information acquisition section) The image information acquisition unit 50 acquires image data of a stereoscopic image displayed by the display unit 10. That is, the image information acquisition unit 50 acquires image data of an image for the left eye and image data of an image for the right eye. The image information acquisition unit 50 also acquires depth information indicating a position in the depth direction of the stereoscopic image. The position in the depth direction of the stereoscopic image refers to the position in the depth direction of a virtual image visually recognized by the user U when an image is displayed on the display surface 10A. The depth direction can also be said to be a direction perpendicular to the display surface 10A of the display unit 10, and is the Z direction in this embodiment. The depth information is associated with the image data. Furthermore, a position in the depth direction is set for each image included in one frame of the stereoscopic image, in other words, a position in the depth direction is set for each position on the display surface 10A. Therefore, it can be said that the image information acquisition unit 50 acquires depth information for each position on the display surface 10A for the stereoscopic image. In the stereoscopic image, a position in the depth direction is set for each pixel P, but the positions in the depth direction may be set to be the same for multiple pixels P constituting one image. The image information acquisition unit 50 may acquire image data and depth information by any method, and may read image data and depth information stored in advance in the storage unit 42, or may receive image data and depth information via a communication unit (not shown). The image information acquisition unit 50 may acquire depth information by calculating a position in the depth direction based on the image data.

[0025] (Drive control unit) The drive control unit 52 controls the microlens array 30 to move the focal position of each lens 32 of the microlens array 30 in the Z direction. The drive control unit 52 moves the focal position of the lens 32 in the Z direction, for example, by controlling application of a voltage to a liquid crystal element included in the microlens array 30. The drive control unit 52 moves the focal position of the lens 32 so that the focal position of the lens 32 repeats a reciprocating movement (oscillation) in the Z direction in which the focal position of the lens 32 moves a predetermined distance in the Z1 direction and then moves a predetermined distance in the Z2 direction. The drive control unit 52 moves the focal position of the lens 32 in a predetermined cycle along the Z direction. In other words, the drive control unit 52 causes the focal position of the lens 32 to move back and forth in the Z direction in a predetermined cycle. In this embodiment, the cycle of the reciprocating movement in the Z direction (the time it takes for the display unit 10 to return to its original position in the Z direction) is constant, but is not limited to being constant and may be changed.

[0026] (Timing setting unit and irradiation control unit) The timing setting unit 54 sets the irradiation timing of the image light L for each pixel P of the display unit 10. The irradiation control unit 56 controls the pixels P of the display unit 10 based on the image data to irradiate the pixels P with the image light L. The irradiation control unit 56 irradiates the image light L for each pixel P at the irradiation timing for each pixel P set by the timing setting unit 54. That is, the irradiation control unit 56 irradiates a certain pixel P of the display unit 10 with the image light L at the irradiation timing set for that pixel P. The timing setting unit 54 sets the irradiation timing based on the focal position of the microlens array 30 (lens 32) in the Z direction (optical axis direction of the image light L). More specifically, the timing setting unit 54 sets the irradiation timing based on the depth information of the stereoscopic image and the focal position of the microlens array 30 in the Z direction. The timing setting unit 54 sets the irradiation timing for each pixel P, but is not limited to different irradiation timings for each pixel P. For example, the irradiation timings of a group of pixels P constituting one image (such as a group of pixels P displaying an image of a house in FIG. 5) may be set to be the same. The settings by the timing setting unit 54 will be described in more detail below.

[0027] 5 to 7 are schematic diagrams for explaining the setting of the irradiation timing. For example, as shown in FIG. 5, the image light L emitted from the pixel P of the display unit 10 is incident on the eye EY of the user U as a light beam having a predetermined opening angle. In this case, the user U unconsciously changes the thickness of the crystalline lens of the eyeball to adjust the focus on the retina to match the opening angle of the light beam. The convergence accommodation conflict refers to a mismatch between the degree of convergence of the left and right eyes (degree of deviation of the left and right eyes) and the state in which the focus is adjusted to the opening angle of the light beam. In contrast, the display device 1 of this embodiment emits the image light L so that the difference between the virtual image opening angle (angle θ1A in the example of FIG. 5), which is the opening angle when it is assumed that light is emitted from the virtual image (convergence position) and enters the eye EY, and the opening angle of the image light L when the image light L is actually emitted from the pixel P and enters the eye EY, becomes small. Here, the image light L changes (refracts) the aperture angle of the light beam when passing through the lens 32 of the microlens array 30, and the degree of change in the aperture angle of the light beam is determined according to the focal position of the lens 32. Therefore, the timing setting unit 54 sets the timing when the focal position of the lens 32 reaches a position where the difference between the virtual image aperture angle and the aperture angle of the image light L becomes small as the irradiation timing.

[0028] More specifically, the timing setting unit 54 acquires depth information for each pixel P. That is, the timing setting unit 54 acquires information on the position in the depth direction (Z direction) for each pixel P. Then, based on the depth information of the pixel P, the timing setting unit 54 sets an irradiation position, which is the position of the focus of the lens 32 when starting irradiation of the image light L to the pixel P. The timing setting unit 54 sets, as the irradiation position for the pixel P, the position of the focus of the lens 32 in the Z direction when the opening angle of the image light L that actually enters the eye EY from the pixel P through the lens 32 matches the opening angle of the light flux (virtual image opening angle) when light is irradiated to the eye EY from the depth direction position of the portion of the stereoscopic image displayed by the pixel P (the position of the virtual image formed by the pixel P). Then, the timing setting unit 54 sets, as the irradiation timing for the pixel P, the timing at which the focus of the lens 32 reaches a position whose distance from the irradiation position is within a predetermined distance range. The timing setting unit 54 sets the irradiation position for each pixel P and sets the irradiation timing for each pixel P. Note that the timing setting unit 54 sets the irradiation position and irradiation timing for each pixel P, but is not limited to different irradiation positions and irradiation timings for each pixel P. For example, the irradiation positions and irradiation timings of a group of pixels P constituting one image (such as a group of pixels P displaying an image of a house in FIG. 5) may be set to be the same.

[0029] In this embodiment, the timing setting unit 54 sequentially acquires information on the position of the focal point of the lens 32 in the Z direction, and when the position of the focal point of the lens 32 in the Z direction approaches the irradiation position by a predetermined distance, it is determined that the irradiation timing has arrived. The timing setting unit 54 may acquire information on the position of the focal point of the lens 32 in the Z direction by any method. For example, when the focal point of the lens 32 is reciprocated in the Z direction at a predetermined period, the position (predicted position) of the focal point of the lens 32 in the Z direction for each time can be grasped. Therefore, the timing setting unit 54 may acquire information on the position of the focal point of the lens 32 in the Z direction from the time information. In this case, the timing setting unit 54 may set the time when the focal point of the lens 32 reaches a predetermined distance from the irradiation position as the irradiation timing based on the information on the predicted position of the focal point of the lens 32 for each time and the information on the irradiation position, and when the current time reaches the irradiation timing, it may determine that the focal point of the lens 32 has reached the irradiation position, and cause the pixel P to be irradiated with the image light L. In addition, the predetermined distance here may be set arbitrarily, but in order to reduce the convergence accommodation contradiction, it is preferable to set it to a distance that reduces the difference between the virtual image opening angle and the opening angle of the image light L. In addition, the timing setting unit 54 may use the quantized value in the depth direction as the position in the depth direction of the pixel P used to set the irradiation position. That is, for example, the depth direction is divided into a plurality of numerical ranges, and a predetermined value within the numerical range is set as a reference position for each numerical range. Then, the timing setting unit 54 extracts a numerical range that includes the position in the depth direction of the pixel P acquired by the image information acquisition unit 50, and treats the reference position for the numerical range as the position in the depth direction of the pixel P used to set the irradiation position.

[0030] In this manner, the timing setting unit 54 sets the irradiation timing for each pixel P. Furthermore, the timing setting unit 54 sets a timing after the irradiation timing as the irradiation stop timing. When it is determined that the irradiation timing for a certain pixel P has been reached, the irradiation control unit 56 starts irradiating the pixel P with the image light L. The irradiation control unit 56 irradiates the pixel P with the image light L from the irradiation timing to the irradiation stop timing, and when the irradiation stop timing is reached, stops irradiating the pixel P with the image light L. The irradiation stop timing may be set arbitrarily. For example, a predetermined time after the irradiation timing may be set as the irradiation stop timing, or a timing at which the distance between the focal position of the lens 32 and the irradiation position falls outside a predetermined distance range immediately after the irradiation timing may be set as the irradiation stop timing.

[0031] In this way, the display device 1 irradiates the pixel P with the image light L when the irradiation timing is reached, and stops irradiating the image light L when the irradiation stop timing is reached. The image light L is incident on the eye EY of the user U between the irradiation timing and the irradiation stop timing. Therefore, the opening angle of the light flux of the image light L incident on the eye EY becomes close to the virtual image opening angle from the virtual image formed by the pixel P, and the convergence accommodation contradiction can be reduced. In addition, since the position of the focal point of the lens 32 moves back and forth in the Z direction, the distance from the irradiation position repeatedly becomes within a predetermined distance range and out of the predetermined distance range. The control device 40 irradiates the pixel P with the image light L every time the distance between the position of the focal point of the lens 32 and the irradiation position becomes within the predetermined distance range, that is, every time the irradiation timing is reached. Therefore, the user U views a stereoscopic image as a moving image. In addition, since the position of the focal point of the lens 32 moves back and forth in the Z direction, there are two times in one period of the back and forth movement when the distance from the irradiation position becomes a predetermined distance. Therefore, it is desirable to set the frequency of the reciprocating movement of the focal position of the lens 32 to at least half the frame rate of the stereoscopic image. However, the frequency (period) of the reciprocating movement of the focal position of the lens 32 may be set arbitrarily.

[0032] An example of the setting of the above-described irradiation timing will be described with reference to Fig. 5 to Fig. 7. In the following, an example will be taken in which an image of a house, an image of a car, and an image of a helicopter are displayed as a stereoscopic image, and the positions in the depth direction (Z direction) of the image of the car, the image of the house, and the image of the helicopter are farther away from the eye EY of the user U in the order of the image of the car, the image of the house, and the image of the helicopter. That is, the virtual image P2 of the image of the car is located on the Z1 side of the virtual image P1 of the image of the house, and the virtual image P3 of the image of the helicopter is located on the Z2 side of the virtual image P1 of the image of the house.

[0033] FIG. 5 is an example of a case where a virtual image P1 of a house image is visually recognized by a user U. In the example of FIG. 5, the opening angle of the light flux (virtual image opening angle) when light is irradiated from the virtual image P1 (the position in the depth direction of the pixel P constituting the image of the house) to the eye EY is set to angle θ1A. Then, when the position of the focus of the lens 32 in the Z direction is the first position, the opening angle of the light flux of the image light L from the pixel P constituting the image of the house is set to angle θ1A. In this case, the first position becomes the irradiation position for the pixel P constituting the image of the house, and the timing setting unit 54 sets the timing when the focus of the lens 32 reaches a position whose distance from the first position is within a predetermined distance range as the irradiation timing for the pixel P constituting the image of the house. When the irradiation timing arrives, the irradiation control unit 56 causes the pixel P constituting the image of the house to be irradiated with the image light L. As a result, the virtual image opening angle from the virtual image P1 of the image of the house and the opening angle of the image light L that actually enters the eye EY of the user U become closer, and the convergence accommodation contradiction can be reduced. In addition, since the image light L is refracted by the eyepiece lens 20 and enters the eye EY, the divergence angle of the light flux of the image light L here refers to the divergence angle of the light flux of the image light L after it has passed through the eyepiece lens 20.

[0034] FIG. 6 is an example of a case where a virtual image P2 of a car image is visually recognized by a user U. In the example of FIG. 6, the opening angle of the light flux (virtual image opening angle) when light is irradiated from the virtual image P2 (the position in the depth direction of the pixel P constituting the image of the car) to the eye EY is set to angle θ2A. When the position of the focal point of the lens 32 in the Z direction is the second position, the opening angle of the light flux of the image light L from the pixel P constituting the image of the car is set to angle θ2A. Since the virtual image P2 is visually recognized in the Z1 direction more than the virtual image P1, the angle θ2A is larger than the angle θ1A in FIG. 5, and the degree of enlargement of the opening angle of the light flux by the lens 32 is larger (or the degree of reduction is smaller) when the second position is the focal point than when the first position is the focal point. In this case, the second position is the irradiation position for the pixel P constituting the image of the car, and the timing setting unit 54 sets the timing when the focal point of the lens 32 reaches a position whose distance from the second position is within a predetermined distance range as the irradiation timing for the pixel P constituting the image of the car. When the irradiation timing arrives, the irradiation control unit 56 irradiates the pixels P constituting the image of the car with light. This makes the virtual image opening angle from the virtual image P2 of the image of the car closer to the opening angle of the image light L actually incident on the eye EY of the user U, thereby reducing the convergence accommodation conflict.

[0035] FIG. 7 is an example of a case where a virtual image P3 of a helicopter image is visually recognized by a user U. In the example of FIG. 7, the opening angle of the light flux (virtual image opening angle) when light is irradiated from the virtual image P3 (the depth direction position of the pixel P constituting the helicopter image) to the eye EY is set to angle θ3A. When the focal position of the lens 32 in the Z direction is the third position, the opening angle of the light flux of the image light L from the pixel P constituting the helicopter image is set to angle θ3A. Note that since the virtual image P3 is visually recognized in the Z2 direction more than the virtual image P1, the angle θ3A is smaller than the angle θ1A in FIG. 5, and the degree of enlargement (or the degree of reduction) of the opening angle of the light flux by the lens 32 is smaller when the focal position is the third position than when the focal position is the first position. In this case, the third position becomes the irradiation position for the pixel P constituting the helicopter image, and the timing setting unit 54 sets the timing when the focus of the lens 32 reaches a position whose distance from the third position is within a predetermined distance range as the irradiation timing for the pixel P constituting the helicopter image. When the irradiation timing arrives, the irradiation control unit 56 causes the pixel P constituting the helicopter image to be irradiated with the image light L. This makes the virtual image opening angle from the virtual image P3 of the helicopter image closer to the opening angle of the image light L that actually enters the eye EY of the user U, thereby reducing the convergence accommodation conflict.

[0036] Since the display device 1 sets the illumination timing for each pixel P, there are cases where only some of the pixels P are lit in one frame. For example, only the image of the house is displayed at the timing of FIG. 5, only the image of the car is displayed at the timing of FIG. 6, and only the image of the helicopter is displayed at the timing of FIG. 7. However, due to the afterimage effect caused by consecutively displaying a plurality of frames, the user U recognizes that he or she is in one image of the house, the car, and the helicopter. Also, the entire image (here, all of the house, the car, and the helicopter) may be displayed within the display time of one frame. In this case, the drive control unit 52 may move the focal position of the lens 32 by at least half the period of the reciprocating movement within the display time of one frame. This makes it possible to cover all the focal positions in the reciprocating movement within the display time of one frame, and to display the entire image.

[0037] (Processing flow) Next, the process flow of the control device 40 described above will be described. FIG. 8 is a flowchart for explaining the process flow of the control device according to this embodiment. The control device 40 causes the drive control unit 52 to move the focal position of the microlens array 30 back and forth in the Z direction at a predetermined cycle. The control device 40 sets the irradiation position for each pixel P from the depth information for each pixel P using the timing setting unit 54 (step S10). The control device 40 sequentially acquires the position of the focal point of the microlens array 30 in the Z direction, and judges for each pixel P whether the focal point of the microlens array 30 has reached within a predetermined distance from the irradiation position (step S12). When the focal point of the microlens array 30 has reached within a predetermined distance from the irradiation position (step S12; Yes), that is, when there is a pixel P for which it is judged that the focal point of the microlens array 30 has reached within a predetermined distance from the irradiation position, the timing setting unit 54 judges that the pixel P has reached the irradiation timing, and the irradiation control unit 56 causes the pixel P to be irradiated with image light L based on the image data (step S14). Thereafter, the irradiation of the image light L is stopped at the irradiation stop timing, and if the process is not to be ended (step S16; No), the process returns to step S10 and continues. On the other hand, if the focal point of the microlens array 30 has not reached within a predetermined distance from the irradiation position (step S12; No), that is, if there is no pixel P for which it is determined that the focal point of the microlens array 30 has reached within a predetermined distance from the irradiation position, the process returns to step S12 and light is not irradiated onto the pixel P until the focal point of the microlens array 30 reaches within a predetermined distance from the irradiation position. If the process is to be ended at step S16 (step S16; Yes), the process ends.

[0038] (effect) As described above, the display device 1 according to the present embodiment provides a stereoscopic image to the user U, and includes the display unit 10, the microlens array 30, the drive control unit 52, the timing setting unit 54, and the irradiation control unit 56. The display unit 10 includes a plurality of self-emitting pixels P, and provides a stereoscopic image to the user U by making the image light L from the pixels P reach the user U. The microlens array 30 is provided closer to the user U than the display unit 10 in the optical axis direction of the image light L, and is capable of changing its focus. The drive control unit 52 moves the position of the focus of the microlens array 30 along the optical axis direction of the image light L (Z direction in this embodiment) at a predetermined period. The timing setting unit 54 sets the irradiation timing of the image light L for each pixel P based on the position of the focus of the microlens array 30 in the optical axis direction (Z direction in this embodiment). The irradiation control unit 56 causes the pixel P to be irradiated with the image light L at the irradiation timing.

[0039] Here, when displaying a stereoscopic image, it is required to provide the user with a stereoscopic image appropriately. In contrast, in this embodiment, the focus of the microlens array 30 is moved in the optical axis direction, and the timing of irradiating the image light L is set based on the position of the focus. Therefore, according to this embodiment, it is possible to make the image light L reach the user U at an appropriate timing based on the position of the focus in the optical axis direction, and a stereoscopic image can be appropriately provided to the user U. Furthermore, as described above, when displaying a stereoscopic image, a convergence accommodation conflict may occur. In contrast, in this embodiment, the focus of the microlens array 30 is moved in the optical axis direction, and the timing of irradiating the image light L is set based on the position of the focus in the optical axis direction, thereby appropriately adjusting the opening angle of the light flux of the image light L and reducing the convergence accommodation conflict.

[0040] The timing setting unit 54 also sets the irradiation timing based on depth information indicating the position in the depth direction (Z direction in this embodiment) of the stereoscopic image. According to this embodiment, since the irradiation timing is set using the depth information as well, it is possible to appropriately adjust the opening angle of the light flux of the image light L in accordance with the stereoscopic image to be displayed, and it is possible to reduce the convergence accommodation conflict.

[0041] Further, the timing setting unit 54 acquires information on the irradiation position, which is a position in the optical axis direction of the focus of the microlens array 30 such that the opening angle of the light beam of the image light L from the pixel P corresponds to the opening angle of the light beam (virtual image opening angle) when the image light L is irradiated toward the user U from a position in the depth direction in the portion of the stereoscopic image displayed by the pixel P. The timing setting unit 54 sets the timing when the focus of the microlens array 30 is within a predetermined distance range from the irradiation position as the irradiation timing for the pixel P. According to this embodiment, it is possible to cause the pixel P whose opening angle of the light beam and the virtual image opening angle are close to each other to emit light, and not cause the pixel P whose opening angle of the light beam and the virtual image opening angle are far from each other to emit light, so that the convergence accommodation contradiction can be appropriately reduced.

[0042] Moreover, the drive control unit 52 uniformly changes the positions in the Z direction of the focal points of the multiple lenses 32 of the microlens array 30. According to the present embodiment, the aperture angle of the light beam is adjusted by the lenses 32, so that a stereoscopic image can be appropriately provided to the user U.

[0043] Moreover, the display device 1 is a head-mounted display, and therefore, a stereoscopic image can be appropriately provided by the head-mounted display.

[0044] (Other examples) In the above description, the display unit 10 is a self-luminous display, but is not limited thereto. For example, as another example of the first embodiment, the display unit 10 may be configured to include a display panel including a plurality of pixels and a light source unit that irradiates light onto the display panel. In this case, for example, the display panel may be a liquid crystal display panel having a plurality of pixel electrodes arranged in a matrix and a liquid crystal layer filled with liquid crystal elements. In addition, the light source unit may be, for example, a backlight provided on the back surface of the display panel or a side light provided on the side surface of the display panel. In such a configuration, the light source unit irradiates light uniformly onto all pixels of the display panel, in other words, does not control the irradiation of light individually for each pixel. However, the light source unit is not limited to one that irradiates light uniformly onto all pixels of the display panel, and may be, for example, a so-called local dimming type that can adjust the irradiation of light for each of a plurality of pixels, for example, that can divide the entire screen into several sections and adjust the intensity of light for each section.

[0045] In this example, since light emission control cannot be performed for each pixel, it is preferable to provide a sensor (gaze detection unit) that detects the direction in which the user U's eyes EY are facing, that is, the gaze of the user U. Then, the timing setting unit 54 detects the gaze position of the user U based on the gaze detection result of the user U by the gaze detection unit. The gaze position refers to the position on the display surface 10A of the image on which the user U is gazing among the stereoscopic image, in other words, the position on which the user U is gazing among the entire area of ​​the display surface 10A. The timing setting unit 54 sets the irradiation timing based on the detection result of the gaze of the user U, the depth information of the stereoscopic image, and the position of the focal point of the microlens array 30 in the Z direction. Specifically, the timing setting unit 54 sets the irradiation position based on the depth information at the gaze position (the position in the depth direction of the gaze position). The timing setting unit 54 sets the Z-direction position of the focal point of the microlens array 30 when the opening angle of the image light L matches the opening angle of the light beam (virtual image opening angle) when light is irradiated to the eye EY from the depth direction position (the position of the virtual image that the user U is gazing at) at the gaze position as the irradiation position. Then, the irradiation control unit 56 causes the light source unit to irradiate light at the irradiation timing when the position of the focal point reaches within a predetermined distance range from this irradiation position, thereby irradiating the image light L from all pixels. In this example, since the virtual image opening angle at the gaze position where the user U is gazing and the opening angle of the image light L match, the convergence accommodation contradiction can be suppressed. Note that the virtual image opening angle and the opening angle of the image light L do not match at a location away from the gaze position, but since it is outside the range of the area where the user U is gazing, the impact on the user U is small.

[0046] Second embodiment Next, a second embodiment will be described. The second embodiment differs from the first embodiment in that the focal position of each lens of the microlens array can be adjusted individually. Explanations of the configurations of the second embodiment and the first embodiment will be omitted.

[0047] FIG. 9 is a schematic diagram of each component of a display device according to the second embodiment. As shown in FIG. 9, a display device 1a according to the second embodiment has a display unit 10, an eyepiece lens 20, a microlens array 30a, and a control device 40a. The display unit 10 in the second embodiment will be described below as a self-luminous display similar to the first embodiment, but may be a display including a light source unit and a display panel described in another example of the first embodiment. However, in the second embodiment, even if the display includes a light source unit and a display panel (i.e., even if the light emission cannot be controlled for each pixel), it is not necessary to detect the line of sight of the user U.

[0048] The microlens array 30a differs from the first embodiment in that the focal points of the lenses 32a can be changed individually. That is, the microlens array 30a of the second embodiment can be said to be an active matrix type variable focus microlens array. The microlens array 30a receives a control signal for controlling the microlens array 30a from the control device 40a.

[0049] FIG. 10 is a schematic block diagram of a control device according to the second embodiment. As shown in FIG. 10, the control unit 44 of the control device 40a according to the second embodiment includes an image information acquisition unit 50, a drive control unit 52a, a focal position setting unit 54a, and an irradiation control unit 56a. The control unit 44 realizes the image information acquisition unit 50, the drive control unit 52a, the focal position setting unit 54a, and the irradiation control unit 56a by reading and executing a program (software) from the storage unit 42, and executes the processes. The control unit 44 may execute these processes by one CPU, or may include multiple CPUs and execute the processes by the multiple CPUs. At least one of the image information acquisition unit 50, the drive control unit 52a, the focal position setting unit 54a, and the irradiation control unit 56a may be realized by a hardware circuit.

[0050] In the second embodiment, the irradiation control unit 56a, based on image data, irradiates each pixel P with image light L. In the second embodiment, each pixel P may be irradiated with image light L according to image data without controlling the irradiation timing of the image light L according to the focal position of the microlens array.

[0051] In the second embodiment, the focal position setting unit 54a acquires depth information for each pixel P. That is, the focal position setting unit 54a acquires information on the position in the depth direction (Z direction) for each pixel P. Then, based on the depth information of the pixel P, the focal position setting unit 54a sets a set focal position, which is the focal position of the lens 32a facing the pixel P when the image light L is irradiated onto the pixel P. Note that the lens 32a facing the pixel P can be said to be the lens 32a into which the image light L from the pixel P is incident, and will hereinafter be referred to as an opposing lens as appropriate.

[0052] The focal position setting unit 54a sets the focal position of the opposing lens in the Z direction when the opening angle of the actual image light L irradiated from the pixel P through the opposing lens to the eye EY matches the opening angle of the light flux (virtual image opening angle) when light is irradiated to the eye EY from the depth direction position of the stereoscopic image portion displayed by the pixel P (the position of the virtual image formed by the pixel P) as the set focal position of the opposing lens. However, the set focal position is not limited to the position where the virtual image opening angle and the opening angle of the actual image light L strictly match. For example, the set focal position may be the focal position of the opposing lens when the difference between the virtual image opening angle and the opening angle of the actual image light L is within a predetermined range. The predetermined range here may be set arbitrarily, but it is preferable to set it to a value that reduces the difference between the virtual image opening angle and the opening angle of the image light L in order to reduce the convergence accommodation contradiction. The focal position setting unit 54a may also use a value quantized in the depth direction as the position in the depth direction of the pixel P used to set the set focal position. That is, for example, the depth direction is divided into a plurality of numerical ranges, and a predetermined value within each numerical range is set as a reference position for each numerical range. Then, the focal position setting unit 54a extracts a numerical range that includes the position in the depth direction of the pixel P acquired by the image information acquisition unit 50, and treats the reference position for that numerical range as the position in the depth direction of the pixel P used to set the set focal position.

[0053] The focal position setting unit 54a sets a set focal position for each opposing lens (for each lens 32a). However, the set focal position is not limited to being different for each lens 32a, and for example, the set focal positions of a group of lenses 32a opposing a group of pixels P constituting one image (for example, a group of pixels P displaying an image of a house in FIG. 5) may be set to be the same.

[0054] The drive control unit 52a moves the Z-direction position of the focal point of the lens 32a to the set focal position. The drive control unit 52a individually controls the Z-direction position of the focal point of each lens 32a to position the focal point of each lens 32a at the set focal position set for each lens 32a. In the second embodiment, by setting the focal position for each lens 32a, it is possible to always match the virtual image opening angle and the opening angle of the image light L over the entire display surface 10A.

[0055] Next, the process flow of the control device 40a of the second embodiment will be described. FIG. 11 is a flowchart for explaining the process flow of the control device according to the second embodiment. The control device 40a sets a set focal position for each lens 32a from the depth information for each pixel P by the focal position setting unit 54a (step S20). The control device 40 moves the focal position of each lens 32a to the set focal position set for each lens 32a by the drive control unit 52a (step S22). In addition, in accordance with this, the control device 40 irradiates each pixel P with image light L based on the image data. As a result, the focal position is positioned at the set focal position set for each lens 32a. If the process is not to be ended (step S24; No), the process returns to step S20 and continues, and if the process is to be ended (step S24; Yes), the process is ended.

[0056] As described above, the display device 1a according to the second embodiment provides a stereoscopic image to the user U, and includes the display unit 10, the microlens array 30a, the focal position setting unit 54a, and the drive control unit 52a. The display unit 10 includes a plurality of pixels, and provides a stereoscopic image to the user U by making the image light L reach the user U. The microlens array 30a is provided closer to the user U than the display unit 10 in the optical axis direction of the image light L, has a plurality of lenses 32a, and can change the focal points of the lenses 32a individually. The focal position setting unit 54a sets a set focal position, which is the focal position in the optical axis direction of the image light L, for each lens 32a, based on depth information indicating the position in the depth direction of the stereoscopic image. The drive control unit 52a moves the focal position of the lens 32a to the set focal position.

[0057] Here, when a stereoscopic image is displayed, it is required to provide the user with an appropriate stereoscopic image. In contrast, in this embodiment, the focal position of the lens 32a is moved to a set focal position set based on the position in the depth direction of the stereoscopic image. Therefore, according to this embodiment, it is possible to make the image light L reach the user U at an appropriate light beam opening angle based on the position in the depth direction of the stereoscopic image, and the stereoscopic image can be appropriately provided to the user U. Furthermore, as described above, when a stereoscopic image is displayed, a convergence accommodation conflict may occur. In contrast, in this embodiment, the convergence accommodation conflict can be reduced by appropriately adjusting the opening angle of the light beam of the image light L. Furthermore, in the second embodiment, by setting the focal position individually for each lens 32a, it is possible to always match the virtual image opening angle and the opening angle of the image light L over the entire area of ​​the display surface 10A.

[0058] Further, the focal position setting unit 54a sets, as a set focal position, the position of the focal point of the lens 32a in the optical axis direction of the image light L such that the difference between the opening angle of the light beam of the image light L passing through the lens 32a and the opening angle of the light beam (virtual image opening angle) when the image light L is irradiated toward the user U from a position in the depth direction in the portion of the stereoscopic image displayed by the image light L falls within a predetermined range. According to this embodiment, the convergence accommodation conflict can be appropriately reduced by reducing the difference between the opening angle of the light beam and the virtual image opening angle.

[0059] As described in other examples, the display unit 10 of the second embodiment may be configured to include a display panel including a plurality of pixels and a light source unit that irradiates light onto the display panel, or may be configured to include a plurality of self-emitting pixels P. In either case, a stereoscopic image can be appropriately provided to the user U.

[0060] (Modification) Next, a modified example will be described. The display device 1b according to the modified example differs from the second embodiment in that the display is enlarged by a concave mirror 20C instead of the eyepiece lens 20. The description of the configuration of the modified example and the second embodiment will be omitted.

[0061] FIG. 12 is a schematic diagram of a display device according to a modified example. As shown in FIG. 12, the display device 1b according to the modified example does not have an eyepiece lens 20, and a half mirror 20B and a concave mirror 20C are provided closer to the eye EY of the user U than the display unit 10 in the optical axis direction of the image light L. The half mirror 20B and the concave mirror 20C can also be considered as optical elements. In the modified example, the image light L emitted from the display unit 10 is reflected by the half mirror 20B and enters the concave mirror 20C. The image light L that enters the concave mirror 20C becomes almost parallel light with a slight spread angle at the concave mirror 20C, and passes through the half mirror 20B to enter the eye EY of the user U.

[0062] In the modified example, the focal position of each lens 32a of the microlens array 30a is individually controlled as in the second embodiment. Therefore, even with the configuration of the modified example, a stereoscopic image can be appropriately provided to the user U as in the second embodiment, and the opening angle of the light flux of the image light L can be appropriately adjusted to reduce the convergence accommodation conflict.

[0063] The modified examples are also applicable to the first embodiment. The configuration of the display device may be other than those of the embodiments and the modified examples shown in FIG.

[0064] Although the embodiments of the present invention have been described above, the embodiments are not limited to the contents of these embodiments. The above-described components include those that can be easily imagined by a person skilled in the art, those that are substantially the same, and those that are within the so-called equivalent range. Furthermore, the above-described components can be appropriately combined, and the configurations of each embodiment can also be combined. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the gist of the above-described embodiments. [Explanation of symbols]

[0065] 1 Display device 10 Display 20 Eyepiece 30, 30a Microlens array 32, 32a Lenses 40, 40a Control device 50 Image information acquisition unit 52, 52a Drive control section 54 Timing setting section 54a Focus position setting section 56, 56a Irradiation control unit L Image light P pixels U User

Claims

1. A display unit including a plurality of pixels and configured to provide an image to a user by emitting light from the pixels; a microlens array that is provided closer to the user than the display unit and has a variable focus; A drive control unit that changes the focal position of the microlens array at a predetermined period; a timing setting unit that sets the light irradiation timing for each pixel based on a focal position of the microlens array; Including, The timing setting unit sets the irradiation timing based on depth information indicating a position in a depth direction of the image, the timing setting unit acquires information on an irradiation position, which is a focal position of the microlens array corresponding to an angle of an opening of a light beam from the pixel when the light is irradiated from a position in a depth direction in a portion of the image displayed by the pixel toward the user, and sets a timing when the focal position of the microlens array is within a predetermined distance range from the irradiation position as the irradiation timing for the pixel. Display device.

2. The display device according to claim 1 , wherein the drive control unit uniformly changes focal positions of a plurality of lenses included in the microlens array.

3. a drive control step of changing the focal position of a microlens array having a variable focus, the microlens array being disposed on the user side of a display unit including a plurality of pixels and emitting light from the pixels, at a predetermined period; a timing setting step of setting the light irradiation timing for each pixel based on a focal position of the microlens array; Including, In the timing setting step, the irradiation timing is set based on depth information indicating a position in a depth direction of the image; In the timing setting step, information on an irradiation position, which is a focal position of the microlens array corresponding to an divergence angle of a light beam from the pixel when the light is irradiated toward the user from a depth direction position in the portion of the image displayed by the pixel, is acquired, and a timing when the focal position of the microlens array is within a predetermined distance range from the irradiation position is set as the irradiation timing for the pixel. Display control method.

4. a drive control step of changing the focal position of a microlens array having a variable focus, the microlens array being disposed on the user side of a display unit including a plurality of pixels and emitting light from the pixels, at a predetermined period; a timing setting step of setting the light irradiation timing for each pixel based on a focal position of the microlens array; The computer executes the following: In the timing setting step, the irradiation timing is set based on depth information indicating a position in a depth direction of the image; In the timing setting step, information on an irradiation position, which is a focal position of the microlens array corresponding to an divergence angle of a light beam from the pixel when the light is irradiated toward the user from a depth direction position in the portion of the image displayed by the pixel, is acquired, and a timing when the focal position of the microlens array is within a predetermined distance range from the irradiation position is set as the irradiation timing for the pixel. Display control program.

Images