Display device, display control method, and display control program

Abstract

To appropriately provide images to a user.SOLUTION: A display device 1a provides a stereoscopic image to a user and includes: a display unit 10 that includes a plurality of pixels P and provides an image to the user by radiating light; a microlens array 30a that is provided closer to the user side than the display unit 10, has a plurality of lenses 32a, and is capable of individually changing the focus of the lenses 32a; and a focus position setting unit that sets a focus position for each lens 32a based on depth information that indicates a position of the image in a depth direction.SELECTED DRAWING: Figure 9

Description

[Technical field]

[0001] The present invention relates to a display device, a display control method, and a display control program. [Background technology]

[0002] By presenting images with different parallax to the right and left eyes of the user, a stereoscopic image is created by utilizing the difference in convergence. There is known a display device that displays a user's head. For example, Patent Document 1 describes a so-called head-mounted display (HMD). A head-mounted display having a microlens array disposed between the display and the optical system. The play is described. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2019 / 044501 Summary of the Invention [Problem to be solved by the invention]

[0004] Here, in a display device that displays a stereoscopic image, it is necessary to provide the image appropriately to the user. What is required is...

[0005] In view of the above problems, the present invention relates to a display device and a display control method capable of providing an appropriate image to a user. and a display control program. [Means for solving the problem]

[0006] The display device according to one aspect of the present invention includes a plurality of pixels, and is configured to irradiate the pixels with light to display the user's image. a display unit for providing an image to a user; and a plurality of lenses provided on the user side relative to the display unit. a microlens array having a lens focus that can be changed individually; A focus position is set for each of the lenses based on depth information indicating a position in a direction of the focus. A position setting unit.

[0007] According to one aspect of the present invention, a display control method includes: A microlens array is provided on the user side and is capable of individually changing the focal points of multiple lenses. The position of the point is set for each lens based on depth information indicating the position in the depth direction of the image. The method includes a focus position setting step for determining a focus position.

[0008] A display control program according to an aspect of the present invention includes: A microlens array is provided on the user side of the camera and is capable of individually changing the focal points of multiple lenses. The position of the focal point of the lens is determined based on depth information indicating the position of the focal point in the depth direction of the image. The computer executes a focus position setting step for setting the focus position for each image. Effect of the Invention

[0009] According to the present invention, an image can be appropriately provided to a user. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic diagram for explaining the convergence accommodation conflict. [Diagram 2] FIG. 2 is a schematic diagram of the display device according to the first embodiment. [Diagram 3] FIG. 3 is a schematic diagram of each component of the display device according to the first embodiment. [Figure 4] FIG. 4 is a schematic block diagram of the control device according to the first embodiment. [Diagram 5] FIG. 5 is a schematic diagram for explaining the setting of the irradiation timing. [Figure 6] FIG. 6 is a schematic diagram for explaining the setting of the irradiation timing. [Figure 7] FIG. 7 is a schematic diagram for explaining the setting of the irradiation timing. [Figure 8] FIG. 8 is a flowchart illustrating a process flow of the control device according to the present embodiment. [Figure 9] FIG. 9 is a schematic diagram of each component of the display device according to the second embodiment. [Figure 10] FIG. 10 is a schematic block diagram of a control device according to the second embodiment. [Figure 11] FIG. 11 is a flowchart illustrating a process flow of the control device according to the second embodiment. [Figure 12] FIG. 12 is a schematic diagram of a display device according to a modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. The present invention is not limited to the embodiments.

[0012] (Convergence regulation conflict) FIG. 1 is a schematic diagram for explaining the convergence accommodation conflict. A display device for displaying a stereoscopic image is By presenting images with different parallax to the right and left eyes of the user, a stereoscopic image is created by utilizing the difference in convergence. When displaying a stereoscopic image, the display surface on which the image is actually displayed is the user's The focal point of the eyes is the convergence point, and the point where the lines of sight of the left and right eyes intersect is the convergence point. As shown in the example, in a stereoscopic image, the focal point in the Z direction, which is the depth direction of the stereoscopic image, The focal position PO1 and the convergence position PO2 may be misaligned. If the position of O2 is misaligned, a so-called convergence accommodation conflict occurs, causing eye strain and so-called 3D sickness. This causes the above problems. Therefore, it is necessary to suppress the convergence accommodation conflict. In (A) of FIG. 1, the focal position PO1 is located closer to the user's eye EY than the convergence position PO2. FIG. 1B shows an example in which the convergence position PO2 is closer to the user's eye EY than the focal position PO1. do.

[0013] (First embodiment) (Overall configuration of the display device) FIG. 2 is a schematic diagram of the display device according to the first embodiment. As shown in FIG. 2, the display device 1 displays a stereoscopic image of a user U. It is a so-called HMD (Head Mounted Display) that is worn on the head. For example, the display device 1 has a display unit 10 attached to a position facing the eye EY of a user U. The display device 1 then provides the content to the user U by displaying an image on the display unit 10 . The configuration of the display device 1 shown in FIG. 2 is an example. For example, the display device 1 may be It may also be equipped with an audio output unit (speaker) that is attached to the ear.

[0014] Since the display device 1 is attached to the user U in this manner, the position of the display device 1 relative to the eye EY of the user U is The display device 1 is not limited to being an HMD worn by the user U, but may be a device mounted on a facility. In such a case, the display device 1 may be, for example, a display device fixed to the The position of the user U relative to the eye EY is fixed, for example, the position of the user U relative to the seat. It is preferable that

[0015] FIG. 3 is a schematic diagram of each component of the display device according to the first embodiment. As shown in FIG. The display device 1 includes a display unit 10, an eyepiece lens 20, a microlens array 30, and a control device. 40 and

[0016] (Display) The display unit 10 is a device that displays a stereoscopic image. The display unit 10 has a plurality of light-emitting pixels P (display elements). Since the pixel P is self-luminous, the display unit 10 can control the emission (light irradiation) of each pixel P individually. Each pixel P of the display unit 10 is, for example, an organic light emitting diode (OLED). It may be a luminescent diode (NIL) or an inorganic light emitting diode (IND). The display unit 10 displays an image. The surface on which the image is displayed is referred to as a display surface 10A. Hereinafter, the direction from the display surface 10A to the eye EY of the user U will be referred to as The direction of the image is defined as direction Z1, and the direction opposite to direction Z1, that is, the direction from the eye EY of the user U to the display surface 10 The direction toward A is called direction Z2. When there is no need to distinguish between directions Z1 and Z2, it is written as direction Z. In FIG. 3, the surface of the display unit 10 on the eye EY side of the user U is designated as a display surface 10A. However, the display surface 10A is not limited to being a surface on the eye EY side of the user U. The display unit 10 may be located inside the surface on the EY side. The pixel P of the display unit 10 receives a control signal for controlling the pixel P from the pixel 10.

[0017] The display unit 10 transmits image light L, which is light irradiated (emitted) from each pixel P, to the eye EY of the user U. By making the left and right eyes reach the left and right eyes, a stereoscopic image is provided to the user U. The light emission of each pixel P is controlled so that an image for the right eye and an image for the left eye are provided. Among the image light L from the pixels P corresponding to the image for the left eye, the image light L from the pixels P corresponding to the image for the left eye enters the left eye of the user U. The image light L from the pixel P corresponding to the image for the right eye is incident on the left eye of the user U, A user U is provided with a stereoscopic image.

[0018] (Eyepiece) The eyepiece lens 20 is provided on the Z1 direction side of the display unit 10. The eyepiece lens 20 receives light (image The eyepiece 20 is an optical element that transmits the light (image light). It is an optical element (lens) located on the eye EY side of the user U. Image light emitted from the display unit 10 The light L passes through the eyepiece lens 20 and enters the eye EY of the user U. In this embodiment, The optical axis of the optical path of the image light L from the eyepiece 20 (eyepiece) to the eye EY of the user U The direction can also be called direction Z.

[0019] In the example of FIG. 3, the optical elements on the Z1 side of the display unit 10 are the eyepiece lens 20 and the mirror lens. Although only the microlens array 30 is shown, the present invention is not limited to the microlens array 30 and other optical elements may be used. may be provided.

[0020] (Microlens Array) The microlens array 30 is disposed closer to the user than the display unit 10 in the optical axis direction of the image light L. Furthermore, the microlens array 30 is provided on the U side in the optical axis direction of the image light L. The microlens array is disposed between the display unit 10 and the eyepiece lens 20 in the viewing direction. The lens 30 is an optical element in which a plurality of lenses 32 are arranged in a matrix on a plane parallel to the display surface 10A. In this embodiment, the pitch of the lenses 32 in the microlens array 30 is That is, the distance between the centers of adjacent lenses 32 is, for example, the pitch of the pixels P of the display unit 10 ( The distance between the centers of adjacent pixels P is approximately equal to the distance between the centers of the adjacent pixels P. The lens 32 is provided at a position facing each pixel P of the display unit 10 in the optical axis direction of the image light L. The microlens array 30 is controlled by a control device 40, which will be described later. The optical fiber 10 receives a control signal that controls the optical fiber array 30.

[0021] The microlens array 30 has a Z-direction (optical axis direction of the image light L) of each lens 32. A so-called variable focus microscope, which can change the focal position (the distance from the lens 32 to the focal point). In the first embodiment, the microlens array 30 has each lens. The focal position of the lens 32 is uniformly changed, and each lens is rotated at the same timing. The focal position of the microlens 32 in the Z direction is the same. For example, it may be constructed by combining a liquid crystal and a Fresnel lens. For example, each lens 32 of the microlens array 30 may be a liquid crystal variable focus lens. Volumetric 3D display using a 3D image sensor" (Shiro Suyama, Tokushima University, Optics, Vol. 40, No. 12 (2011)) The variable focus lens disclosed in the document may be used.

[0022] (Control device) The control device 40 is a device that controls each part of the display device 1. FIG. 1 is a schematic block diagram of a control device 40. In this embodiment, the control device 40 is a computer. The memory unit 42 stores the calculation contents and program contents of the control unit 44. RAM (Random Access Memory) is a memory that stores various information such as programs. Access Memory (ROM) and Read Only Memory (ROM) Among storage devices and external storage devices such as HDD (Hard Disk Drive), The program for the control unit 44 stored in the storage unit 42 is may be stored in a readable recording medium.

[0023] The control unit 44 is a computing device, for example a CPU (Central Processing The control unit 44 includes an image information acquisition unit 50 and a drive control unit. The control unit 44 includes a memory unit 42, a timing setting unit 54, and an irradiation control unit 56. 2 and executes a program (software) to obtain image information 50. A drive control unit 52, a timing setting unit 54, and an irradiation control unit 56 are realized, and their processing is The control unit 44 may execute these processes by a single CPU. Alternatively, multiple CPUs may be provided and the processing may be executed by the multiple CPUs. At least an information acquisition unit 50, a drive control unit 52, a timing setting unit 54, and an irradiation control unit 56 The other one may be realized by a hardware circuit.

[0024] (Image information acquisition section) The image information acquisition unit 50 acquires image data of the stereoscopic image displayed by the display unit 10. That is, the image information acquisition unit 50 acquires image data of an image for the left eye and image data of an image for the right eye. The image information acquisition unit 50 also acquires depth information indicating a position in the depth direction of the stereoscopic image. The position in the depth direction of the stereoscopic image is the position where the image is displayed on the display surface 10A. The depth direction refers to the position in the depth direction of the virtual image that is visually recognized by the user U when the In other words, the direction is perpendicular to the display surface 10A of the display unit 10, and in this embodiment, the direction is the Z direction. The depth information is associated with the image data. Moreover, a stereoscopic image is For each image in a frame, a position in the depth direction is set. In other words, Therefore, a position in the depth direction is set for each position on the display surface 10A. The image information acquisition unit 50 acquires depth information for each position on the display surface 10A for the stereoscopic image. In addition, the position in the depth direction is set for each pixel P in the stereoscopic image. However, for multiple pixels P that make up one image, the positions in the depth direction are the same. The image information acquisition unit 50 may acquire image data and The image data and the depth information may be acquired from the image data and the depth information stored in advance in the storage unit 42. Alternatively, image data and depth information may be received via a communication unit (not shown). The image information acquisition unit 50 may also calculate the position in the depth direction based on the image data. By doing so, depth information may be acquired.

[0025] (Drive control unit) The drive control unit 52 controls the microlens array 30 to The drive control unit 52 moves the focal position of each of the lenses 32 in the Z direction. By controlling the application of voltage to the liquid crystal elements included in the lens array 30, the lens 3 The drive control unit 52 moves the position of the focal point of the lens 32 in the Z direction. Reciprocating movement (vibration) in the Z direction, which moves a specified distance in one direction and then moves a specified distance in the Z2 direction. The focus position of the lens 32 is moved repeatedly. The position of the focal point 32 is moved in the Z direction at a predetermined period. In other words, the drive control The unit 52 causes the focal point of the lens 32 to move back and forth in the Z direction at a predetermined cycle. Now, what is the period of reciprocating movement in the Z direction (the period until the display unit 10 returns to its original position in the Z direction)? The period of time is constant, but is not limited to being constant, and the period may be changed.

[0026] (Timing setting unit and irradiation control unit) The timing setting unit 54 sets the irradiation timing of the image light L for each pixel P of the display unit 10. The illumination control unit 56 controls the pixel P of the display unit 10 based on the image data. The image light L is irradiated. The irradiation control unit 56 irradiates the image light L for each pixel P set by the timing setting unit 54. At the irradiation timing, the image light L is irradiated to each pixel P. The image light L is irradiated to a pixel P of the display unit 10 at an irradiation timing set for the pixel P. The timing setting unit 54 controls the micro-illumination in the Z direction (the optical axis direction of the image light L). The timing of irradiation is set based on the focal position of the lens array 30 (lens 32). In other words, the timing setting unit 54 determines the depth information of the stereoscopic image and the microphone position in the Z direction. The timing of irradiation is set based on the focal position of the lens array 30. The timing setting unit 54 sets the irradiation timing for each pixel P. For example, a group of pixels P constituting one image (e.g., the house in FIG. 5) may be different. The timing of irradiation of a group of pixels P that display an image may be set to the same. The settings of the timing setting unit 54 will now be described in more detail.

[0027] 5 to 7 are schematic diagrams for explaining the setting of the irradiation timing. For example, in FIG. As shown in the figure, the image light L emitted from the pixel P of the display unit 10 is a light beam having a predetermined opening angle. Then, the light beam enters the eye EY of the user U. In this case, the user U adjusts the angle of the light beam to match the opening angle. The lens of the eye adjusts to focus on the retina by unconsciously changing its thickness. Convergence accommodation conflict focuses on the degree of convergence of the left and right eyes (the degree to which the left and right eyes are apart) and the opening angle of the light beam. In contrast, the display device of this embodiment 1 is the opening angle when it is assumed that light is emitted from the virtual image (convergence position) and enters the eye EY. A certain virtual image opening angle (angle θ1A in the example of FIG. 5) and the image light L actually emitted from pixel P The image light L is emitted so that the difference between the opening angle of the image light L when it is incident on the eye EY is small. Here, the image light L is divided into a light beam as it passes through the lenses 32 of the microlens array 30. The opening angle changes (refracts), and the degree of change in the opening angle of the light beam is Therefore, the timing setting unit 54 determines whether the position of the focal point of the lens 32 is equal to the virtual image opening angle. The timing at which the difference between the angle of the image light L and the angle of the image light L becomes small is called the irradiation timing. Set as.

[0028] More specifically, the timing setting unit 54 acquires depth information for each pixel P. That is, The timing setting unit 54 acquires information on the position of each pixel P in the depth direction (Z direction). Then, the timing setting unit 54 sets the pixel P to the pixel P based on the depth information of the pixel P. The irradiation position, which is the focal position of the lens 32 when irradiation of the image light L starts, is set. The pixel setting unit 54 determines the depth direction position of the portion of the stereoscopic image displayed by the pixel P ( The luminous flux opening when light is irradiated to the eye EY from the position of the virtual image formed by the pixel P of The image light that actually enters the eye EY from the pixel P through the lens 32 at the virtual image opening angle The position of the focal point of the lens 32 in the Z direction when the opening angles of L and P coincide is The timing setting unit 54 sets the irradiation position at the time when the distance from the irradiation position is The timing when the focus of the lens 32 reaches a position within a predetermined distance range is determined for that pixel P. The timing setting unit 54 sets the irradiation position for each pixel P. The timing setting unit 54 sets the irradiation timing for each pixel P. The irradiation position and timing are set for each pixel P, but the irradiation position and timing are different for each pixel P. For example, a group of pixels P constituting an image (e.g., the image of a house in FIG. 5) may be The irradiation positions and irradiation timings of the pixels (e.g., a group of pixels P displaying a pixel) may be set to be the same.

[0029] In this embodiment, the timing setting unit 54 adjusts the focal position of the lens 32 in the Z direction. The information is sequentially acquired, and the position of the focal point of the lens 32 in the Z direction is a predetermined distance from the irradiation position. When the target object approaches the target position, it is determined that the irradiation timing has arrived. The information on the focal position of the lens 32 in the image sensor 1 may be obtained by any method. For example, When the focal point of the lens 32 is reciprocated in the Z direction at a predetermined period, the focal point of the lens 32 at each time is The position (predicted position) of the point in the Z direction can be grasped. Information on the position of the focal point of the lens 32 in the Z direction may be obtained from the time information. The timing setting unit 54 receives information on the predicted focal position of the lens 32 at each time and information on the irradiation position. Based on the information, the time when the focal point of the lens 32 reaches a predetermined distance from the irradiation position is determined as the irradiation timing. When the current time reaches the irradiation timing, the focus of the lens 32 is set to the irradiation position. When the pixel P reaches the predetermined distance, the image light L may be irradiated onto the pixel P. can be set arbitrarily, but in order to reduce the convergence accommodation conflict, the virtual image opening angle and the opening angle of the image light L should be set It is preferable to set the distance so that the difference between the target angle and the target angle is small. The setting unit 54 sets the quantized value in the depth direction as the depth direction of the pixel P used to set the irradiation position. For example, the depth direction may be divided into multiple ranges of values. For each numerical range, a predetermined value within the numerical range is set as the reference position. The timing setting unit 54 then sets the depth of the pixel P acquired by the image information acquisition unit 50. A numerical range that includes the position in the direction of the reference is extracted, and the reference position for the numerical range is calculated by It is treated as the position in the depth direction of pixel P used to set the shooting position.

[0030] In this manner, the timing setting unit 54 sets the irradiation timing for each pixel P. The timing setting unit 54 sets the timing after the irradiation timing as the irradiation stop timing. When the irradiation control unit 56 determines that the irradiation timing for a certain pixel P has arrived, When the pixel P is turned off, the irradiation control unit 56 starts irradiating the pixel P with the image light L. The pixel P is irradiated with the image light L from the start of irradiation to the irradiation stop timing. When the timing is reached, the irradiation of the image light L to the pixel P is stopped. The timing may be set arbitrarily. For example, the timing may be set to stop irradiation after a predetermined time from the irradiation timing. Alternatively, the focal position of the lens 32 and the irradiation position may be set as The timing when the distance between the target object and the target falls outside a predetermined distance range may be set as the timing to stop irradiation. stomach.

[0031] In this way, the display device 1 irradiates the pixel P with the image light L when the irradiation timing is reached. When the irradiation stop timing is reached, the irradiation of the image light L is stopped. The light is incident on the eye EY of the user U during the period from the start of irradiation to the end of irradiation. The divergence angle of the incident image light L is close to the divergence angle of the virtual image formed by the pixel P. The focal position of the lens 32 is set to the Z direction. Since the laser reciprocates in the opposite direction, the distance between the laser and the irradiation position is within a specified distance range and outside the specified distance range. The control device 40 controls the distance between the focal position of the lens 32 and the irradiation position. Whenever the distance falls within a predetermined range, that is, whenever the irradiation timing is reached, the image light L Therefore, the user U can visually recognize a stereoscopic image as a moving image. The focal position of the lens 32 moves back and forth in the Z direction, so that the focal position of the lens 32 moves back and forth in one period. There are two timings when the distance becomes the predetermined distance. It is desirable to set the frequency of the movement to at least half the frame rate of the stereoscopic image. The frequency (period) of the reciprocating movement of the focal position of the lens 32 may be set arbitrarily.

[0032] An example of the setting of the above-described irradiation timing will be described with reference to Figs. 5 to 7. The image of the house, the image of the car, and the image of the helicopter are displayed as a stereoscopic image. The image of the helicopter is located far from the user U's eye EY in the depth direction (Z direction). In other words, the virtual image P2 of the image of the car is smaller than the virtual image P1 of the image of the house. The virtual image P3 of the helicopter image is located on the one side in the Z2 direction from the virtual image P1 of the house image. Located to the side.

[0033] FIG. 5 shows an example in which a user U visually recognizes a virtual image P1 of an image of a house. In the example of FIG. Light is irradiated onto the eye EY from the virtual image P1 (the depth direction position of pixel P that constitutes the image of the house). The divergence angle of the light beam (the virtual image divergence angle) in this case is the angle θ1A. When the focal position of the lens 32 is at the first position, the image light L from the pixel P constituting the image of the house The opening angle of the light beam is assumed to be angle θ1A. In this case, the first position forms an image of a house. This is the irradiation position for pixel P, and the timing setting unit 54 determines whether the distance from the first position is a predetermined distance. The timing at which the focus of the lens 32 reaches a position within the distance range is determined by the image of the house. The irradiation control unit 56 sets the irradiation timing for the element P. Then, image light L is irradiated onto pixels P constituting the image of the house. The opening angle of the virtual image from the image P1 is close to the opening angle of the image light L actually incident on the eye EY of the user U. The image light L is deflected by the eyepiece lens 20, and the convergence accommodation conflict can be reduced. The light beam of the image light L is refracted and enters the eye EY. Therefore, the opening angle of the light beam of the image light L is the opening angle of the eyepiece lens 20. This refers to the divergence angle of the light beam of image light L after passing through the

[0034] FIG. 6 shows an example in which a virtual image P2 of a car is visually recognized by a user U. In the example of FIG. Light is irradiated onto the eye EY from the virtual image P2 (the depth direction position of pixel P that constitutes the image of the car). The divergence angle of the light beam (the virtual image divergence angle) in this case is the angle θ2A. When the focal position of the lens 32 is at the second position, the image light L from the pixel P constituting the image of the car The opening angle of the light beam is θ2A. Note that the virtual image P2 is located in the Z1 direction from the virtual image P1. 5, the angle θ2A is larger than the angle θ1A in FIG. 5, and when the first position is the focus, The degree of expansion of the opening angle of the light beam by the lens 32 is greater when the second position is the focal point than when the In this case, the second position constitutes the image of the car. The timing setting unit 54 sets the distance from the first position to the second position as a predetermined value. The timing at which the focus of the lens 32 reaches a position within the fixed distance range is determined as the time when the image of the car is formed. The irradiation control unit 56 sets the irradiation timing for the pixel P. When this happens, light is irradiated onto pixel P that constitutes the image of the car. This creates a virtual image of the car. The opening angle of the virtual image from P2 is close to the opening angle of the image light L that actually enters the eye EY of the user U. This makes it possible to reduce the convergence adjustment conflict.

[0035] FIG. 7 shows an example in which a user U visually recognizes a virtual image P3 of a helicopter. In this example, the distance from the virtual image P3 (the depth direction position of pixel P that constitutes the image of the helicopter) to the eye EY The divergence angle of the light beam (the virtual image divergence angle) when light is irradiated to the object is defined as angle θ3A. When the focal position of the lens 32 in the direction is at a third position, an image of the helicopter is formed. The opening angle of the light flux of the image light L from the pixel P is set to be angle θ3A. Since the virtual image P1 is viewed in the Z2 direction, the angle θ3A is smaller than the angle θ1A in FIG. The lens 32 is more effective when the third position is the focal point than when the first position is the focal point. In this case, the degree of expansion of the opening angle of the light beam becomes smaller (or the degree of contraction becomes larger). The three positions are the irradiation positions for the pixels P that compose the helicopter image, and the timing is set. The focus of the lens 32 reaches a position whose distance from the third position is within a predetermined distance range. The timing at which the image is shot is set as the irradiation timing for pixel P that constitutes the helicopter image. When the irradiation timing arrives, the irradiation control unit 56 controls the image forming unit 54 to irradiate the helicopter. The image light L is irradiated onto the element P. This causes the virtual image P3 of the helicopter image to open up. The angle becomes closer to the opening angle of the image light L that actually enters the eye EY of the user U, resulting in a convergence accommodation conflict. can be made smaller.

[0036] In addition, since the display device 1 sets the irradiation timing for each pixel P, In some cases, only pixel P in the image is lit. For example, in the timing of Figure 5, only the image of the house is displayed. At the timing shown in Fig. 6, only the image of the car is displayed, and at the timing shown in Fig. 7, the image of the helicopter is displayed. Only the image is displayed. However, the persistence of images created by multiple frames in succession may cause the user to The U recognizes that a house, a car, and a helicopter are in one image. Also, in one frame Even if the entire image (here, the house, the car, and the helicopter) is displayed within the display time, In this case, the drive control unit 52 adjusts the focal position of the lens 32 within the display time of one frame. This allows the display time of one frame to be Within this time, all focal positions within the reciprocating movement can be covered, and the entire image can be displayed. It becomes possible.

[0037] (Processing flow) Next, the process flow of the control device 40 described above will be described. 1 is a flowchart illustrating a process flow of the control device. 2, the focal position of the microlens array 30 is moved back and forth in the Z direction at a predetermined period. The control device 40 uses the timing setting unit 54 to set the depth information for each pixel P. The control device 40 sets the irradiation position of the microlens array 30 (step S10). The position of the focal point of the microlens array 30 in the Z direction is sequentially acquired, and the focal point of the microlens array 30 is the irradiation position It is determined for each pixel P whether it has reached within a predetermined distance from the microlens (step S12). When the focal point of the lens array 30 reaches within a predetermined distance from the irradiation position (step S12; Yes), That is, it is determined that the focal point of the microlens array 30 has reached within a predetermined distance from the irradiation position. If there is a pixel P to be irradiated, the timing setting unit 54 determines whether the pixel P is in the irradiation timing. When it is determined that the pixel P has been reached, the irradiation control unit 56 irradiates the pixel P with the image light L based on the image data. After that, the irradiation of the image light L is stopped at the irradiation stop timing (step S14). If the process is not to be ended (step S16; No), the process returns to step S10 and continues. On the other hand, if the focal point of the microlens array 30 does not reach within a predetermined distance from the irradiation position, In this case (step S12; No), that is, the focal point of the microlens array 30 is not at the irradiation position. If there is no pixel P that is determined to have reached within the predetermined distance from the position, the process returns to step S12. The light is directed to the pixel P until the focal point of the microlens array 30 reaches within a predetermined distance from the irradiation position. When the process is ended in step S16 (step S16; Yes), End processing.

[0038] (effect) As described above, the display device 1 according to the present embodiment provides a stereoscopic image to the user U. The display unit 10, the microlens array 30, the drive control unit 52, and the timing The display unit 10 includes a plurality of self-emitting pixels P. By allowing image light L from the pixel P to reach the user U, a stereoscopic image is provided to the user U. The microlens array 30 is positioned closer to the user than the display unit 10 in the optical axis direction of the image light L. The drive control unit 52 controls the microlens array 3 The position of the focus of the image light L is changed at a predetermined interval along the optical axis direction (Z direction in this embodiment). The timing setting unit 54 moves the microlens array 30 in the optical axis direction (the present embodiment) of the focal point. In the embodiment, the timing of irradiating the image light L is set for each pixel P based on the position in the Z direction. The illumination control unit 56 causes the pixel P to be illuminated with the image light L at the illumination timing.

[0039] When displaying a stereoscopic image, it is necessary to provide the stereoscopic image appropriately to the user. In contrast, in the present embodiment, the focal point of the microlens array 30 is adjusted to a The focal point is moved in the axial direction, and the timing of irradiation of the image light L is set based on the position of the focal point. Therefore, according to this embodiment, the user can see the image at an appropriate timing based on the position of the focal point in the optical axis direction. Therefore, it is possible to transmit the image light L to the user U, and a stereoscopic image can be appropriately provided to the user U. Furthermore, as mentioned above, when displaying a stereoscopic image, convergence accommodation conflict may occur. In contrast, in this embodiment, the focal point of the microlens array 30 is aligned along the optical axis. While moving the focus in the optical axis direction, the timing of irradiating the image light L is set based on the position of the focus in the optical axis direction. By doing so, the opening angle of the light beam of the image light L can be appropriately adjusted to reduce the convergence accommodation conflict. can be done.

[0040] The timing setting unit 54 also determines the position of the stereoscopic image in the depth direction (the Z direction in this embodiment). The irradiation timing is set based on the depth information indicating the position. The illumination timing is set using the image information, so the image light L is adjusted according to the displayed 3D image. This makes it possible to appropriately adjust the opening angle of the light beam, thereby reducing the convergence accommodation conflict. Cut.

[0041] The timing setting unit 54 also determines whether the opening angle of the light flux of the image light L from the pixel P is greater than or equal to the opening angle of the light flux of the pixel P. Image light L is projected from a position in the depth direction of the stereoscopic image displayed by A microlens array that corresponds to the divergence angle of the light beam (virtual image divergence angle) when irradiated with The information on the irradiation position, which is the position of the focus of the laser beam 30 in the optical axis direction, is obtained. The unit 54 detects when the focal point of the microlens array 30 is within a predetermined distance range from the irradiation position. The timing is set as the irradiation timing for the pixel P. , the pixel P whose light beam opening angle and virtual image opening angle are close to each other is caused to emit light, and the light beam opening angle and the virtual image opening angle are Since it is possible to prevent the pixel P from emitting light, the convergence accommodation conflict can be appropriately reduced. This can be done.

[0042] The drive control unit 52 also controls the focal points of the lenses 32 of the microlens array 30. In this embodiment, the lens 32 uniformly changes the position of the light beam in the Z direction. By adjusting the opening angle, a stereoscopic image can be appropriately provided to the user U.

[0043] The display device 1 is a head-mounted display. The display is capable of providing a stereoscopic image adequately.

[0044] (Other examples) In the above description, the display unit 10 is a self-luminous display. For example, as another example of the first embodiment, the display unit 10 may be a display panel including a plurality of pixels. The display panel may include a light source unit that irradiates light onto the display panel. The display panel is made up of a plurality of pixel electrodes arranged in a matrix and a liquid crystal layer filled with liquid crystal elements. The light source unit may be provided on the rear surface of the display panel. The light source may be a backlight provided on a side of the display panel, or a side light provided on a side of the display panel. In such a configuration, the light source unit uniformly irradiates light onto all pixels of the display panel. In other words, the light irradiation is not controlled individually for each pixel. The light source is not limited to one that uniformly irradiates all pixels of the display panel with light. For example, For example, the entire screen can be divided into several sections, and the light intensity can be adjusted for each section. The brightness may be adjusted by a so-called local dimming method.

[0045] In this example, since it is not possible to control the light emission for each pixel, the eye EY of the user U is directed It is preferable to provide a sensor (gaze detection unit) that detects the direction, i.e., the gaze of the user U. The timing setting unit 54 determines whether or not the line of sight of the user U is detected by the line of sight detection unit. Based on this, the gaze position at which the user U is gazing is detected. This refers to the position on the display surface 10A of the image that the user U is gazing at. The timing setting unit 54 determines the position of the user U in the entire area of ​​the camera 0A. The detection result of the line of sight of the user U, the depth information of the stereoscopic image, and the focal point of the microlens array 30 are The irradiation timing is set based on the position in the Z direction. The determining unit 54 determines the distance from the gaze position based on the depth information at the gaze position (the position in the depth direction of the gaze position). The timing setting unit 54 sets the irradiation position based on the depth direction position (uniform The divergence angle of the light beam when light is irradiated onto the eye EY from the position of the virtual image that the user U is gazing at (the virtual image) When the opening angle of the image light L coincides with the image opening angle, the Z The position in the direction is set as the irradiation position. Then, the irradiation control unit 56 sets the irradiation position When the focal position reaches within a predetermined distance from the target position, the light source unit is caused to irradiate light. In this example, the image light L is emitted from all pixels. Since the virtual image opening angle at the gaze position is the same as the opening angle of the image light L, the convergence accommodation contradiction is eliminated. In addition, the virtual image opening angle and the opening angle of the image light L match at a location away from the gaze position. However, since it is outside the range of the area that the user U is gazing at, the impact on the user U is small.

[0046] Second embodiment Next, a second embodiment will be described. In the second embodiment, a microlens array The difference from the first embodiment is that the focal positions of the lenses in A and B can be adjusted individually. In the second embodiment, the description of the configuration common to the first embodiment will be omitted.

[0047] FIG. 9 is a schematic diagram of each component of the display device according to the second embodiment. As shown in FIG. The display device 1a according to the second embodiment includes a display unit 10, an eyepiece lens 20, and a microlens array. The display unit 10 in the second embodiment includes a display 30a and a control device 40a. In the following, the display will be described as a self-luminous display similar to the first embodiment. The display may include the light source unit and the display panel described in the other embodiments. However, in the second embodiment, even if the display includes a light source unit and a display panel (i.e., a picture Even if light emission cannot be controlled for each element, there is no need to detect the line of sight of the user U.

[0048] The microlens array 30a has a feature that the focal points of the multiple lenses 32a can be changed individually. That is, the microlens array 30a of the second embodiment is different from that of the first embodiment. This can be said to be an active matrix type variable focus microlens array. The microlens array 30a receives a control signal from the control device 40a to control the microlens array 30a. Receive the number.

[0049] FIG. 10 is a schematic block diagram of a control device according to the second embodiment. As described above, the control unit 44 of the control device 40a according to the second embodiment controls the image information acquisition unit 50 and the driving The control unit 44 includes a control unit 52a, a focal position setting unit 54a, and an irradiation control unit 56a. The program (software) is read from the storage unit 42 and executed to obtain image information. The imaging unit 50, the drive control unit 52a, the focal position setting unit 54a, and the irradiation control unit 56a are realized. The control unit 44 executes these processes using one CPU. Alternatively, multiple CPUs may be provided and the processing may be executed by the multiple CPUs. In addition, the image information acquisition unit 50, the drive control unit 52a, the focal position setting unit 54a, and the irradiation control unit 5 At least one of steps 6a and 6b may be realized by a hardware circuit.

[0050] In the second embodiment, the irradiation control unit 56a applies an image to each pixel P based on image data. In the second embodiment, the light L is irradiated according to the focal position of the microlens array. The image light L is irradiated to each pixel P according to image data without controlling the irradiation timing of the image light L. It may be irradiated.

[0051] In the second embodiment, the focal position setting unit 54a acquires depth information for each pixel P. That is, the focal position setting unit 54a determines the position of each pixel P in the depth direction (Z direction). Then, the focal position setting unit 54a acquires the depth information of the pixel P. When the image light L is irradiated onto the pixel P, the focal position of the lens 32a facing the pixel P is A certain set focal position is set. Note that the lens 32a facing the pixel P is the lens The lens 32a is the lens onto which the image light L is incident, and will be referred to as the opposing lens hereinafter where appropriate.

[0052] The focal position setting unit 54a determines the depth direction position of the portion of the stereoscopic image displayed by the pixel P. The luminous flux when light is irradiated onto the eye EY from the position (the position of the virtual image formed by that pixel P) The actual light irradiated from pixel P through the opposing lens to the eye EY is The position of the focal point of the opposing lens in the Z direction when the opening angles of the image light L of the opposing lens coincide is called the focal point of the opposing lens in the Z direction. The focal position of the lens is set as the focal position of the virtual image. For example, the virtual image opening angle and the virtual image opening angle are not limited to being exactly the same as the set focal position. The focal position of the opposing lens when the difference between the opening angle of the actual image light L and the focal position of the opposing lens is within a predetermined range is set. The predetermined range may be set arbitrarily, but it is necessary to set it as a range that minimizes the convergence accommodation conflict. In order to achieve this, the difference between the virtual image opening angle and the opening angle of the image light L is set to a value that is small. It is preferable that the focal position setting unit 54a sets a quantized value in the depth direction. This may be used as the position in the depth direction of pixel P used to set the focal position. For example, the depth direction is divided into multiple numerical ranges, and for each numerical range, A predetermined value within the range is set as a reference position. Extract a numerical range including the position in the depth direction of pixel P acquired by image information acquisition unit 50. The reference position for the numerical range is determined in the depth direction of the pixel P used to set the set focus position. Treat it as a position in .

[0053] The focal position setting unit 54a sets a set focal position for each opposing lens (for each lens 32a). However, the focal position setting is not limited to different for each lens 32a. For example, Facing a group of constituent pixels P (such as a group of pixels P displaying an image of a house in FIG. 5) The set focus positions of the group of lenses 32a may be set to the same.

[0054] The drive control unit 52a moves the position of the focal point of the lens 32a in the Z direction to a set focal position. The drive control unit 52a controls the position of the focal point of each lens 32a in the Z direction individually. By this, the focus of each lens 32a is set for each lens 32a. In the second embodiment, the focal position is set for each lens 32a. By setting the angle, the virtual image opening angle and the opening angle of the image light L are always kept constant over the entire area of ​​the display surface 10A. It is possible to achieve this.

[0055] Next, a process flow of the control device 40a of the second embodiment will be described. 1 is a flowchart illustrating a process flow of the control device according to the embodiment. The point position setting unit 54a determines a set focal position for each lens 32a from the depth information for each pixel P. (Step S20). The control device 40 sets each of the The focal position of the lens 32a is set to a set focal position for each lens 32a. (Step S22). In addition, the control device 40 moves the image data Based on the above, each pixel P is irradiated with image light L. The focus position is set to the set focus position set for 2a. If the process is not to be ended (step If the process is to be ended (Step S24; No), the process returns to Step S20 and continues. If the answer is YES in step S24, the process ends.

[0056] As described above, the display device 1a according to the second embodiment provides a stereoscopic image to the user U. The display unit 10, the microlens array 30a, and the focal position setting unit 54 The display unit 10 includes a plurality of pixels, and displays image light L to a user. By making the light reach U, a stereoscopic image is provided to the user U. The microlens array 30a A plurality of lenses 32 are provided on the user U side of the display unit 10 in the optical axis direction of the image light L. a, and the focal point of the lens 32a can be changed individually. Based on the depth information indicating the position in the depth direction of the image, the focal point in the optical axis direction of the image light L is The drive control unit 52a sets a set focal position, which is a position of the lens 32a, for each lens 32a. The focal position of 2a is moved to the set focal position.

[0057] When displaying a stereoscopic image, it is necessary to provide the stereoscopic image appropriately to the user. In contrast, in the present embodiment, the position of the stereoscopic image in the depth direction is The focal position of the lens 32a is moved to the set focal position. According to this method, image light is projected to a user U at an appropriate light beam opening angle based on the position in the depth direction of a stereoscopic image. L, and a stereoscopic image can be appropriately provided to the user U. For example, as mentioned above, when displaying a stereoscopic image, a convergence accommodation conflict may occur. In contrast, in this embodiment, the opening angle of the light beam of the image light L is appropriately adjusted to perform convergence adjustment. In the second embodiment, the lens 32a is provided with an individual By separately setting the focal position, the virtual image opening angle and the image light L are It is possible to always match the opening angle.

[0058] The focal position setting unit 54a determines the opening angle of the light flux of the image light L passing through the lens 32a and From a position in the depth direction in the stereoscopic image displayed by the image light L toward the user U The difference between the luminous flux opening angle (virtual image opening angle) when the image light L is irradiated is within a specified range. The focal position of the lens 32a in the optical axis direction of the image light L is set as the set focal position. According to this embodiment, the difference between the light beam opening angle and the virtual image opening angle is reduced, thereby improving the convergence. Congestion adjustment conflicts can be appropriately reduced.

[0059] As described in the other examples, the display unit 10 of the second embodiment is a display panel including a plurality of pixels. and a light source unit that irradiates light onto the display panel. The pixel P may be included in the configuration. In either case, a stereoscopic image is appropriately provided to the user U. can.

[0060] (Modification) Next, a modified example will be described. The display device 1b according to the modified example has a The difference from the second embodiment is that the display is enlarged by a concave mirror 20C. In this regard, description of the configuration common to the second embodiment will be omitted.

[0061] FIG. 12 is a schematic diagram of a display device according to a modified example. As shown in FIG. The display device 1b does not have an eyepiece lens 20, and the display A half mirror 20B and a concave mirror 20C are provided on the eye EY side of the user U from the unit 10. The half mirror 20B and the concave mirror 20C can also be considered as optical elements. The image light L emitted from the display unit 10 is reflected by the half mirror 20B and enters the concave mirror 20. The image light L incident on the concave mirror 20C is reflected by the concave mirror 20C with a slight divergence angle. The light is then converted into almost parallel light, passes through the half mirror 20B, and enters the eye EY of the user U. .

[0062] In the modified example, each lens 32 of the microlens array 30a is arranged in the same manner as in the second embodiment. Therefore, even in the configuration of the modified example, the focal position of the second embodiment is controlled. In the same manner as in the above, a stereoscopic image can be appropriately provided to the user U, and the opening angle of the light beam of the image light L can be appropriately adjusted. By adjusting the convergence and accommodation conflict, the convergence and accommodation conflict can be reduced.

[0063] The modified example can also be applied to the first embodiment. The configuration and modifications other than those shown in FIG. 12 may be used.

[0064] Although the embodiments of the present invention have been described above, the present invention is not limited to the contents of these embodiments. In addition, the above-mentioned components are easily conceivable by a person skilled in the art, and are not substantially In addition, the above-mentioned components are appropriately Appropriate combinations are possible, and the configurations of the respective embodiments can also be combined. Furthermore, various omissions, substitutions or modifications of the components may be made without departing from the spirit of the above-described embodiments. Changes can be made. [Explanation of symbols]

[0065] 1 Display device 10 Display 20 Eyepiece 30, 30a Microlens array 32, 32a Lenses 40, 40a Control device 50 Image information acquisition unit 52, 52a Drive control section 54 Timing setting section 54a Focus position setting section 56, 56a Irradiation control unit L Image light P pixels U User

Claims

1. A display unit that includes a plurality of pixels and provides an image to a user by emitting light; a microlens array provided on the user side of the display unit, the microlens array having a plurality of lenses and capable of uniformly changing the focal points of the lenses; A drive control unit that moves the focal position of the lens at a predetermined period; a gaze detection unit that detects a gaze position at which the user is gazing in the image; a timing setting unit that sets an irradiation position for adjusting an opening angle of a light beam when the light is irradiated based on a depth direction position at the gaze position; Including, Display device.

2. The timing setting unit sets, as the irradiation position, a focal position of the microlens array at which a difference between an opening angle of a light beam when the light is irradiated to the user from a position in a depth direction at the gaze position and an opening angle of the light falls within a predetermined range. The display device according to claim 1 .

3. An image information acquisition unit that acquires depth information indicating a position in a depth direction of the image, the timing setting unit divides a depth direction into a plurality of numerical ranges, sets a predetermined value within each numerical range as a reference position, and sets the reference position for the numerical value included in the depth information as a position in the depth direction in setting the irradiation position. The display device according to claim 1 .

4. the drive control unit moves the focal position of the lens at least a half period of the predetermined period within a display time of one frame. The display device according to claim 1 .

5. a drive control step of moving, at a predetermined cycle, a position of a focal point of a microlens array having a plurality of lenses, the focal point of the lenses being uniformly changeable, the microlens array including a plurality of pixels and disposed closer to the user than a display unit that irradiates light; a gaze detection step of detecting a gaze position at which the user is gazing in the image; a timing setting step of setting an irradiation position for adjusting an opening angle of a light beam when the light is irradiated based on a depth direction position at the gaze position; having Display method.

6. a drive control step of moving, at a predetermined cycle, a position of a focal point of a microlens array having a plurality of lenses, the focal point of the lenses being uniformly changeable, the microlens array including a plurality of pixels and disposed closer to the user than a display unit that irradiates light; a gaze detection step of detecting a gaze position at which the user is gazing in the image; a timing setting step of setting an irradiation position for adjusting an opening angle of a light beam when the light is irradiated based on a depth direction position at the gaze position; to cause a computer to execute Display program.

Images