Virtual image display device and head-mounted display device

Abstract

To prevent distortion from forming in the projected virtual image when adjusting the convergence distance. [Solution] The system includes display elements 11A and 11B for displaying images, an optical member OE into which image light ML corresponding to the image is incident, a reflective member RE for reflecting the image light ML from the optical member OE, and a display control device 88 which corrects the image displayed on the display elements 11A and 11B so as to cancel out the distortion of the virtual image projected by the optical member OE and the reflective member RE. The display control device 88 adjusts the convergence distance of the virtual image by a convergence distance adjustment amount in the horizontal direction, i.e., a convergence adjustment shift amount L1 and L2, and changes the amount of distortion correction for the virtual image in conjunction with the image conversion shift amount d1x, which is at least a part of the convergence distance adjustment amount.

Description

[Technical Field]

[0001] The present invention relates to a virtual image display device and a head-mounted display device that enable the observation of virtual images. [Background technology]

[0002] A known head-mounted display device comprises a display element that displays an image corrected by a control unit, an optical element into which image light corresponding to the image is incident, and a reflective element that reflects the image light from the optical element and projects a virtual image corresponding to the image. The optical element and the reflective element correct distortion in a first direction related to the virtual image, and the control unit corrects the image according to distortion in a second direction that intersects with the first direction related to the virtual image (Patent Document 1). [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2023-151368 [Overview of the project] [Problems that the invention aims to solve]

[0004] In the above-mentioned Patent Document 1, no specific considerations have been given to adjusting the convergence distance. For example, in the apparatus of Patent Document 1, if the images displayed on a pair of display elements are shifted symmetrically so as to move closer together or further apart horizontally, a virtual image can be formed at a desired convergence angle. However, with such a method, distortion may be formed in the projected virtual image due to the shift of the displayed image. In addition, with such a method, image defects may occur due to the shift of the displayed image on the display elements. [Means for solving the problem]

[0005] In one aspect of the present invention, a virtual image display device includes a display element that displays an image, an optical member onto which image light corresponding to the image is incident, a reflecting member that reflects the image light from the optical member and projects a virtual image corresponding to the image, and a control device that corrects the image displayed on the display element so as to cancel distortion of the virtual image projected by the optical member and the reflecting member. When the control device adjusts the convergence distance of the virtual image by an amount of convergence distance adjustment related to the horizontal direction, the correction amount of distortion related to the virtual image is changed in conjunction with the converted shift amount of the image, which is at least a part of the amount of convergence distance adjustment.

[0006] A head-mounted display device in one aspect of the present invention includes a first device including the above virtual image display device and a second device including the above virtual image display device. The amount of convergence distance adjustment of the first device and the amount of convergence distance adjustment of the second device are the same in magnitude and opposite in direction.

Brief Description of the Drawings

[0007] [Figure 1] It is an external perspective view for explaining the wearing state of the head-mounted display device of the embodiment. [Figure 2] It is a side cross-sectional view for explaining the internal structure of the head-mounted display device. [Figure 3] It is a diagram for explaining the virtual image formed on the eye side and the original image displayed on the display element. [Figure 4] It shows the distortion state of the virtual image formed by correcting the distortion in the vertical direction in the optical system. [Figure 5] It is a diagram for explaining the setting of the convergence angle in the head-mounted display device. [Figure 6] It is a block diagram for explaining the circuit configuration of the head-mounted display device. [Figure 7] It is a diagram for explaining the display surface of the display element for the right eye. [Figure 8] It is a diagram for explaining the coordinate system before distortion conversion and the coordinate system after distortion conversion in the control unit. [Figure 9] It exemplifies a coefficient table for coordinate conversion with respect to the converted shift amount. [Figure 10]This is a block diagram that partially illustrates the display control device and display elements. [Figure 11] This diagram illustrates the correction of chromatic aberration in the optical system of a head-mounted display device. [Figure 12] This diagram illustrates linear interpolation for adjusting the tonal range of an image. [Figure 13] This is a flowchart illustrating image processing in a head-mounted display device. [Figure 14] The original initial image captured from the user's terminal is shown as an example. [Figure 15] This example shows an image before distortion correction using conversion shift. [Figure 16] This example shows an image after distortion correction using conversion shift. [Figure 17] This shows the virtual images observed by both eyes at each convergence distance. [Modes for carrying out the invention]

[0008] The following describes a virtual image display device and a head-mounted display device, which are embodiments of one invention, with reference to Figures 1 and 2, etc.

[0009] Figure 1 illustrates the wearing state of a head-mounted display device (HMD) 100. The HMD 100 causes the observer or wearer (US) wearing it to perceive an image as a virtual image. In Figure 1, X, Y, and Z are Cartesian coordinate systems. The +X direction corresponds to the lateral direction where the eyes (EY) of the observer or wearer (US) wearing the head-mounted display device 100 are aligned. The +Y direction corresponds to the upward direction perpendicular to the lateral direction where the eyes (EY) of the wearer (US) are aligned. The +Z direction corresponds to the forward or frontal direction for the wearer (US). The ±Y directions are parallel to the vertical axis or vertical direction.

[0010] The head-mounted display device 100 comprises a first virtual image display device 100A for the right eye, a second virtual image display device 100B for the left eye, a pair of temple-shaped support devices 100C that support the display devices 100A and 100B, and a user terminal 90 which is an information terminal. The first virtual image display device 100A consists of a display drive unit 102 located at the top and an exterior member 103 that is spectacle-lens-shaped and covers the area in front of the eye. Similarly, the second virtual image display device 100B consists of a display drive unit 102 located at the top and an exterior member 103 that is spectacle-lens-shaped and covers the area in front of the eye. The support devices 100C are mounting members that are attached to the head of the wearer US and support the upper end of the exterior member 103 via the display drive unit 102. The first virtual image display device 100A and the second virtual image display device 100B are optically reversed left and right, and a detailed explanation of the second virtual image display device 100B is omitted.

[0011] Figure 2 is a lateral cross-sectional view illustrating the optical structure of the first virtual image display device 100A. The first virtual image display device 100A comprises a display element 11, an imaging optical system 20, and a display control device 88. The imaging optical system 20 comprises a projection lens 21, a prism mirror 22, and a see-through mirror 23. Of the imaging optical system 20, the projection lens 21 and the prism mirror 22 correspond to the display drive unit 102 shown in Figure 1, and the see-through mirror 23 corresponds to the exterior member 103 shown in Figure 1. The combination of the display element 11, the projection lens 21, and the prism mirror 22 is called the projection optical system 12, and these are fixed in a case 51 in a state of mutual alignment. The case 51 is a housing or support member, formed of a light-shielding material, and supports the display control device 88 that operates the display element 11. The case 51 has an aperture 51a. The aperture 51a allows the projection optical system 12 to emit image light ML toward the outside.

[0012] The first virtual image display device 100A corrects distortion in a first direction related to the virtual image using an optical element OE, which is a prism mirror 22, and a reflective element RE, which is a see-through mirror 23. In this embodiment, for example, the prism mirror 22 and the see-through mirror 23 correct distortion in a first direction related to the virtual image, specifically in the vertical direction. Furthermore, the first virtual image display device 100A corrects the image or display image according to distortion in a second direction that intersects with the first direction related to the virtual image, using a display control device 88. In this embodiment, for example, the display control device 88 corrects distortion in the horizontal or left-right direction inherent in the virtual image by signal processing. Here, the second direction or left-right direction corresponds to the scanning direction of the display element 11. Distortion in the first direction related to the virtual image is eliminated in the first virtual image display device 100A as a result of the correction by the prism mirror 22 and the see-through mirror 23. Furthermore, any remaining distortion inherent in the virtual image is caused by the prism mirror 22 and the see-through mirror 23, and is canceled out by the distortion correction in the display control device 88, making it invisible to the eye EY. The projection lens 21 also affects the distortion of the virtual image, and the above correction is performed on the entire imaging optical system 20 of the first virtual image display device 100A.

[0013] The display element 11 is a self-emissive display device. The display element 11 is, for example, an organic electroluminescent (OLED) display, and forms a color still image or moving image on a two-dimensional display surface 11a. The display element 11 is arranged along an xy plane that is slightly rotated and tilted around the X axis relative to the XY plane. The display element 11 is driven by a display control device 88, which is the control unit, to perform display operations. In the example shown in Figure 2, the display element 11 is arranged in an inverted direction, with the upper +y direction of the display element 11 facing downwards, i.e., in the -Y direction, relative to the coordinates of the imaging optical system 20. In this case, due to the inverted characteristics of the imaging optical system 20, the image displayed on the display element 11 has the same orientation of display content as the original image before correction by the display control device 88.

[0014] The display element 11 is not limited to an organic EL display, but can be replaced with a display device using inorganic EL, organic LED, LED array, laser array, quantum dot light-emitting element, etc. The display element 11 is not limited to a self-emissive image light generation device, but may be composed of an LCD or other light modulation element, and an image may be formed by illuminating the light modulation element with a light source such as a backlight. Instead of an LCD, an LCOS (Liquid crystal on silicon, LCoS is a registered trademark) or a digital micromirror device may be used as the display element 11.

[0015] Of the imaging optical system 20, the projection lens 21 is an optical component OE and includes a first lens 21o, a second lens 21p, and a third lens 21q. The projection lens 21 receives image light ML emitted from the display element 11 and directs it onto the prism mirror 22. The projection lens 21 focuses the image light ML emitted from the display element 11 into a state close to a parallel beam. The prism mirror 22 has an incident surface 22a corresponding to the incident part, an internal reflective surface 22b corresponding to the reflective part, and an exit surface 22c corresponding to the exit part. The prism mirror 22 emits the image light ML incident from the front in a direction inclined with respect to the direction in which the incident direction is reversed (the direction of the light source as seen from the prism mirror 22). The see-through mirror 23 has a reflective surface 23a and an outer surface 23o. The see-through mirror 23 magnifies the intermediate image formed on the light emission side of the prism mirror 22.

[0016] The imaging optical system 20 is an off-axis optical system OS due to factors such as the see-through mirror 23 being a concave mirror. In this embodiment, the projection lens 21, prism mirror 22, and see-through mirror 23 are arranged non-axisymmetrically and have non-axisymmetric optical surfaces. The fact that the imaging optical system 20 is an off-axis optical system OS means that in the optical elements 21, 22, and 23 constituting the imaging optical system 20, the optical path as a whole bends before and after the incidence of light rays onto multiple reflective or refractive surfaces. In this imaging optical system 20, the optical axis AX is bent in an off-axis plane parallel to the YZ plane corresponding to the plane of paper, and the optical elements 21, 22, and 23 are arranged along this off-axis plane. The optical axis AX of the imaging optical system 20 includes optical axis portions AX1, AX2, and AX3 that are arranged along the off-axis plane and are inclined toward each other before and after the reflective surfaces. The overall optical axis AX has a Z-shaped arrangement. In other words, in the imaging optical system 20, the optical path P1 from the projection lens 21 to the internal reflection surface 22b, the optical path P2 from the internal reflection surface 22b to the see-through mirror 23, and the optical path P3 from the see-through mirror 23 to the pupil position PP are arranged in a Z-shape, folded back in two stages. The imaging optical system 20 is arranged vertically. Correspondingly, the off-axis surface (a surface parallel to the YZ plane), which is the reference plane, extends parallel to the vertical Y direction. In this case, the optical elements 21, 22, and 23 constituting the first virtual image display device 100A are arranged with their height positions changed in the vertical direction, and an increase in the width of the first virtual image display device 100A can be prevented.

[0017] In the imaging optical system 20, the optical path P1 from the projection lens 21 to the internal reflective surface 22b extends slightly diagonally upward or nearly parallel to the Z direction relative to the viewpoint. The optical path P2 from the internal reflective surface 22b to the see-through mirror 23 extends diagonally downward toward the front. When the horizontal plane (XZ plane) is used as a reference, the inclination of optical path P2 is greater than that of optical path P1. The optical path P3 from the see-through mirror 23 to the pupil position PP extends slightly diagonally upward or nearly parallel to the Z direction relative to the rear. In the illustrated example, the optical axis portion AX3 is at approximately -10° toward the +Z direction, with downward being negative. In other words, the exit optical axis EX, which is an extension of the optical axis portion AX3, extends at an angle of approximately 10° downward with respect to the central axis HX which is parallel to the forward +Z direction.

[0018] The incident and exit surfaces of the first lens 21o constituting the projection lens 21 are, for example, optical surfaces made of freeform surfaces, and have asymmetry with respect to the vertical direction parallel to the YZ plane and intersecting the optical axis AX, with respect to the optical axis AX, and have symmetry with respect to the horizontal direction or the X direction with respect to the optical axis AX. The incident and exit surfaces of the second lens 21p constituting the projection lens 21 are, for example, optical surfaces made of freeform surfaces, and have asymmetry with respect to the vertical direction parallel to the YZ plane and intersecting the optical axis AX, with respect to the optical axis AX, and have symmetry with respect to the horizontal direction or the X direction with respect to the optical axis AX. The incident and exit surfaces of the third lens 21q constituting the projection lens 21 are, for example, optical surfaces made of freeform surfaces, and have asymmetry with respect to the vertical direction parallel to the YZ plane and intersecting the optical axis AX, with respect to the optical axis AX, and have symmetry with respect to the horizontal direction or the X direction with respect to the optical axis AX.

[0019] The prism mirror 22 is an optical component with refractive and reflective functions that combine the functions of a mirror and a lens, and it reflects the image light ML from the projection lens 21 while refracting it. Specifically, the prism mirror 22 allows the image light ML to enter the interior through the incident surface 22a, totally reflects the incident image light ML in a non-front direction by the internal reflection surface 22b, and emits the incident image light ML to the outside through the exit surface 22c. The optical surfaces constituting the prism mirror 22, namely the incident surface 22a, the internal reflection surface 22b, and the exit surface 22c, are, for example, optical surfaces made of free-form surfaces, and have asymmetry with respect to the vertical direction parallel to the YZ plane and intersecting the optical axis AX, with respect to the optical axis AX, and have symmetry with respect to the horizontal direction or X direction with respect to the optical axis AX. The prism mirror 22 is made of, for example, resin, but can also be made of glass. The internal reflection surface 22b is not limited to one that reflects the image light ML by total internal reflection, but can also be a reflective surface made of a metal film or a dielectric multilayer film. In this case, a reflective film consisting of a single layer or multilayer made of a metal such as Al or Ag is deposited on the inner reflective surface 22b by vapor deposition or the like, or a sheet-like reflective film made of metal is attached.

[0020] The see-through mirror 23 is a curved, plate-shaped reflective optical element that functions as a concave surface mirror and reflects image light ML from the prism mirror 22. In other words, the see-through mirror 23 reflects image light ML from the prism mirror 22, which is positioned in the emission region of the projection optical system 12, toward the pupil position PP. The see-through mirror 23 covers the pupil position PP where the eye EY or pupil is located and has a concave shape toward the pupil position PP, and a convex shape toward the outside world. The see-through mirror 23 is a concave transparent mirror that covers the entire effective area of ​​the screen within the field of view. The see-through mirror 23 is a mirror plate having a structure in which a transparent mirror film 23c is formed on the surface or back surface of a plate-like body 23b. The reflective surface 23a of the see-through mirror 23 is, for example, an optical surface made of a free-form surface, and has asymmetry with respect to the vertical direction parallel to the YZ plane and intersecting the optical axis AX, and symmetry with respect to the horizontal or X direction, with respect to the optical axis AX. The reflective surface 23a of the see-through mirror 23 is, for example, a free-form surface. By making the see-through mirror 23 a free-form or aspherical surface, aberrations can be reduced, and in particular, when a free-form surface is used, it becomes easier to reduce aberrations in the imaging optical system 20, which is an off-axis optical system OS or a non-coaxial optical system.

[0021] The see-through mirror 23 is a transmissive reflective element that transmits some light during reflection, and the reflective surface 23a or mirror film 23c of the see-through mirror 23 is formed of a semi-transparent reflective layer. As a result, ambient light OL passes through the see-through mirror 23, enabling see-through viewing of the outside world and allowing a virtual image to be superimposed on the ambient image. In this case, if the plate-like body 23b supporting the mirror film 23c is thin to a few millimeters or less, the change in magnification of the ambient image can be kept small. The reflectivity of the mirror film 23c to the image light ML and ambient light OL is set to 10% to 50% within the expected incident angle range of the image light ML, from the viewpoint of ensuring the brightness of the image light ML and facilitating observation of the ambient image through see-through. The plate-like body 23b, which is the base material of the see-through mirror 23, is formed of resin, for example, but can also be made of glass. The plate-like body 23b is formed of the same material as the support plate 61 that supports it from the surroundings and has the same thickness as the support plate 61. The mirror film 23c is formed, for example, from a dielectric multilayer film consisting of multiple dielectric layers with adjusted film thicknesses. The mirror film 23c may also be a single-layer or multilayer film of a metal such as Al or Ag with adjusted film thickness. The mirror film 23c can be formed by lamination, but it can also be formed by attaching a sheet-like reflective film.

[0022] To explain the optical path, the image light ML from the display element 11 enters the projection lens 21 and is emitted from the projection lens 21 in a nearly collimated state. The image light ML that has passed through the projection lens 21 enters the prism mirror 22, passes through the incident surface 22a while being refracted, is reflected by the internal reflective surface 22b with a high reflectivity of nearly 100%, and is refracted again by the exit surface 22c. The image light ML from the prism mirror 22 enters the see-through mirror 23 and is reflected by the reflective surface 23a with a reflectivity of about 50% or less. The image light ML reflected by the see-through mirror 23 enters the pupil position PP where the wearer's eye EY or pupil is located. External light OL that has passed through the see-through mirror 23 and the surrounding support plate 61 also enters the pupil position PP. In other words, the wearer US wearing the first virtual image display device 100A can observe a virtual image created by the image light ML superimposed on the external image.

[0023] Although a detailed explanation is omitted, in this embodiment, on the off-axis surface or reference surface parallel to the XY cross-section of the see-through mirror 23, the angle of the tangent of the reflective surface 23a passing through the intersection of the reflective surface 23a and the optical axis AX with respect to the Z axis, i.e., the arrangement angle, is relatively large. As a result, in the first virtual image display device 100A, vertical distortion is suppressed, but trapezoidal distortion mainly in the horizontal direction occurs. However, even just suppressing vertical distortion eliminates the need for the display control device 88 to maintain a frame buffer, as will be described in detail later. This makes it possible to handle distortion correction with an inexpensive general-purpose FPGA or other integrated circuit, and also reduces power consumption.

[0024] Figure 3 is a hypothetical diagram illustrating the virtual image AA formed on the eye EY side and the original image BB displayed on the display element 11 when no correction is made by the display control device 88. In Figure 3, the solid line shows the virtual image AA corrected in the vertical direction via the imaging optical system 20 of the first virtual image display device 100A, and the dashed line shows the display area SB of an ideal rectangular image corresponding to the original image BB. In the following explanation, the image observed by the eye EY will be referred to as a virtual image or projected image, and the image formed on the display control device 88 will be referred to as an image or displayed image.

[0025] Before distortion correction is performed by the display control device 88 (described later), the virtual image AA formed by the first virtual image display device 100A has a length D1 in the left-right direction corresponding to the second direction in one of the vertical directions corresponding to the first direction of the projected image AA1 corresponding to the virtual image AA, which is shorter than the length D2 in the left-right direction in the other vertical direction of the projected image AA1. For example, if the shape of the image BB before correction is a rectangle, then due to a biased optical correction with respect to direction during projection by the imaging optical system 20 of the first virtual image display device 100A, the corrected projected image AA1 becomes trapezoidal. In the example in Figure 3, the upper side or the +Y direction (corresponding to the -y direction) is one side, and the lower side or the -Y direction (corresponding to the +y direction) is the other side. The display control device 88 performs distortion correction in the opposite direction to the trapezoidal projected image AA1 that narrows at the top through signal processing, so that the corrected virtual image that is finally visible is the same rectangle as the original image BB.

[0026] Figure 4 illustrates the distortion state of the virtual image AA formed by correcting vertical distortion in the imaging optical system 20 of the first virtual image display device 100A. As shown in the figure, the virtual image AA includes a corrected first correction region SP1 obtained by optically correcting the first region AP1 of the original image BB, and a plurality of corrected second correction regions SP2 obtained by optically correcting a plurality of second regions AP2 aligned horizontally with the first region AP1 of the image before correction. The first and second regions AP1 and AP2 are introduced on the premise of subsequent signal processing correction and are unit regions with a certain extent. The first region AP1 is a reference region set one per row extending horizontally. The first region AP1 and the second region AP2 correspond to one division when the entire pixel area of ​​the image BB corresponding to the virtual image AA is divided vertically and horizontally into 10 sections. In the original image BB, if the horizontal is 1920 pixels and the vertical is 1080 pixels, one section is 192 × 108 pixels.

[0027] The first region AP1 and the second region AP2 shown in Figure 4 are illustrative examples, and their ranges can be set as appropriate. The ranges of the first correction region SP1 and the second correction region SP2 are determined in accordance with the first region AP1 and the second region AP2 set in this manner.

[0028] The number of pixels in the left-right direction of the corrected first correction area SP1 (width relative to the pixels of the original image BB) is within ±10% of the number of pixels in the left-right direction of the corrected second correction area SP2. In this embodiment, with respect to the trapezoidal shape of the virtual image AA, the distortion interval or width of the second correction area SP2 in the lateral direction changes approximately constant within a range of ±20 pixels relative to the average value of the lateral lengths of the first and second correction areas SP1 and SP2. Furthermore, the lateral distortion changes approximately constant within a range of ±20 pixels in the vertical direction of the virtual image AA. In other words, the lateral distortion of the virtual image changes gradually according to the vertical and horizontal position when viewed in units of correction areas SP1 and SP2, making distortion correction by signal processing in the display control device 88 relatively easy. The distortion inherent in the imaging optical system 20 of the first virtual image display device 100A is corrected by the display control device 88, which will be described later, to cancel out these distortions.

[0029] Figure 5 illustrates the setting of the convergence angle by the head-mounted display device 100. In the first virtual image display device 100A for the right eye, the display element 11 and the imaging optical system 20 constitute the first device 1A. In the second virtual image display device 100B for the left eye, the display element 11 and the imaging optical system 20 constitute the second device 1B.

[0030] Referring to Figure 5, convergence refers to the convergence of the EY (eyes) when viewing something close up, and the angle between the visual axes XE1 and XE2 of the EY is called the convergence angle θ. When the standard observation distance set on the head-mounted display device 100, that is, the distance at which a virtual or projected image is observed due to parallax, is defined as the reference convergence distance L0, the convergence angle θ = θ0 is given by θ0 = 2 × tan in the frontal direction. -1 It is given by (a / 2L0), where the value a is the distance between the eye sockets (EY).

[0031] In the head-mounted display device 100 of this embodiment, the positions of the virtual or projected images displayed by both virtual image display devices 100A and 100B are set to be tilted inward by half the convergence angle θ, with respect to the front-view visual axes XEA0 and XEB0. Specifically, the optical axes AX of both virtual image display devices 100A and 100B are adjusted from the outset to coincide with the visual axes XE10 and XE20, which correspond to the convergence angle θ=θ0, by adjusting the relative positioning of the optical system. As a result, a virtual image that coincides with the reference convergence distance L0 is observed by both virtual image display devices 100A and 100B. In the specific example of fabrication, the reference convergence distance L0 was set to 5m.

[0032] The head-mounted display device 100 of this embodiment allows for adjustment of the virtual image observation distance by differentiating the image processing in the two virtual image display devices 100A and 100B. In other words, by adjusting the convergence angle θ, the virtual image can be projected at any position within a distance range spanning the far convergence distance LA and the near convergence distance LB, with the reference convergence distance L0 in between. When adjusting the convergence angle θ by image processing, one method is to shift the image or display image formed on the display surface 11a of the display element 11 by a converted distance equivalent to half the convergence angle θ θ / 2 in the +x or -y direction. However, with this method, if aberrations such as lateral distortion remain in the imaging optical system 20, the resulting projected image, the virtual image, will have distortion at least in the lateral direction, which may hinder precise and less strenuous binocular vision. As will be described in detail later, the head-mounted display device 100 of this embodiment has a display control device 88 that, in order to suppress the generation of distortion associated with the adjustment of the observation distance or convergence angle as described above, stores a table and conversion formulas that convert two-dimensional azimuth angles related to two directions perpendicular and mutually orthogonal to the visual axes XE10 and XE20 corresponding to the basic convergence angle θ=θ0 in the space in front of the eyes into xy coordinate positions on the display surface 11a. The display control device 88 then rereads the conversion table and conversion formulas each time the convergence distance is changed, and projects the original image corresponding to the two-dimensional distribution of azimuth angles in front of the eyes as a virtual image that is close to the original.

[0033] Note that in Figure 5, for the sake of simplicity, the Y-axis is shown perpendicular to the plane of the paper and the Z-axis is shown parallel to the plane of the paper. However, strictly speaking, the Y-axis is tilted by about 10° relative to the plane of the paper in relation to the emission optical axis EX (see Figure 2), and the Z-axis is also tilted by about 10° relative to the plane of the paper.

[0034] Referring to Figure 6, the circuit system 70 of the head-mounted display device 100 will be described. The head-mounted display device 100 includes a display control device 88, a pair of display elements 11, and a user terminal circuit 91 as the circuit system 70. One display element 11A is incorporated into the first virtual image display device 100A, and the other display element 11B is incorporated into the second virtual image display device 100B. The display control device 88 functions as a control device CNT. In the illustrated example, the display control device 88 is shown as being incorporated into the first virtual image display device 100A, but it may be independent of the first virtual image display device 100A and the second virtual image display device 100B.

[0035] The display control device 88 includes an arithmetic processing unit 81a, a storage device 81m, and a data communication interface 81c.

[0036] The memory device 81m stores programs that cause the first virtual image display device 100A and the second virtual image display device 100B to perform display operations. The memory device 81m also stores images acquired from the user terminal 90, which is an information terminal, and images generated by the arithmetic processing unit 81a. The memory device 81m also has a frame memory 83, which stores image data corresponding to the image data generated by the arithmetic processing unit 81a and for output to the display elements 11. The image data output from the display control device 88 to the pair of display elements 11 are slightly different from each other in order to form parallax, as will be described later.

[0037] Furthermore, the storage device 81m has a non-volatile memory 87, which stores various data such as parameters used in the image correction calculations described later.

[0038] The display control device 88 causes a pair of display elements 11A and 11B to perform display operations. Each display element 11A and 11B includes an auxiliary circuit 85 that incorporates a scanning driver and a data driver around the display surface 11a. When displaying each frame image, the display control device 88 outputs data signals corresponding to the image data stored in the frame memory 83 or image data that has undergone correction processing thereto, along with timing signals, etc., to the auxiliary circuit 85 in units of scan lines. The auxiliary circuit 85 rewrites the display state of the display surface 11a according to the data signals etc. input from the display control device 88. The display control device 88 outputs data to the auxiliary circuit 85 line by line for each image data, and the display is performed on the display surface 11a of each display element 11A and 11B in the scanning direction corresponding to the x direction in Figure 2. The x direction in Figure 2 corresponds to the horizontal or left-right direction of the display surface 11a. The image data stored in the frame memory 83 increases by one line at a time for each frame. After all the lines of the image have been displayed on the display element 11, the memory is reset, and the image data for the next frame is stored in the same manner.

[0039] The display control device 88 can receive display data corresponding to image data from the user terminal circuit 91 via the data communication interface 81c and store it in the frame memory 83. The display control device 88 performs a batch of processing on the display data or image data stored in the frame memory 83, including distortion correction to compensate for the remaining distortion of the imaging optical system 20 in the lateral direction and adjustment of the convergence angle between the virtual image display devices 100A and 100B to achieve the desired observation distance, and outputs the processed display data, which is the image data, to the display elements 11A and 11B.

[0040] Figure 7 illustrates the display surface 11a of the display element 11A for the right eye. The display surface 11a has a rectangular frame-shaped extended area A2 surrounding the four sides of the basic area A1, which corresponds to a predetermined image size. The image size of the basic area A1 is set to a standard size, for example, 1920 × 1080 pixels. The extended area A2 has a pixel width of ΔW in the long side direction, i.e., in the x-direction, in the left and right short side portions, and a pixel width of ΔL in the short side direction, i.e., in the y-direction, in the top and bottom long side portions. The pixel width ΔW is a margin required when adjusting the convergence angle or convergence distance between the virtual image display devices 100A and 100B. The pixel width ΔL is provided as an adjustment margin in case the relative positioning of the virtual image display devices 100A and 100B is insufficient, but it is an unnecessary margin if the relative positioning of the virtual image display devices 100A and 100B can be done with high precision. In this specific example, the pixel width ΔW is set to 16 pixels, and the pixel width ΔL is set to 12 pixels. In this case, the image size of the display surface 11a is 1952 × 1104 pixels. Note that the display surface 11a of the display element 11B for the left eye has the same structure as the display surface 11a of the display element 11A for the right eye, and therefore its explanation is omitted.

[0041] Figure 8 illustrates the coordinate system before and after correction in the display control device 88. Figure region AR1 in Figure 8 is the coordinate system of the normal state before correction and shows the initial image IM0. Figure region AR2 in Figure 8 is the coordinate system of the display state on the display element 11A after correction when the target of congestion distance adjustment is the reference congestion distance L0 and shows the corrected image IM1. Figure region AR3 in Figure 8 is the coordinate system of the display state on the display element 11A after correction when the target of congestion distance adjustment is the congestion distance La1 (see Figure 5) and shows the corrected image IM2. In a specific example, the xy coordinates of the four corner vertex pixels VE1, VE2, VE3, and VE4 on the display surface 11a of the display element 11A are (-975.5, +551.5), (+975.5, +551.5), (+975.5, -551.5), and (-975.5, -551.5). Furthermore, the x and y coordinates of the four corner vertex pixels IV1, IV2, IV3, and IV4 of the basic region A1 corresponding to the initial image IM0 are (-959.5, +539.5), (+959.5, +539.5), (+959.5, -539.5), and (-959.5, -539.5).

[0042] Initial image IM0 is the original display image before correction by the display control device 88. Corrected image IM1 is the display image after distortion correction by the display control device 88. Corrected image IM1 is a display image in which the distortion generated in the imaging optical system 20 is corrected in the lateral direction, i.e., the x-direction, so that the virtual image formed on the eye side EY is rectangular. Corrected image IM2 is a display image after adjusting the convergence distance while performing distortion correction by the display control device 88. Corrected image IM2 is not simply corrected image IM1 shifted in the +x direction in response to the change in the convergence distance from L0 to La1, but also has a modification made to the amount of distortion correction in the lateral direction. Corrected image IM2 is an image in which the distortion generated in the imaging optical system 20 is corrected while taking into account the change in the visual axis XE1 (see Figure 5), so that the virtual image formed on the eye side EY is rectangular.

[0043] As shown in FIG. 6, the display control device 88 performs various image processes including arithmetic processing for distortion correction and convergence distance adjustment so that the initial image IM0 with a rectangular outline corresponding to the input signal from the user terminal circuit 91 becomes the corrected images IM1 and IM2 with a non-rectangular outline to be displayed on the display surface 11a.

[0044] The arithmetic processing device 81a acquires an image or a display image from the user terminal 90 and stores it in the storage device 81m. The image is a display image to be displayed on the head-mounted display device 100, specifically, the initial image IM0 shown in FIG. 8.

[0045] The arithmetic processing device 81a has a correction unit 86 for performing distortion correction and convergence distance adjustment as shown in FIG. 8. The correction unit 86 not only generates a distortion image that cancels out the trapezoidal screen distortion generated in the imaging optical system 20 of the first virtual image display device 100A and displays it on the display element 11, but also, when the setting of the convergence distance changes, generates a distortion image that cancels out the trapezoidal screen distortion generated in the imaging optical system 20 in accordance with the new convergence distance setting and displays it on the display element 11. The correction unit 86 is a correction circuit that generates a distortion image adjusted to the convergence distance with simple calculation processing and enables adjustment of the distortion state and display range by parameter adjustment. The correction unit 86 is implemented, for example, in a general-purpose FPGA together with other circuits constituting the display control device 88.

[0046] The images shown in the figure area AR2 and the figure area AR3 in FIG. 8 are both corrected images to be displayed on the display surface 11a, which are wide inverted trapezoids on the upper side, where D1 < D2 and D1' < D2', but the lengths D1, D2 and the lengths D1', D2' are different values, and the positions of the vertices at the four corners are also different in the x direction.

[0047] Returning to FIG. 6, in the arithmetic processing device 81a, the correction unit 86 has a coordinate conversion unit 8a and a gradation conversion unit 8b.

[0048] The coordinate transformation unit 8a performs a correction process, which is a type of image processing, to transform the horizontal position or the x-coordinate position. Specifically, the coordinate transformation unit 8a performs a correction process to achieve the target convergence distance while distorting the initial image IM0 to compensate for the horizontal distortion aberration that occurs in the imaging optical system 20 shown in Figure 2, thereby creating the corrected image IM1 or corrected image IM2 shown in Figure 8.

[0049] The gradation conversion unit 8b converts the gradation or brightness for each pixel corresponding to the coordinates converted or corrected by the coordinate conversion unit 8a. In other words, the gradation conversion unit 8b adjusts the gradation of the corrected images IM1 and IM2 by interpolation. The gradation adjustment by the gradation conversion unit 8b takes into account that when the initial image IM0 is converted to the corrected image IM1, the x-coordinate values ​​(i.e., the pixel points after coordinate conversion) are shifted by an amount including the integer and decimal parts from the original pixel points or grid points based on the pixel spacing, resulting in fractional positions or intermediate positions corresponding to the decimal part. From the coordinate values ​​of such intermediate positions, the unit calculates the gradation that is estimated to be appropriate for the pixel points that are located around the intermediate positions (specifically in the horizontal direction) on the display elements 11A and 11B and are actually displayed.

[0050] The following describes the specific processing in the coordinate transformation unit 8a. Here, the lateral distortion aberration occurring in the imaging optical system 20 shown in Figure 2 is assumed to be a function determined by an algebraic equation that includes the original position. The xy coordinate system, in which the center of the image or display image before correction is the origin and the size of one pixel is 1, is transformed into the following UV coordinate system by distortion correction or distortion transformation, which corresponds to compensation for distortion aberration.

[0051] When the coordinates of the initial image IM0 before correction are denoted as y in the vertical direction, the coordinates of the initial image IM0 before correction are denoted as x in the horizontal direction, the coordinates of the corrected image IM1 after correction are denoted as U in the horizontal direction, the coefficient corresponding to the length of the corrected image IM1 after correction is denoted as a, and the coefficient corresponding to the angle between the vertical side and the horizontal side of the corrected image IM1 after correction is denoted as b, the coordinate transformation unit 8a of the display control device 88 corrects the initial image IM0 so that it satisfies the following transformation formula. U = ax + bxy … (1) The coefficient 'a' corresponds to the relative ratio of the original lengths in the left-right direction. The coefficient 'b' corresponds to the slopes of the opposite sides of the trapezoid that are not parallel to each other, i.e., the slopes of the legs or meridians of the trapezoid.

[0052] Furthermore, if V is the vertical coordinate of the corrected image IM1, then the following transformation formula is satisfied. V=y … (2)

[0053] Furthermore, if c is a coefficient corresponding to the distortion between one tilt in the left-right direction and the other tilt in the left-right direction of the corrected image IM1 after correction, the coordinate transformation unit 8a of the display control device 88 corrects the image so that it satisfies the following transformation formula. U = ax + bxy + cy … (1)' The coefficient c corresponds to the imbalance in the slopes of opposite sides of the trapezoid that are not parallel to each other, that is, the imbalance in the slopes of the legs or meridians of the trapezoid. In other words, if c=0, transformation formula (1)' is the same as transformation formula (1), and it means a coordinate transformation to the corrected image IM1 of the isosceles trapezoid type.

[0054] In the above, coefficients a, b, and c are parameters that can be rewritten.

[0055] To easily adjust the convergence distance of the corrected image IM1, the initial image IM0 before correction can be moved in the x-direction according to, for example, the convergence distance adjustment amount (corresponding to the convergence adjustment shift amounts L1 and L2 described later). Here, the convergence distance adjustment amount is obtained by replacing the lateral displacement angle of the virtual image for the eye EY with a lateral positional displacement in the x-direction on the display control device 88. Changing the convergence distance as a difference from the reference convergence distance L0 shown in Figure 5 at the initial image IM0 stage using this method is called image shift, and the pixel-level value that realizes such an image shift is called the image shift amount L1X. Image processing using such image shift is performed by the display control device 88, and the display control device 88 can suppress distortion related to the projected virtual image in conjunction with the image shift amount L1X of the image. However, when using such convergence distance adjustment, the virtual image corresponding to the image that extends horizontally beyond the original display area, i.e., the basic area A1 shown in Figure 7, will generally not be displayed, resulting in image loss. Furthermore, even virtual images projected from the basic region A1 will have some distortion, as eye movement and other factors are not adequately corrected for distortion aberration.

[0056] To obtain a corrected image IM2 with precisely adjusted vergence distance, it is possible to gradually modify the coordinate transformation coefficients a, b, and c in accordance with the vergence distance adjustment amount (corresponding to the vergence adjustment shift amounts L1 and L2 described later) for the original image IM0 before correction. Changing the vergence distance as a difference from the reference vergence distance L0 shown in Figure 5 at the coordinate transformation stage using this method is called a converted shift, and the pixel-level value that realizes such a converted shift is called the converted shift amount d1x. Such a converted shift is based on the visual axis XE1, and if the accurate coefficients a, b, and c are determined in advance through simulation or actual measurement, it is possible to project a virtual image with almost no distortion using the entire display surface 11a shown in Figure 7. Such image processing is performed by the display control device 88, which changes the correction amount that cancels out distortion related to the projected virtual image in conjunction with the converted shift amount d1x of the image.

[0057] Figure 9 illustrates a coefficient table that sets variable coefficients a, b, and c for a converted shift amount dx. Here, (ak, bk, ck) = (a1x, b1x, c1x) represents the coefficient set (a, b, c) when the converted shift amount d1x is between -16 and +16, where k is an integer. However, for example, when the integer k is between +1 and +3, the coefficient set (a1x +3 ,b1x +3 ,c1 x+3 The coefficients (a1x, b1x, c1x) are common to all. In other words, the coefficient set (a1x, b1x, c1x) is provided in units of 3 pixels, i.e., multiple pixels, for changes in the converted shift amount d1x, and is kept constant within the range of variation of 3 pixels. This takes into account that for shift amounts of about ±1 pixel, the amount of distortion in the imaging optical system 20 shown in Figure 2 hardly changes. However, the coefficient set (a1x, b1x, c1x) may be provided in units of 1 pixel for changes in the converted shift amount d1x, or in units of 2 pixels or 4 pixels or more for changes in the converted shift amount d1x. The converted shift amount dx has an upper limit of +16, a maximum shift amount dxmax=16, a lower limit of -16, and a minimum shift amount -dxmax=-16, but can be increased or decreased according to the setting range of the convergence distance. However, in accordance with the setting of the maximum shift amount (upper limit) dxmax, etc., it is necessary to match the pixel width ΔW of the extended area A2 provided on the display surface 11a with the maximum shift amount dxmax.

[0058] The above describes the congestion distance adjustment for driving the display element 11A of the first virtual image display device 100A, but the congestion distance adjustment for driving the display element 11B of the second virtual image display device 100B is the same. However, in the case of the display element 11B, when a conversion shift is performed, the conversion shift amount d2x is adjusted from -16 to +16, and the coefficient set (a,b,c) corresponding to this is coefficient set (ak,bk,ck)=(a2x,b2x,c2x). Note that the direction of the conversion shift is reversed between the first virtual image display device 100A and the second virtual image display device 100B, so d2x=-d1x holds. Also, in the case of the display element 11B, when an image shift is performed, if the image shift amount in that case is L2X, then L2X=-L1X.

[0059] Figure 10 is a block diagram that partially illustrates the display control device 88 and the display elements 11A and 11B. In the coordinate transformation unit 8a of the correction unit 86 shown in the figure, the image signal of the first virtual image display device 100A for the right eye and the image signal of the second virtual image display device 100B for the left eye are received, and distortion transformation processing is performed on each image signal.

[0060] As shown in Figure 10, the coefficient sets (a,b,c)=(ak,bk,ck), that is, the coefficient sets (a1x,b1x,c1x) and (a2x,b2x,c2x), which give the transformation formula for coordinate transformation, are recorded in the non-volatile memory 87 of the storage device 81m, which can be rewritten from the outside. The correction unit 86 obtains the coefficient values ​​recorded in the non-volatile memory 87 and performs distortion calculation. Specifically, the coordinate transformation unit 8a of the correction unit 86 receives an input signal J1 corresponding to the image signal for the first virtual image display device 100A and a congestion adjustment shift amount L1, and an input signal J2 corresponding to the image signal for the second virtual image display device 100B and a congestion adjustment shift amount L2. Here, the congestion adjustment shift amount L1 corresponds to the horizontal congestion distance adjustment amount for the right eye EY, and the congestion adjustment shift amount L2 corresponds to the horizontal congestion distance adjustment amount for the left eye EY. The coordinate transformation unit 8a performs calculations on the input signal J1 using a coefficient set (a1x, b1x, c1x) corresponding to the congestion adjustment shift amount L1, and outputs the output signal K1 to the display element 11A. The coordinate transformation unit 8a performs calculations on the input signal J2 using a coefficient set (a2x, b2x, c2x), and outputs the output signal K2 to the display element 11B. Here, the coefficient sets (a1x, b1x, c1x) and (a2x, b2x, c2x) are selected from the coefficient table exemplified in Figure 9 according to the converted shift amounts d1x, d2x determined from the congestion adjustment shift amounts L1, L2 by a rule described later. Note that the correction unit 86 may output two output signals with different distortion conditions using a single input system instead of two input systems.

[0061] The non-volatile memory 87 stores the values ​​of the right eye coefficient set (a1x, b1x, c1x) and the left eye coefficient set (a2x, b2x, c2x). During distortion processing by the correction unit 86, the coefficients a1x, b1x, c1x, a2x, b2x, c2x are retrieved, and calculation processing involving coordinate transformation or conversion is performed. This allows the right and left eye images to be output to the display elements 11A and 11B with different distortion states. The coordinate transformation coefficients recorded in the non-volatile memory 87 can be accessed externally and rewritten, for example, via a serial communication method such as I2C. The correction unit 86 also performs interface conversion to convert the image signal to one suitable for the display elements 11A and 11B. An example of a correction unit 86 used in practice is an FPGA, and a specific example is the Lattice LIFCL-17-8MG121C.

[0062] The correction achieved by the coordinate transformation unit 8a, as shown in Figure 8, not only corrects the distortion of the imaging optical system 20 as shown in Figure 4, but also makes it possible to set the convergence distance of the virtual image display devices 100A and 100B to a target value. In this embodiment, the upper side of the display element 11 is positioned downward with respect to the coordinates of the imaging optical system 20, i.e., in the -Y direction, in a reverse orientation. The image that has passed through the imaging optical system 20 is inverted vertically and horizontally by the relay system, and the virtual image seen by the wearer US is trapezoidally corrected, so that the orientation of the displayed content matches that of the original image.

[0063] Furthermore, when converting a color image in the coordinate transformation unit 8a, it is desirable to suppress the occurrence of chromatic aberration, and the image is transformed so that the image containing each wavelength is displayed on the display element 11 as follows.

[0064] Specifically, as shown in Figure 11, the display elements 11A and 11B display a corrected image IM2 consisting of a first image IMG containing a first wavelength, a second image IMB containing a second wavelength shorter than the first wavelength, and a third image IMR containing a third wavelength longer than the first wavelength. In this case, the horizontal length E1 in the first image IMG is the length between the horizontal length E2 in the second image IMB and the horizontal length E3 in the third image IMR, and the vertical length F1 in the first image IMG is the same as the vertical length F2 in the second image IMB and the vertical length F3 in the third image IMR. This allows the correction amount to be changed for each wavelength band, thereby improving the display accuracy of the color image.

[0065] Returning to Figure 6, the gradation adjustment process in the gradation conversion unit 8b will now be explained. In the gradation conversion unit 8b, for example, linear interpolation is performed.

[0066] After the coordinate transformation unit 8a derives the corrected coordinate values, the grayscale transformation unit 8b performs interpolation to properly derive the grayscale values ​​of each pixel of the display elements 11A and 11B. The simplest method for this is nearest neighbor interpolation. This method uses the grayscale value of the closest coordinate value as is, but when displaying lines or characters, for example, jagged edges or character distortion are likely to occur. On the other hand, if the grayscale values ​​of the four nearest points are taken and cubic spline interpolation is used, a smoother image can be displayed, but the calculation process becomes complex. In this embodiment, N-division linear interpolation is performed, which is based on linear interpolation and can be performed with simple calculations when deriving the grayscale values.

[0067] The following shows the calculation method used during linear interpolation, which corresponds to grayscale adjustment, with reference to Figure 12. Figure 12 shows area BR1, which illustrates the relationship between the pixel center coordinate PM and the converted coordinate TM in the i-th row pixel PE of the display element 11. In Figure 12, the value Up i,j This indicates the x-coordinate, and the value Vp i,j This indicates the y-coordinate, and the value Rp i,j This indicates the gradation value of red, and the value Gp i,j This indicates the green grayscale value, value Bpi,j indicates the blue gradation value. The interpolation calculation is performed for each line and each wavelength.

[0068] The figure area BR2 in FIG. 12 is a figure for explaining the pixel center coordinates PM in a green image and the gradation value corresponding to the converted coordinates TM. As shown in the figure area BR2 of FIG. 12, two points B after correction or conversion i,j , B i,j+1 's gradation value Gb i,j , Gb i,j+1 and the coordinates B i,j (Ub i,j , Vb i,j ), B i,j+1 (Ub i,j+1 , Vb i,j+1 ), for the pixel P of the display element 11A existing between the two points, regarding the gradation value Gp i,j and the coordinates (Up i,j , Vp i,j ), if we set them as the gradation value Gp i,j of the green pixel of the display elements 11A and 11B, then the green gradation value Gp i,j is calculated by the following formula. Gp i,j = Gb i,j + α(Gb i,j+1 - Gb i,j ) However, the value α is a coefficient corresponding to the slope of the line segment indicating the relationship between coordinates and gradation and is a coefficient that enables gradation interpolation and is 1 or less. [[ID=5))

[0069] Although detailed explanation is omitted, for red and blue pixels, the gradation value is calculated by the same method as above.

[0070] The above has explained the gradation adjustment by linear interpolation, but gradation adjustment can be performed not only by linear interpolation but also using various other approximation methods.

[0071] ​​The user terminal circuit 91 shown in Figure 6 is incorporated into the user terminal 90 and comprises a main control unit 91a, a storage device 91m, a data communication interface 91c, a mobile wireless communication device 91t, and a user interface device 91i. The user terminal circuit 91 can communicate with various devices such as external servers via a communication network (not shown) using the mobile wireless communication device 91t. The storage device 91m stores a basic program for operating the user terminal circuit 91, and also stores multiple application software programs that run on this basic program, such as a video viewer and a web browser. The user terminal circuit 91 operates in response to requests from the user interface device 91i operated by the user, outputting videos and still images stored in the storage device 91m in a predetermined format to the display control device 88, or acquiring videos and still images corresponding to various content via the mobile wireless communication device 91t, and outputting the acquired display data to the display control device 88 in a predetermined format. In the above description, the display control device 88 performs distortion correction processing and congestion distance adjustment on the display data input from the user terminal circuit 91. However, the user terminal circuit 91 may also perform distortion correction processing and congestion distance adjustment on the display data.

[0072] The image processing to be displayed on the display surface 11a of the display elements 11A and 11B will be explained below with reference to Figure 13.

[0073] First, in the display control device 88 shown in Figure 6, the arithmetic processing unit 81a receives the congestion distance setting (congestion adjustment shift amounts L1, L2 in Figure 10) from the user terminal 90 via the data communication interface 81c (step S11). The congestion distance setting is within the range of congestion distance LA to LB shown in Figure 5. The congestion distance setting can be set by the user, for example, but it may also be received as data associated with the initial image IM0, which will be described later. The arithmetic processing unit 81a converts the congestion distance setting into pixel-level congestion adjustment shift amounts L1, L2 (step S12). The congestion adjustment shift amounts L1, L2 are set to a maximum of ±50 pixels, for example.

[0074] Next, the arithmetic processing unit 81a acquires the initial image IM0 as display data (input signals J1, J2 in Figure 10) from the user terminal 90 via the data communication interface 81c, that is, it reads out the initial image IM0, which is image data, and stores it in the storage device 81m (step S13). The user terminal circuit 91 outputs the video or still image stored in the storage device 91m to the display control device 88 in a predetermined format, or outputs the video or still image acquired via the mobile wireless communication device 91t to the display control device 88 in a predetermined format.

[0075] Next, in the display control device 88, the arithmetic processing unit 81a determines whether the absolute values ​​of the congestion adjustment shift amounts L1 and L2 are less than or equal to the maximum shift amount (upper limit) dxmax (step S14), and performs different processing depending on the result. Here, the maximum shift amount dxmax corresponds to the congestion distances La1 and Lb1 shown in Figure 5.

[0076] The arithmetic processing unit 81a performs distortion correction using only conversion shift without image shifting if the absolute values ​​of the congestion adjustment shift amounts L1 and L2 are less than or equal to the upper limit, which is the maximum shift amount dxmax (step S15). At this time, a process is performed to change the correction amount that cancels out the distortion according to the target conversion shift amount, specifically by selecting or changing a coefficient based on the coefficient table shown in Figure 9. The conversion shift amount d1x for the display element 11A becomes equal to the congestion adjustment shift amount L1, and the conversion shift amount d2x for the display element 11B becomes equal to the congestion adjustment shift amount L2. As a result, distortion correction and congestion distance adjustment are performed together.

[0077] Next, the arithmetic processing unit 81a, acting as the grayscale conversion unit 8b of the correction unit 86, performs grayscale adjustment for each pixel in the corrected image IM2, assuming that the initial image IM0 is coordinate-transformed to the corrected image IM2 (step S16). In other words, the arithmetic processing unit 81a performs grayscale adjustment for each pixel point that constitutes the display elements 11A and 11B by interpolation from the pixel points of the corrected image IM2 after the coordinate transformation.

[0078] The processing unit 81a outputs image data that has undergone grayscale adjustment to the corrected image IM2, which has undergone distortion correction and convergence distance adjustment, to the display elements 11A and 11B, or reads image data such as the corrected image IM2 line by line from the frame memory 83 of the storage device 81m, transfers it to the display elements 11A and 11B, and displays the image on the display surface 11a (step S17). The trapezoidal corrected image IM2, etc., displayed on the display element 11, which has undergone grayscale adjustment, is perceived as a rectangular image in the virtual image projected via the imaging optical system 20.

[0079] If the absolute values ​​of the congestion adjustment shift amounts L1 and L2 exceed the upper limit, which is the maximum shift amount dxmax, the processing unit 81a first performs distortion correction by image shift as the first stage of correction (step S21). In this case, the image shift amount L1X with respect to the display element 11A is, for example, a positive value and becomes L1-dxmax, and the image shift amount L2X with respect to the display element 11B is a negative value due to left-right symmetry and becomes L2+dxmax. The image shift amounts L1-dxmax and L2+dxmax are part of the congestion distance adjustment amount (corresponding to the congestion adjustment shift amounts L1 and L2 described later). Note that if the image shift amount L1X is a negative value and the image shift amount L2X is a positive value, the image shift amounts are L1-(-dxmax)=L1+dxmax and L2+(-dxmax)=L2-dxmax.

[0080] Next, the arithmetic processing unit 81a performs distortion correction by conversion shift as the second stage of correction (step S22). At this time, the correction amount that cancels out the distortion is changed according to the desired conversion shift amount. The absolute value of the conversion shift amount d1x for the display element 11A becomes equal to the maximum shift amount dxmax, and the absolute value of the conversion shift amount d2x for the display element 11B also becomes equal to the maximum shift amount dxmax. Strictly speaking, if the congestion adjustment shift amount L1 is positive, the conversion shift amount d1x is also positive, but in this case, due to left-right symmetry, the congestion adjustment shift amount L2 becomes negative, so the conversion shift amount d2x becomes negative.

[0081] Refer to Figures 14-16 for a detailed explanation of how to adjust the congestion distance, etc.

[0082] Figure 14 illustrates the original initial image IM0 captured from the user terminal 90, etc., in step S13.

[0083] FIG. 15 shows an image IM01 before performing distortion correction by conversion shift in steps S15 and S22. The right column is the image J01 being processed for the display element 11A for the right eye, corresponding to the input signal J1 in FIG. 10. The left column is the image J02 being processed for the display element 11B for the left eye, corresponding to the input signal J2 in FIG. 10. The convergence adjustment shift amounts L1 and L2 attached to each image IM01 correspond to the convergence distance LA in FIG. 5, the convergence distance Lai (L0 < Lai ≦ La1), the reference convergence distance L0, the convergence distance Lbi (Lb1 ≦ Lbi < L0), and the convergence distance LB in order from the top, and the image shift amount L1X is L1 + dxmax, 0, 0, 0, L1 - dxmax. Also, the image shift amount L2X is L2 - dxmax, 0, 0, 0, L2 + dxmax. That the image shift amounts L1X and L2X are 0 means that no image shift is performed.

[0084] In a specific example, when the convergence adjustment shift amount L1 is -50 pixels, an image shift of LX1 = L1 - (-dxmax) = -34 pixels is performed, and conversely, when it is +50 pixels, an image shift of L1 - dxmax = 34 pixels is performed. When the convergence adjustment shift amount L2 is +50 pixels, an image shift of LX2 = L2 - dxmax = +34 pixels is performed, and conversely, when it is -50 pixels, an image shift of L2 + dxmax = -34 pixels is performed. When the convergence adjustment shift amount L1 is +8 pixels, no image shift is performed.

[0085] Note that when performing image shift, the image is cut at one end, but from the viewpoint of ensuring symmetry, the image is cut at both the left and right ends. Specifically, the image to the left of the column marked with ☆ and the image to the right of the column marked with △ are cut, resulting in a narrower viewing angle.

[0086] Figure 16 shows the corrected image IM2, etc., after distortion correction by converted shift. The right column shows the display surface 11a of the display element 11A for the right eye and corresponds to the output signal K1 to the display element 11A. The left column shows the display surface 11a of the display element 11B for the left eye and corresponds to the output signal K2 to the display element 11B. The converted shift amounts d1x and d2x attached to the corrected image IM2, etc., correspond to the above-mentioned congestion distances LA, Lai, L0, Lbi, and LB, respectively from top to bottom. The converted shift amount d1x is -dxmax, L1, 0, L1, and +dxmax. The converted shift amount d2x is +dxmax, L2, 0, L2, and -dxmax.

[0087] In a specific example, if the congestion adjustment shift amount L1 is, for example, +50 pixels, a conversion shift of dxmax pixels is performed, and the set of coefficients for coordinate transformation (a1x, b1x, c1x) in that case is as shown in Figure 9 (a1x +16 ,b1x +16 ,c1 x+16 ) and if the congestion adjustment shift amount L2 is, for example, -50 pixels, a conversion shift of -dxmax pixels is performed, and the set of coefficients for coordinate transformation at that time (a2x, b2x, c2x) is, in this embodiment, assuming that there is left-right symmetry, as shown in Figure 9 (a1x -16 ,b1x -16 ,c1 x-16 This corresponds to (a1x, b1x, c1x). Also, when the congestion adjustment shift amount L1 is -50 pixels, a conversion shift of -dxmax pixels is performed, and when the congestion adjustment shift amount L2 is +50 pixels, a conversion shift of +dxmax pixels is performed. When the congestion adjustment shift amount L1 is +8 pixels, a conversion shift of +8 pixels is performed, and the coefficient set for coordinate transformation in this case (a1x, b1x, c1x) is as shown in Figure 9 (a1x +9 ,b1x +9 ,c1 x+9 ) and when the congestion adjustment shift amount L2 is -8 pixels, a conversion shift of -8 pixels is performed, and the coefficient set for coordinate transformation at that time (a2x, b2x, c2x) is, in this embodiment, assuming left-right symmetry, as shown in Figure 9 (a1x -9 ,b1x -9 ,c1 x-9This corresponds to ). Also, if the congestion adjustment shift amount L1 is -8 pixels, a conversion shift of approximately -8 pixels is performed in the same manner as above, and if the congestion adjustment shift amount L2 is +8 pixels, a conversion shift of approximately +8 pixels is performed in the same manner as above.

[0088] Figure 17 shows the virtual image CC observed by the binocular eye at various convergence distances LA, Lai, L0, Lbi, and LB. As explained with respect to Figure 15, the image-shifted virtual image CC has the image cut off at both the left and right edges.

[0089] The head-mounted display device 100 of the embodiment described above includes display elements 11A and 11B for displaying images, an optical member OE into which image light ML corresponding to the image is incident, a reflective member RE that reflects the image light ML from the optical member OE and projects a virtual image corresponding to the image, and a display control device 88 which corrects the image displayed on the display elements 11A and 11B so as to cancel out the distortion of the virtual image projected by the optical member OE and the reflective member RE. The display control device 88 adjusts the convergence distance of the virtual image by a convergence distance adjustment amount in the horizontal direction, i.e., a convergence adjustment shift amount L1 and L2, and changes the amount of distortion correction for the virtual image in conjunction with the image conversion shift amount d1x, which is at least a part of the convergence distance adjustment amount.

[0090] In the head-mounted display device 100 described above, when the display control device 88 adjusts the convergence distance of the virtual image by a convergence distance adjustment amount in the horizontal direction, it changes the amount of distortion correction for the virtual image in conjunction with the image conversion shift amount d1x, which is at least a part of the convergence distance adjustment amount, thereby suppressing the occurrence of distortion in the projected virtual image.

[0091] [Variations and other variations] Although the present invention has been described in reference to the embodiments described above, the present invention is not limited to the embodiments described above, and can be implemented in various forms without departing from the spirit thereof, for example, the following modifications are also possible.

[0092] In the virtual image display devices 100A and 100B, the imaging optical system 20 suppresses vertical distortion and generates trapezoidal distortion mainly in the horizontal direction. However, it may also generate distortion in both the vertical and horizontal directions, such as barrel distortion, or it may generate asymmetrical distortion in the up, down, left, and right directions. In this case, the display control device 88 needs to correct distortion or curvature in the x and y directions, and it needs to change the correction amount for distortion correction related to the virtual image in conjunction with the converted shift amount. At this time, when the initial image IM0 is converted to the corrected image IM1, the coordinate values ​​in the x and y directions become the intermediate positions of the pixel points or grid points. Therefore, it is desirable to calculate accurate gradation for the pixel points located around the intermediate positions vertically and horizontally on the display elements 11A and 11B from the coordinate values ​​of such intermediate positions.

[0093] In the virtual image display devices 100A and 100B, the components of the imaging optical system 20 are merely illustrative. In the imaging optical system 20, the projection lens 21 is configured with three lenses, but it may also have one, two, or four or more lenses. Furthermore, the imaging optical system 20 may be configured without a projection lens 21.

[0094] A dimming device can be attached to the exterior side of the see-through mirror 23 to control the light transmitted through the see-through mirror 23. The dimming device adjusts the transmittance electrically, for example. A mirror liquid crystal, an electronic shade, etc., can be used as the dimming device. The dimming device may also adjust the transmittance according to the ambient light intensity.

[0095] The head-mounted display device 100 may be equipped with only one of the virtual image display devices 100A or 100B. In this case, it becomes possible to adjust the display position without degrading the display state of the virtual image according to the viewpoint.

[0096] In a specific embodiment, the virtual image display device comprises a display element for displaying an image, an optical element to which image light corresponding to the image is incident, a reflective element that reflects the image light from the optical element and projects a virtual image corresponding to the image, and a control device that corrects the image to be displayed on the display element so as to cancel out the distortion of the virtual image projected by the optical element and the reflective element. The control device adjusts the convergence distance of the virtual image by a convergence distance adjustment amount in the horizontal direction, and changes the amount of distortion correction for the virtual image in conjunction with the image conversion shift amount, which is at least a part of the convergence distance adjustment amount.

[0097] In the above-described virtual image display device, when the control device adjusts the convergence distance of the virtual image using a convergence distance adjustment amount in the horizontal direction, it changes the amount of distortion correction for the virtual image in conjunction with the image conversion shift amount, which is at least a part of the convergence distance adjustment amount. This makes it possible to suppress the occurrence of distortion in the projected virtual image.

[0098] In specific aspects, the control device performs a first-stage correction when the convergence distance adjustment amount exceeds a predetermined upper limit, by applying an image shift to the display element corresponding to the difference from the upper limit, and a second-stage correction in which distortion related to the virtual image is canceled out in conjunction with the converted image shift amount corresponding to the upper limit. When the convergence distance adjustment amount is large, the amount of convergence adjustment shift that is insufficient with the converted shift can be compensated for by the image shift.

[0099] In specific aspects, the control device performs a correction to offset distortion related to the virtual image in conjunction with the converted shift amount, provided that the amount of congestion distance adjustment does not exceed a predetermined upper limit. If the amount of congestion distance adjustment is not large, the congestion distance can be adjusted using only the converted shift.

[0100] In specific aspects, the display element has an extended region for performing correction to cancel out distortion related to virtual images in conjunction with the converted shift amount, within a range where the convergence distance adjustment amount does not exceed a predetermined upper limit. In this case, even if the convergence distance is adjusted by the converted shift, it is possible to prevent image loss.

[0101] In specific aspects, the control device performs a correction to cancel out distortion related to the virtual image by performing a coordinate transformation using a transformation formula that includes the original coordinate position and coefficients, and changes the coefficients of the coordinate formula in accordance with the change in the amount of converted shift.

[0102] In specific terms, the change in the converted shift amount is on a per-pixel or per-pixel basis.

[0103] In specific aspects, the control device performs gradation adjustment at the pixel points of the display element by interpolation from the pixel points after coordinate transformation. In this case, it is possible to suppress image degradation when adjusting the congestion distance.

[0104] In specific aspects, the control device corrects distortion in a first direction corresponding to the horizontal direction, and corrects distortion in a second direction different from the first direction of the virtual image using optical and reflective elements. As a result, the control unit does not need to process distortion correction according to the second direction distortion of the virtual image, and only needs to correct distortion in one direction, thus reducing the load on the control unit for image distortion correction.

[0105] In a specific embodiment, the head-mounted display device comprises a first device equipped with the above-mentioned virtual image display device and a second device equipped with the above-mentioned virtual image display device, wherein the convergence distance adjustment amount of the first device and the convergence distance adjustment amount of the second device are the same in magnitude but opposite in direction. [Explanation of Symbols]

[0106] A1…Basic area, A2…Extended area, AA…Virtual image, AA1…Corrected image, BB…Original image, AX…Optical axis, CNT…Control device, IM0…Initial image, IM1,IM2…Corrected images, IV1,IV2,IV3,IV4…Vertex pixels, J01,J02…Image, J1,J2…Input signals, K1,K2…Output signals, L0…Reference convergence distance, LA,LB,La1,L0,Lb1…Convergence distance, L1,L2…Convergence adjustment shift amount, L1X,L2X…Image shift amount, ML… Image light, OE…Optical element, OL…Field light, PE…Pixel, PP…Pupil position, RE…Reflective element, SB…Display area, TM…Converted coordinates, PM…Pixel center coordinates, VE1,VE2,VE3,VE4…Vertex pixels, XE1,XE2…Visual axis, XE10,XE20…Reference visual axis, XEA0,XEB0…Front view visual axis, a,b,c…Coefficients, a1x,b1x,c1x,a2x,b2x,c2x…Coefficients, d1x,d2x…Conversion shift amount, ΔL,ΔW…Pixel width, dxm ax...Maximum shift amount (upper limit), θ...Convergence angle, 11A, 11B...Display element, 11a...Display surface, 12...Projection optical system, 20...Imaging optical system, 21...Projection lens, 22...Prism mirror, 23...See-through mirror, 23a...Reflective surface, 51...Case, 51a...Aperture, 61...Support plate, 70...Circuit system, 81a...Arithmetic processing unit, 81c...Data communication interface, 81m...Storage device, 83...Frame memory, 85...Ancillary circuitry, 86...Correction unit, 87...Non-volatile 88…Display control device, 8a…Coordinate transformation unit, 8b…Grayscale conversion unit, 90…User terminal, 91…User terminal circuit, 91a…Main control device, 91c…Data communication interface, 91i…User interface device, 91m…Storage device, 91t…Wireless communication device for mobile devices, 100…Head-mounted display device, 100A, 100B…Virtual image display device, 100C…Support device, 102…Display drive unit, 103…External component, EY…Eye, US…Wearer

Claims

1. A display element that displays an image, An optical member into which image light corresponding to the aforementioned image is incident, A reflective member that reflects image light from the optical element and projects a virtual image corresponding to the image, The system includes a control device that corrects the image displayed on the display element so as to cancel out the distortion of the virtual image projected by the optical element and the reflective element, When the control device adjusts the convergence distance of the virtual image by a convergence distance adjustment amount in the horizontal direction, it changes the distortion correction amount for the virtual image in conjunction with the converted shift amount of the image, which is at least a part of the convergence distance adjustment amount. Virtual image display device.

2. The control device, when the congestion distance adjustment amount exceeds a predetermined upper limit, performs a first-stage correction by applying an image shift to the display element corresponding to the difference with the upper limit, and a second-stage correction by canceling out the distortion related to the virtual image in conjunction with the converted shift amount of the image corresponding to the upper limit. The virtual image display device according to claim 1.

3. The control device, when the congestion distance adjustment amount does not exceed the predetermined upper limit, performs a correction to cancel out the distortion related to the virtual image in conjunction with the converted shift amount. The virtual image display device according to claim 2.

4. The display element has an extended region for performing a correction to cancel out the distortion related to the virtual image in conjunction with the converted shift amount, within a range where the convergence distance adjustment amount does not exceed the predetermined upper limit. The virtual image display device according to claim 2.

5. The control device performs a correction to cancel out the distortion of the virtual image by a coordinate transformation using a transformation formula that includes the original coordinate position and coefficients, and changes the coefficients of the transformation formula in accordance with the change in the converted shift amount. The virtual image display device according to claim 4.

6. The change in the converted shift amount is in units of one pixel or multiple pixels. The virtual image display device according to claim 5.

7. The control device performs gradation adjustment at the pixel points of the display element by interpolation from the pixel points after coordinate transformation. The virtual image display device according to claim 5.

8. The control device corrects the distortion in the first direction corresponding to the horizontal direction, The optical member and the reflective member correct the distortion in the second direction, which is different from the first direction of the virtual image. The virtual image display device according to claim 1.

9. A first device comprising a virtual image display device according to any one of claims 1 to 8, A second device comprising a virtual image display device according to any one of claims 1 to 8, Equipped with, The congestion distance adjustment amount of the first device and the congestion distance adjustment amount of the second device are the same in magnitude but opposite in direction. Head-mounted display device.

Images