Element image group generating device and program thereof
The elemental image group generation device in light field HMDs addresses the computational burden by pre-calculating optical paths and tracing vectors, enabling rapid generation of elemental images based on head movement, thus enhancing processing speed.
Patent Information
- Application Number
- JP2020156877
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2020-09-18
- Publication Date
- 2024-11-11
- Estimated Expiration
- 2040-09-18
AI Technical Summary
Light field HMDs require significant computational resources and time to generate elemental image groups due to the complexity of optical path calculations.
An elemental image group generation device that includes an optical path calculation means and a ray tracing calculation means, pre-calculating optical paths from display pixels to the pupil plane and reversing vectors to facilitate rapid generation of elemental images based on head movement.
The device significantly reduces the time required to generate elemental image groups by performing ray tracing processing efficiently, leveraging pre-calculated optical paths and head movement information.
Smart Images

Figure 0007580175000001 
Figure 0007580175000002 
Figure 0007580175000003
Abstract
Description
[Technical field]
[0001] The present invention relates to an element image group generating device and a program for generating an element image group to be displayed by a light field HMD. [Background technology]
[0002] In recent years, the use of virtual reality (VR) has been expanding in various fields such as education, medicine, and entertainment. The optical system of a head mounted display (HMD), a VR display device, has a simple structure that enlarges the image on the display using an eyepiece. This HMD creates a sense of three-dimensionality by displaying images with parallax on two displays, one for the left eye and one for the right eye.
[0003] One of the problems that this HMD cannot solve is eye strain caused by a mismatch between focal length and convergence distance. This problem is caused by the fact that the position where the eyes focus due to the accommodation function does not match the position where the gaze directions of both eyes intersect due to the convergence function. This problem can be solved by using a light field HMD that virtually reproduces the light source of a 3D object.
[0004] The light field HMD includes a display, a microlens array, and an eyepiece (Non-Patent Documents 1 and 2). Note that the light field HMD does not necessarily require an eyepiece, and may include only a display and a microlens array (Patent Document 1). The light field HMD displays on the display a group of elemental images for generating a light source for a 3D object.
[0005] In a light field HMD, the generation of elemental images is classified into a process of calculating the optical path from each pixel of the display through the optical elements to the pupil plane, and a process of tracing rays from the pupil plane to a 3D object placed in the virtual space and acquiring color information. [Prior art documents] [Patent documents]
[0006] [Patent Document 1] JP 2020-073988 A [Non-patent literature]
[0007] [Non-Patent Document 1] H. Huang and H. Hua, “Generalized methods and strategies for modeling and optimizing the optics of 3D head-mounted light field displays,” Opt. Express, 27(18), 25154, (2019). [Non-Patent Document 2] D. Lanman and D. Luebke, “Near-Eye Light Field Displays,” ACM Trans. Graph. 32(6), 1-10 (2013). Summary of the Invention [Problem to be solved by the invention]
[0008] However, light field HMDs have the problem that the amount of calculation required to generate element pixel groups is large and the generation takes time.
[0009] Therefore, an object of the present invention is to provide an element image group generating device and a program therefor that can generate an element image group at high speed. [Means for solving the problem]
[0010] In order to solve the above problem, the element image group generating device of the present invention is an element image group generating device that generates an element image group to be displayed on a light field HMD equipped with a display and an optical element array composed of optical elements, and is configured to include an optical path calculation means and a ray tracing calculation means.
[0011] According to this configuration, the optical path calculation means calculates in advance the optical paths of light rays that pass from each pixel of the display through the optical elements and reach a specified pupil plane, and calculates a tracing vector by inverting the directional vector of the light ray that has reached the pupil plane. The ray tracing calculation means performs ray tracing processing to obtain color information of points where the extension line of the tracing vector contacts an object in the virtual space based on head movement information of the wearer of the light field HMD, and generates a group of element images. The optical path calculation means includes an optical element correspondence means for calculating an optical element back vector of a ray passing through the center of the optical element from each pixel of the display, and a pupil plane retroreflection means for calculating the tracking vector by reversing the direction of the optical element back vector at the pupil plane. The light field HMD further includes an eyepiece. The optical path calculation means further includes an eyepiece refraction means for extending the optical element post-vector and calculating the eyepiece post-vector of a light ray that passes through the center of the optical element and is refracted by the eyepiece. The pupil plane retroreflection means calculates a tracing vector on the pupil plane by reversing the direction of the eyepiece rear vector.
[0012] In this way, the element image group generating device performs in advance calculation of the optical path from each pixel of the display through the optical element to the pupil plane. Then, based on the result of the optical path calculation, the element image group generating device performs ray tracing processing according to the head movement of the wearer of the light field HMD, thereby enabling the element image group to be generated at high speed. In other words, the element image group generating device performs optical path calculation in advance and appropriately performs ray tracing processing according to the head movement of the wearer, thereby enabling the generation time of the element image group to be shortened.
[0013] The present invention can be realized by a program for causing a computer to function as the element image group generating device described above. Effect of the Invention
[0014] According to the present invention, a group of elemental images can be generated at high speed. [Brief description of the drawings]
[0015] [Figure 1] 1 is a block diagram showing a configuration of an element image group generating device according to a first embodiment. [Diagram 2] FIG. 2 is an explanatory diagram for explaining optical path calculation in the first embodiment. [Diagram 3]FIG. 2 is an explanatory diagram for explaining a ray tracing process in the first embodiment. [Figure 4] 4 is a flowchart showing the operation of the optical path calculation device according to the first embodiment. [Diagram 5] 4 is a flowchart showing the operation of the ray tracing calculation apparatus according to the first embodiment. [Figure 6] FIG. 11 is a block diagram showing the configuration of an element image group generating device according to a second embodiment. [Figure 7] 13A and 13B are explanatory diagrams for explaining the setting of unique numbers and center position flags of element images in the second embodiment, where (a) is a front view of a display and (b) is a side view of the display. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0016] Hereinafter, each embodiment of the present invention will be described with reference to the drawings. However, each embodiment described below is intended to embody the technical idea of the present invention, and unless otherwise specified, the present invention is not limited to the following. In addition, the same symbols are used for the same means, and the description may be omitted.
[0017] (First embodiment) The configuration of an element image group generating device 1 according to the first embodiment will be described with reference to FIG. The elemental image group generating device 1 generates and displays an elemental image group to be displayed on a light field type HMD 2, and includes an optical path calculation device (optical path calculation means) 3 and a ray tracing calculation device (ray tracing calculation means) 4, as shown in Fig. 1. In this embodiment, the optical path calculation device 3 is implemented in a workstation, and the ray tracing calculation device 4 is built into the HMD 2. In addition, the HMD 2 and the optical path calculation device 3 perform wireless communication.
[0018] [HMD configuration] The configuration of the HMD 2 will be described with reference to FIGS. The HMD 2 is a typical light field type HMD, and includes a display 20, a microlens array (optical element array) 22, an eyepiece 24, and a storage unit 25, as shown in FIGS.
[0019] The display 20 is a flat display in which pixels 21 are arranged two-dimensionally, similar to a typical smartphone. The display 20 is a right-eye display 20 R and left eye display 20 L and displays a group of elemental images for the right eye and the left eye. Here, a group of elemental images for the right eye and a group of elemental images for the left eye may be displayed on the left and right sides of a single display 20, respectively.
[0020] The microlens array 22 is a two-dimensional array of microlenses (optical elements) 23. The microlens array 22 includes a microlens array 22 for the right eye. R and the microlens array 22 for the left eye. L (See FIG. 3.) For example, the microlens 23 is a tiny biconvex lens. The eyepiece lens 24 is a lens that magnifies an intermediate image formed after light rays from an elemental image displayed by the display 20 pass through the microlens array 22. The eyepiece lens 24 is a right eyepiece lens 24. R and left eyepiece 24 L (See FIG. 3.) Here, the eyepiece lens 24 is located on the side of the HMD 2 worn by the wearer.
[0021] The pupil plane α is a virtual plane on which both eyes of the person wearing the HMD 2 are positioned. Moreover, the image plane β is a virtual plane on which the image of the 3D object A is formed.
[0022] The storage means 25 is a storage device such as a memory or an SD card that stores various information necessary for generating a group of elemental images. For example, the storage means 25 stores an information table I, head movement information H, and virtual environment information K. Details of the various information stored in the storage means 25 will be described later.
[0023] [Configuration of the optical path calculation device] The optical path calculation device 3 performs optical path calculation in advance, and calculates a tracing vector V3 by inverting the directional vector of a light ray that has reached the pupil plane α, as shown in Fig. 2. Optical path calculation refers to calculating the optical path of a light ray that travels from each pixel 21 of the display 20 through the microlens 23 and the eyepiece lens 24 to reach the pupil plane α.
[0024] As shown in FIG. 1, the optical path calculation device 3 includes a microlens array correspondence means (optical element correspondence means) 30, an eyepiece lens refraction means 31, a pupil surface retroreflection means 32, and an information table generation means 33.
[0025] <Microlens array support means> The microlens array correspondence means 30 performs microlens array correspondence processing. The microlens array correspondence processing is processing for associating a pixel 21 with a microlens 23 and calculating a post-microlens vector (post-optical element vector) V1 of a light ray passing from the pixel 21 through the center of the microlens 23, as shown in Fig. 2. Note that, although only one pixel 21 is associated with the microlens 23 in Fig. 2, all pixels 21 are associated with the microlens 23. Also, although only one of the right eye or the left eye is shown in Fig. 2, both eyes have the same configuration.
[0026] Here, the microlens array correspondence means 30 receives parameters including the display pixel arrangement, the microlens array arrangement, the microlens array focal length, the microlens array pitch, and the eyepiece focal length.
[0027] The display pixel arrangement represents the arrangement (i, j) of the pixels 21 that make up the display 20. The microlens array arrangement represents the arrangement (k, l) of the microlenses 23 that constitute the microlens array 22. The microlens array focal length represents the focal length of the microlenses 23 that constitute the microlens array 22 . The microlens array pitch represents the pitch (spacing) of the microlenses 23 that constitute the microlens array 22. The eyepiece focal length represents the focal length of the eyepiece 24 .
[0028] Specifically, the microlens array correspondence means 30 calculates the vector of a light ray passing through a microlens 23 associated with the pixel 21 of the display 20, using the pixel 21 of the display 20 as a light source. As shown in FIG. 2, the starting position of the light ray is the pixel position O of each pixel 21 constituting the display 20. Dij In addition, the post-microlens vector V1 is the pixel position O Dij Then, the center position O of the microlens 23 corresponding to the pixel 21 is calculated. LAkl is a unit vector pointing to
[0029] There are several methods for associating the pixels 21 with the microlenses 23. For example, the simplest method is to associate the closest microlens 23 with each pixel 21 of the display 20. This method uses the display pixel arrangement and the microlens array arrangement from the above-mentioned parameters.
[0030] Another method is to confirm that light rays from pixels 21 of display 20 pass through microlenses 23 and enter the eyeball of the wearer, and then assign the microlenses 23. In this method, the pixel 21 located within the radius of the microlens 23 from the center is assigned, with the point at which a straight line that starts from the center of pupil plane α, is refracted by eyepiece 24, passes through the center of microlens 23, and touches the display 20 as the center. In this method, of the above-mentioned parameters, the display pixel arrangement, the microlens array arrangement, the microlens array focal length, the microlens array pitch, and the eyepiece focal length are used.
[0031] Thereafter, the microlens array correspondence means 30 outputs the display pixel layout and the post-microlens vector V1 to the eyepiece refraction means 31.
[0032] <Eyepiece lens refraction means> The eyepiece refraction means 31 receives the display pixel arrangement and the microlens rear direction vector V1 from the microlens array correspondence means 30, and performs eyepiece refraction processing.
[0033] 2, eyepiece refraction processing is a process for calculating eyepiece post-vector V2 of a light ray that passes through the center of microlens 23 and is refracted by eyepiece 24 by extending microlens post-vector V1. That is, in eyepiece refraction processing, a light ray that is emitted from pixel 21 and passes through microlens 23 is treated as incident light, and the optical path of the light ray refracted by eyepiece 24 is calculated. In this eyepiece refraction processing, the eyepiece focal length is used from the above-mentioned parameters.
[0034] The starting position of the eyepiece lens rear vector V2 is the contact position O between the microlens rear vector V1 and the eyepiece lens 24. EP Moreover, the post-eyepiece vector V2 is a unit vector pointing in the direction refracted by the eyepiece 24.
[0035] Thereafter, the eyepiece refraction means 31 outputs the eyepiece post-vector V2 to the pupil plane retroreflection means 32.
[0036] <Pupil surface retroreflection means> The pupil plane retroreflection means 32 receives the eyepiece rear vector V2 from the eyepiece refraction means 31 and performs pupil plane retroreflection processing.
[0037] Pupil plane retroreflection processing is processing for calculating a tracing vector V3 by inverting the direction of eyepiece rear vector V2 on pupil plane α, as shown in Fig. 2. That is, in pupil plane retroreflection processing, the position where a light ray (eyepiece rear vector V2) that has passed through eyepiece 24 reaches pupil plane α is calculated by determining whether it comes into contact with a plane corresponding to pupil plane α. Here, the sign of eyepiece rear vector V2 is inverted to set tracing vector V3 = (u, v, w) for performing ray tracing processing. The position where eyepiece rear vector V2 reaches pupil plane α is referred to as pupil plane position O. Eye =(x,y,z).
[0038] Then, the pupil plane retroreflection means 32 reflects the display pixel arrangement, pupil plane position O Eye And the tracing vector V3 is output to the information table generating means 33.
[0039] <Information table generation means> The information table generating means 33 performs an information table generating process. This information table generating process is carried out based on the display pixel arrangement and pupil plane position O input from the pupil plane retroreflection means 32. Eye and the tracing vector V3 as one file called information table I. Then, the information table generating means 33 writes the information table I into the storage means 25.
[0040] [Configuration of ray tracing calculation device] The ray tracing calculation device 4 performs ray tracing processing based on the wearer's head movement information H, and generates an element image group. The ray tracing processing is a process for acquiring color information of a portion where an extension line of the tracing vector V3 contacts a 3D object A in a virtual space (FIG. 3). As shown in FIG. 1, the ray tracing calculation device 4 includes a twin-eye display means 40, a head movement means 41, a ray tracing means 42, and an element image group generation means 43.
[0041] <Two-lens display means> The binocular display means 40 refers to the information table I stored in the storage means 25 and performs binocular display processing. The binocular display processing shifts the tracking vector V3 in the horizontal direction by a preset interocular distance d to obtain the tracking vectors V3 of the right eye and the left eye. 3R ,V 3L The interocular distance d represents the distance between the wearer's eyes and can be set arbitrarily. In other words, in the binocular display process, in order to replicate the information table I for both eyes from one eye, the pupil plane position O is generated according to the interocular distance d. Eye The right eye pupil plane position O EyeR1 ,O EyeR2 , the pupil plane position of the left eye O EyeL1 ,O EyeL2 , the tracking vector of the right eye V 3R1 ,V 3R2 , and the left eye tracking vector V 3L1 ,V 3L2 is shown in the figure. Thereafter, the twin eye display means 40 outputs the right eye and left eye information table I to the head movement means 41.
[0042] Before describing head movement means 41, head movement information H and virtual environment information K stored in storage means 25 will be described below. The head movement information H is information that represents the head movement of the wearer. The head movement information H includes the head position difference and the integral value of the angular velocity for each frame. Here, an acceleration sensor (not shown) provided in the HMD 2 measures the head movement information H and writes it to the storage means 25.
[0043] The virtual environment information K is information required to render the 3D object A in the virtual space. This virtual environment information K includes mesh information (vertex coordinates, normal vector, color information) of the 3D object A in the virtual space, and light source information in the virtual space (for example, the position, color, and brightness of the light source B). For example, the virtual environment information K can be a file in OBJ format. Here, it is assumed that an administrator of the element image group generating device 1 manually writes the virtual environment information K into the storage means 25.
[0044] <Head movement means> The head movement means 41 receives the information table I of the right eye and the left eye from the two-eye display means 40 and performs head movement processing. This head movement processing is to calculate the tracking vector V3 and the pupil plane position O of both eyes based on the center of gravity position of the head of the wearer. Eye This is a process of applying a head rotation matrix determined from head movement information H to the
[0045] Specifically, the head movement means 41 determines the pupil plane position O from the information table I for the right eye and the left eye. EyeR ,O EyeL and the tracking vector V 3R ,V 3L The head movement means 41 also reads out head movement information H from the storage means 25. As described above, the head movement information H includes the head position difference and the integral value of the angular velocity for each frame. Therefore, the head movement means 41 extracts the translation matrix T H From the integral of the angular velocity, the head rotation matrix R = [R x ,R y ,R z Then, the head moving means 41 calculates the pupil plane position O EyeR ,O EyeL and the tracking vector V 3R ,V 3L Apply the rotation matrix R to
[0046] Then, the head movement means 41 calculates the pupil plane position O EyeR ,O EyeL and the tracking vector V 3R ,V 3L are stored in information table I for the right and left eyes, and this information table I is output to the ray tracing means 42.
[0047] <Ray tracing method> The ray tracing means 42 calculates a tracing vector V 3R ,V 3LRay tracing is performed to obtain color information at the points where the ray comes into contact with 3D object A.
[0048] Specifically, the ray tracing means 42 determines the display pixel arrangement, pupil plane position O, and the like from the information table I input from the head movement means 41. EyeR ,O EyeL and the tracking vector V 3R ,V 3L The ray tracing means 42 also reads out the virtual environment information K from the storage means 25. As described above, the virtual environment information K includes mesh information of the 3D object A in the virtual space and light source information in the virtual space (see FIG. 3). Therefore, the ray tracing means 42 extracts the pupil plane position O EyeR ,O EyeL The tracking vector V starting from 3R ,V 3L 3, a ray tracing process is performed to determine the points of contact with the surface of the 3D object A. Furthermore, the ray tracing means 42 calculates the pixel value (e.g., RGB value) of each pixel 21 of the display 20 by allocating color information of the points of contact with the 3D object A to the display pixel arrangement. Note that in FIG. 3, the color of the 3D object A is expressed by a type of hatching.
[0049] Thereafter, the ray tracing means 42 outputs color information (pixel value) of each pixel 21 of the display 20 to the element image group generating means 43.
[0050] <Element image group generating means> The elemental image group generating means 43 receives color information of each pixel 21 of the display 20 from the ray tracing means 42 and performs elemental image group generating processing. This elemental image group generating processing is processing in which the pixel value of each pixel 21 of the display 20 acquired by the ray tracing means 42 is stored in each pixel of an elemental image to generate elemental image groups for both eyes.
[0051] After that, when the element image group generating means 43 stores pixel values for the pixels of all the element images, it outputs the element image groups for the right and left eyes to the display 20. That is, the element image group generating means 43 outputs the element image group for the right eye to the display 20 for the right eye. R and outputs the elemental images for the left eye to the left-eye display 20 L Output to.
[0052] The ray tracing process and element image group generation process are explained in the following reference documents 1 and 2, so a detailed explanation will be omitted.
[0053] Reference 1: Katayama, Conversion method from 3D model to integral stereoscopic image, NHK Giken R&D / No.128, July 2011, [online], [Retrieved July 9, 2020], Internet〈URL:https: / / www.nhk.or.jp / strl / publica / rd / rd128 / PDF / P04-10.pdf / 〉 Reference 2: Technology for generating integral 3D images from 3D models, NHK STRL R&D / No. 123, September 2010, [online], [Retrieved July 9, 2020], Internet〈URL: https: / / www.nhk.or.jp / strl / publica / rd / rd123 / PDF / P64.pdf / 〉
[0054] [Operation of the optical path calculation device] The operation of the optical path calculation device 3 will be described with reference to FIG. As shown in FIG. 4, in step S1, the microlens array correspondence means 30 associates the pixel 21 with the microlens 23, and calculates a post-microlens vector V1 of a light ray passing from the pixel 21 through the center of the microlens 23.
[0055] In step S2, the eyepiece refraction means 31 calculates a post-eyepiece vector V2 of a light ray that passes through the center of the microlens 23 and is refracted by the eyepiece 24 by extending the post-microlens vector V1. In step S3, the pupil plane retroreflection means 32 calculates a tracing vector V3 on the pupil plane α by inverting the direction of the eyepiece rear vector V2. In step S4, the information table generating means 33 generates the information table I.
[0056] [Operation of the ray tracing calculation device] The operation of the ray tracing calculation device 4 will be described with reference to FIG. As shown in FIG. 5, in step S10, the twin-eye display means 40 shifts the tracking vector V3 in the horizontal direction by a preset interocular distance d to obtain the tracking vectors V 3R ,V 3L Generate. In step S11, the head movement means 41 calculates a tracking vector V 3R ,V 3L and pupil plane position O EyeR ,O EyeL The head rotation matrix R determined from the head movement information H is applied to the
[0057] In step S12, the ray tracing means 42 calculates a tracing vector V 3R ,V 3L Ray tracing is performed to obtain color information at the points where the ray comes into contact with 3D object A. In step S13, the element image group generating means 43 stores the pixel value acquired in step S12 in each pixel of the element image, and generates an element image group for both eyes. In step S14, the element image group generating means 43 outputs the element image group generated in step S13 to the display 20.
[0058] In step S15, the element image group generating means 43 determines whether or not generation of element image groups has been completed for all frames. If the process does not end (No in step S15), the ray tracing calculation device 4 returns to the process in step S11. If completed (Yes in step S15), the ray tracing calculation device 4 ends the process.
[0059] In this manner, the ray tracing calculation device 4 generates a moving image by repeatedly performing the processes of steps S11 to S14 for each frame.
[0060] [Actions and Effects] As described above, the element image group generating device 1 according to the first embodiment performs in advance calculation of the optical path from each pixel 21 of the display 20 through the microlens 23 and the eyepiece 24 to the pupil plane α, and generates the information table I. Then, the element image group generating device 1 performs ray tracing processing according to the head movement of the wearer based on this information table I, thereby enabling the element image group generating device 1 to generate an element image group at high speed. This enables the element image group generating device 1 to significantly shorten the image generation process in the light field type HMD 2, and shorten the time to generate an element image group for each frame. Furthermore, if the HMD 2 has a similar configuration, the element image group generating device 1 can reuse the information table I and omit the optical path calculation.
[0061] Here, since the light field method is similar to the integral method, the advantages of the light field method compared to the integral method will be explained. In the case of the integral method, because the 3D images seen from various viewpoints are displayed on the display, there is no need to regenerate the elemental image group even if the viewpoint position changes, and there is no need to perform light path calculations in advance. On the other hand, in the case of the light field method, only the 3D image seen from the eyeball position is displayed on the HMD 2. For this reason, in the light field method, when the viewpoint position changes, it is necessary to generate the elemental image group according to that viewpoint position, and there is an advantage in performing light path calculations in advance.
[0062] Second embodiment With reference to FIG. 6, the configuration of an element image group generating device 1B according to the second embodiment will be described with respect to the differences from the first embodiment. The elemental image group generating device 1B performs ray tracing in advance only around the central visual field of the wearer, and generates elemental image groups partially, thereby speeding up the ray tracing process. As shown in Fig. 6, the elemental image group generating device 1B includes an optical path calculation device (optical path calculation means) 3B and a ray tracing calculation device (ray tracing means) 4B.
[0063] [HMD configuration] The configuration of the HMD 2 will be described with reference to FIG. 6, the HMD 2B includes a display 20, a microlens array (optical element array) 22, an eyepiece 24, a storage unit 25, and a line-of-sight detection device 26. The HMD 2B also includes a ray tracing calculation device 4B built-in. The configuration other than the gaze detection device 26 is similar to that of the first embodiment, and therefore a description thereof will be omitted.
[0064] The gaze detection device 26 detects the gaze direction of the wearer and outputs it to the ray tracing calculation device 4B as a gaze direction vector G. Here, the gaze detection device 26 can use a general gaze direction detection method.
[0065] [Configuration of the optical path calculation device] As shown in FIG. 6, the optical path calculation device 3B includes a microlens array correspondence means (optical element correspondence means) 30B, an eyepiece lens refraction means 31, a pupil surface retroreflection means 32, and an information table generation means 33B. Incidentally, the eyepiece lens refraction means 31 and pupil surface retroreflection means 32 are similar to those in the first embodiment, and therefore their explanation will be omitted.
[0066] <Microlens array support means> The microlens array correspondence means 30B performs a microlens array correspondence process, similarly to the first embodiment. Furthermore, when calculating the post-microlens vector V1, the microlens array correspondence means 30B assigns a unique number of the element image to each of the display pixel arrangements.LAkl The microlens array correspondence means 30B assigns the same unique number to pixels 21 that correspond to the pixels 21. In other words, pixels 21 that are assigned the same unique number belong to the same element image. Furthermore, the microlens array correspondence means 30B sets a center pixel flag to the pixel 21 that is located at the center among the pixels 21 that are assigned the same unique number.
[0067] 7(a) and (b), the microlens array correspondence means 30B assigns the unique number '1' to the 16 pixels 21 at the top left because they belong to the same element image. Furthermore, among the pixels 21 with unique number '1', the pixel 21 in the second column from the top and the second row from the left is located in the center, so the microlens array correspondence means 30B sets a center pixel flag to this pixel 21. Note that in FIG. 7, the pixel 21 to which the center pixel flag has been set is shown hatched.
[0068] Moreover, the microlens array correspondence means 30B assigns the unique number '2' to 16 pixels constituting the element image adjacent to the right. The microlens array correspondence means 30B also sets a central pixel flag for the pixel 21 located at the center among the pixels 21 with the unique number '2' (the same applies to the unique numbers '3' to '16').
[0069] Thereafter, the microlens array correspondence means 30B outputs to the eyepiece refraction means 31 the display pixel arrangement, the post-microlens vector V1, the unique number of the element image, and the center pixel flag.
[0070] <Information table generation means> The information table generating means 33B is a display pixel arrangement, a pupil plane position O Eye The information table generating means 33B generates an information table I including the unique number and the central pixel flag of the element image in addition to the tracking vector V3.
[0071] [Configuration of ray tracing calculation device] As shown in FIG. 6, the ray tracing calculation device 4B includes a twin-eye display means 40, a head movement means 41, a ray tracing means 42B, and an elemental image group generation means 43. Since the means other than the ray tracing means 42B are the same as those in the first embodiment, the description thereof will be omitted.
[0072] <Ray tracing method> The ray tracing means 42B performs ray tracing processing using a tracing vector V3 in a direction closest to the line of sight of the wearer, and obtains color information of the portion where the ray comes into contact with the 3D object A.
[0073] Specifically, the ray tracing means 42B determines the display pixel arrangement, pupil plane position O, and the like from the information table I input from the head movement means 41. EyeR ,O EyeL and the tracking vector V 3R ,V 3L The ray tracing means 42B also reads out the virtual environment information K from the storage means 25. The ray tracing means 42B also receives an input of the line of sight direction vector G from the line of sight detection device .
[0074] Next, the ray tracing means 42B compares the tracing vector V3 of the pixel 21 having the central pixel flag with the line of sight vector G, and selects the tracing vector V that is most parallel to the line of sight vector G. 3R ,V 3L Then, the ray tracing means 42B calculates the most parallel tracing vector V 3R ,V 3L Based on the unique number of the pixel 21 having the extracted unique number, the unique number of the element image included in the central visual field is extracted. Furthermore, the ray tracing means 42B extracts the tracing vector V 3R ,V 3L The ray tracing process is performed.
[0075] The size of the central visual field can be set arbitrarily. For example, the size of the central visual field is half the total visual field that can be displayed. If there is room for the rendering speed of the elemental image group, the size of the central visual field may be changed according to the speed of the calculation processing, such as by enlarging the size of the central visual field.
[0076] In the example of Figure 7, the tracking vector V 3R ,V 3L The unique number of the pixel 21 is assumed to be '6'. The size of the central visual field is assumed to be a 3×3 element image. In this case, the ray tracing means 42B extracts unique numbers '1 to 3, 5, 7, 9 to 11' of the surrounding element images based on the unique number '6'. Then, the ray tracing means 42B extracts the tracing vectors V 3R ,V 3L The ray tracing process is performed.
[0077] [Actions and Effects] As described above, the element image group generating device 1B according to the second embodiment generates only element images corresponding to the line of sight direction, using the information table I and the line of sight direction vector G. This enables the element image group generating device 1B to reduce the amount of calculation and generate an element image group at higher speed.
[0078] Although the embodiment of the present invention has been described in detail above, the present invention is not limited to this, and includes design modifications and the like within the scope of the gist of the present invention. In the above-described embodiments, the optical element array is described as the microlens array 22, but the present invention is not limited to this.
[0079] In the above-described embodiments, the optical path calculation device 3 is mounted on a workstation, and the ray tracing calculation device 4 is built into the HMD 2; however, this is not limiting. For example, both the optical path calculation device 3 and the ray tracing calculation device 4 may be mounted on a workstation. In this case, the element image group generated by the ray tracing calculation device 4 is output to the HMD 2. In addition, both the optical path calculation device 3 and the ray tracing calculation device 4 may be built into the HMD 2.
[0080] In the above-described embodiments, the HMD 2, 2B is described as having the eyepiece lens 24, but the eyepiece lens 24 may not be provided. In this case, the eyepiece refraction means 31 does not perform eyepiece refraction processing. The pupil plane retroreflection means 32 performs pupil plane retroreflection processing using the microlens rear vector V1 instead of the eyepiece rear vector V2. In this case, the pupil plane retroreflection processing calculates a tracing vector V3 on the pupil plane α, the direction of which is reversed from that of the microlens rear vector V1.
[0081] The present invention can also be realized by a program that causes hardware resources such as a CPU, memory, and hard disk of a computer to operate as the elemental image group generating device described above. These programs may be distributed via a communication line, or may be written onto a recording medium such as a CD-ROM or a flash memory and distributed. [Explanation of symbols]
[0082] 1,1B element image group generation device 2,2B HMD 20 Display 21 pixels 22 Microlens array (optical element array) 23 Microlens (optical element) 24 Eyepiece 25 Memory means 26 Eye gaze detection device 3,3B Optical path calculation device (light path calculation means) 30, 30B Microlens array corresponding means (optical element corresponding means) 31 Eyepiece refracting means 32 pupil surface retroreflection means 33, 33B Information table generating means 4,4B Ray tracing calculation device (ray tracing calculation means) 40 Binocular display means 41 Head movement means 42,42B Ray tracing means 43 Element image group generation means
Claims
1. An element image group generating device for generating an element image group to be displayed on a light field HMD including a display and an optical element array, an optical path calculation means for calculating an optical path of a light ray passing through an optical element from each pixel of the display and reaching a predetermined pupil plane in advance, and calculating a tracing vector obtained by inverting the direction vector of the light ray that has reached the pupil plane; a ray tracing calculation means for performing a ray tracing process to obtain color information of a portion where an extension line of the tracing vector contacts an object in a virtual space based on head movement information of a wearer of the light field HMD, and generating the element image group; Equipped with The optical path calculation means an optical element correspondence means for calculating a post-optical element vector of a ray passing through a center of the optical element from each pixel of the display; A pupil plane retroreflection means for calculating the tracking vector by inverting the direction of the optical element post-vector on the pupil plane; Equipped with The light field HMD further includes an eyepiece, the optical path calculation means further includes an eyepiece refraction means for extending the optical element post vector and calculating an eyepiece post vector of a light ray passing through a center of the optical element and refracted by the eyepiece; The elemental image group generating device, wherein the pupil plane retroreflection means calculates the tracking vector by reversing the direction of the eyepiece rear vector on the pupil plane.
2. The ray tracing calculation means a binocular display means for calculating the tracking vectors of both eyes by shifting the tracking vectors in a horizontal direction by a preset interocular distance; a head movement means for applying a head rotation matrix determined from the head movement information to the tracking vectors of both eyes based on a center of gravity position of the head of the wearer; a ray tracing means for performing the ray tracing process using the tracking vector of both eyes to which the rotation matrix has been applied, and acquiring color information of a portion where the ray comes into contact with the object; an element image group generating means for storing color information acquired by the ray tracing means in each pixel of an element image and generating the element image group; 2. The element image group generating device according to claim 1, further comprising:
3. The element image group generating device according to claim 2, characterized in that the ray tracing means performs the ray tracing process using the tracing vector in a direction closest to the line of sight of the wearer, and acquires the color information of the point where the ray contacts the object.
4. A program for causing a computer to function as the element image group generating device according to any one of claims 1 to 3.
Citation Information
Patent Citations
Image processing apparatus and image processing program
JP2019149777A
Wearable 3D augmented reality display
JP2020024438A
Three-dimensional augmented reality display comprising variable focus and / or object recognition
JP2020073988A
Head-Mounted Light Field Display Using Integral Imaging and Relay Optics
JP2020510241A
