Element image generating device and program thereof

Abstract

To provide an element image generating device capable of generating element images at high speed.SOLUTION: An element image generating device 3 includes a viewpoint position calculation unit 31 that calculates the viewpoint position of an observer, a movement speed calculation unit 32 which calculates the movement speed of the viewpoint position, a pixel belonging determination unit 33 that determines that each pixel of display means included in the projection lens area belongs to an element lens corresponding to the projection lens area, ray vector calculation unit 34 that calculates a vector of a ray connecting the center of the element lens and the center of each pixel belonging to the element lens, a ray update determination unit 35 that determines to update color information of a ray passing through the eyeball peripheral area, and a ray update unit 36 that updates the color information of the ray in accordance with the determination result of the ray update determination unit 35.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to an element image generating device and a program therefor. [Background technology]

[0002] To realize interactive 3D images with a high sense of realism and immersion, it is necessary to generate and display high-quality 3D images in real time. High quality in 3D images refers to high resolution, high frame rate, wide depth reproduction range, and wide viewing area. The integral method is one of the 3D image methods based on the principle of integral photography (Non-Patent Documents 1, 2). In this integral method, elemental images are displayed on a 2D display, and the viewer can view 3D images with horizontal and vertical parallax by viewing them through a lens array arranged on the surface side of the display means.

[0003] A viewpoint-tracking integral method has been proposed, which uses a camera to measure the viewpoint position of one observer and displays a 3D image at that viewpoint position (Non-Patent Document 3). Viewpoint tracking narrows the viewing zone (optical viewing zone) determined by the optical configuration of the 3D display, increasing the light density, thereby expanding the depth reproduction range, and dynamically controls the optical viewing zone, thereby expanding the viewing zone. In other words, viewpoint tracking can improve the quality of 3D images.

[0004] The advantage of the integral method is that it can reproduce a high density group of light rays, and the elemental images are composed of a high density group of light rays when the subject is viewed from various directions. A method has been proposed in which a 3D model is photographed with multiple virtual cameras placed in a virtual space, and elemental images are generated by rearranging the images (Non-Patent Document 4). In this method, the number of virtual cameras is proportional to the number of viewpoints, so the amount of calculation increases and the frame rate decreases as the number of virtual cameras increases. In addition, in this method, since a virtual viewpoint position is set, approximate light rays are reproduced, rather than the original light rays determined by the lens array and 2D display.

[0005] Therefore, a method for generating elemental images based on the ray tracing method described in Non-Patent Document 5 has also been proposed (Non-Patent Document 6). Compared to the method using a virtual camera, this method does not require the placement of a virtual camera, so a high frame rate can be maintained even when there are a large number of viewpoints. In addition, the use of ray tracing makes it possible to reproduce the original light rays determined by the lens array and two-dimensional display. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] G. Lippmann, “Epreuves, reversibles donnant la sensation du relief,” J. Phys., 7, 1, pp.821-825(1908) [Non-Patent Document 2] Takagi, “3D Images and Flat Panel 3D Display Technology Optics,” Optics, Vol. 35, No. 8, pp. 400-409, (2006) [Non-Patent Document 3] N. Okaichi, H. Sasaki, M. Kano, J. Arai, M. Kawakita, and T. Naemura, “Design of optical viewing zone suitable for eye-tracking integral 3D display,” OSA Contin., 4, 5, pp.1415-1429 (2021) [Non-Patent Document 4] K. Yanaka, “Integral photography using hexagonal fly's eye lens and fractional view,” Proc. SPIE 6803, 68031K (2008) [Non-Patent Document 5] Computer Graphics (Revised New Edition), CG-ARTS Association, 2015 [Non-Patent Document 6] Shujun Xing, Xinzhu Sang, Xunbo Yu, Chen Duo, Bo Pang, Xin Gao, Shenwu Yang, YanXin Guan, Binbin Yan, Jinhui Yuan, and Kuiru Wang, "High-efficient computer-generated integral imaging based on the backward ray-tracing technique and optical reconstruction," Opt. Express 25, pp.330-338 (2017) Summary of the Invention [Problem to be solved by the invention]

[0007] We study a method to generate elemental images with high quality and high frame rate by combining viewpoint tracking and ray tracing. However, the viewpoint tracking integral method has a visibility problem caused by latency. This latency is the time required from when the camera starts capturing the observer's image to when the 3D image is completely displayed, and is the total time of the viewpoint position calculation process and the generation of elemental images. If this latency is large, when an observer views a 3D image while moving the viewpoint position quickly, the observer will observe side lobes beyond the designed optical viewing area, which will deteriorate visibility.

[0008] SUMMARY OF THE PRESENT EMBODIMENTS Accordingly, an object of the present invention is to provide an element image generating device and a program therefor that can generate element images at high speed. [Means for solving the problem]

[0009] In order to solve the above problems, the element image generating device of the present invention is an element image generating device that uses ray tracing and viewpoint tracking to generate an element image to be displayed on a three-dimensional display equipped with a display means and a lens array, and is configured to include a viewpoint position calculation unit, a movement speed calculation unit, a pixel belonging determination unit, a ray vector calculation unit, a ray update determination unit, and a ray update unit.

[0010] According to this configuration, the viewpoint position calculation section receives an image of the viewer's face and calculates the viewpoint position of the viewer from the input face image. The movement speed calculation unit calculates the movement speed of the viewpoint position from the change in the viewpoint position calculated by the viewpoint position calculation unit. The pixel belonging determination unit obtains a projection lens area in which each element lens of the lens array is projected from the viewpoint position onto the display means, and determines that each pixel of the display means included in the projection lens area belongs to the element lens corresponding to the projection lens area.

[0011] The light vector calculation section calculates light vectors connecting the center of the element lens and the center of each pixel belonging to the element lens. The light ray update determination unit sets an eyeball peripheral area that is wide when the moving speed of the viewpoint position is fast and narrow when the moving speed of the viewpoint position is slow, and determines to update color information of the light ray passing through the set eyeball peripheral area. The light ray update unit updates the color information of the light ray according to the determination result of the light ray update determination unit.

[0012] A light ray that passes through the eyeball peripheral region is likely to be observed by an observer, whereas a light ray that does not pass through the eyeball peripheral region is unlikely to be observed by an observer. Therefore, the element image generating device updates color information of light rays that are likely to be observed by an observer every frame, while reducing the frequency of updating color information of light rays that are unlikely to be observed by an observer. This reduces the load of the color information update process on the element image generating device, enabling it to generate element images at high speed.

[0013] The present invention can also be realized by a program for causing a computer to function as the element image generating device described above. Effect of the Invention

[0014] According to the present invention, element images can be generated at high speed. [Brief description of the drawings]

[0015] [Figure 1] 1 is a schematic configuration diagram of an element image generating system according to an embodiment. [Diagram 2] 1A to 1C are explanatory diagrams illustrating an optical viewing area of ​​a three-dimensional display in an embodiment. [Diagram 3] 1A to 1D are explanatory diagrams for explaining the width of an optical viewing zone in an embodiment. [Figure 4] 6A and 6B are diagrams illustrating updating of color information according to the moving speed of the viewpoint position in an embodiment. [Diagram 5] 1 is a block diagram showing a configuration of an element image generating device according to an embodiment. [Figure 6] 1A and 1B are diagrams illustrating the corresponding relationship between pixels and lens elements in viewpoint tracking in an embodiment. [Figure 7] 1A to 1D are explanatory diagrams for explaining how to determine the element lens to which a pixel belongs in an embodiment. [Figure 8] FIG. 11 is an explanatory diagram for explaining updating of color information of a light ray in the embodiment. [Figure 9] 4 is an explanatory diagram illustrating a light ray for which color information is to be updated in the embodiment. FIG. [Figure 10] 11 is a graph showing an example of a function that determines the radius of the eyeball peripheral region in an embodiment. [Figure 11] 11 is a graph showing an example of a function that determines the radius of the eyeball peripheral region in an embodiment. [Figure 12] 11 is a graph showing an example of a function that determines the radius of the eyeball peripheral region in an embodiment. [Figure 13] 4 is a flowchart showing an operation of the element image generating device according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] (Embodiment) Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, each embodiment described below is intended to embody the technical idea of ​​the present invention, and unless otherwise specified, the present invention is not limited to the following. In addition, the same symbols are used for the same means, and descriptions thereof may be omitted.

[0017] [Outline of element image generation system] An overview of an element image generating system 1 according to an embodiment will be described with reference to FIG. The element image generating system 1 generates and displays element images, and includes a photographing camera 2, an element image generating device 3, and a three-dimensional display 4.

[0018] Here, the display method of the 3D image is assumed to be the integral method in the elemental image generating system 1. Note that the display method of the 3D image in the elemental image generating system 1 is not limited to the integral method, and may be any light reproduction method such as the lenticular method, the line light source array method, or the point line light source array method.

[0019] The photographing camera 2 is a camera that photographs an image of the observer's face in order to calculate the viewpoint position of the observer. For example, the photographing camera 2 is a general camera disposed above a three-dimensional display 4 described later. Although one camera is illustrated as the photographing camera 2 in FIG. 1, the photographing camera 2 may be a stereo camera.

[0020] The element image generating device 3 generates element images to be displayed on the three-dimensional display 4 by using a ray tracing method and viewpoint tracking (described in detail later). Then, the element image generating device 3 outputs the generated element images to the three-dimensional display 4.

[0021] The three-dimensional display 4 is a display that displays the element images generated by the element image generating device 3. In this embodiment, the three-dimensional display 4 is an integral type display including a display means 40 and a lens array 42 (see FIG. 6). The display means 40 includes pixels 41 arranged two-dimensionally. For example, the display means 40 is a flat panel display such as a liquid crystal display or an organic EL display. The lens array 42 includes element lenses 43 arranged in an array. For example, the element lenses 43 are biconvex microlenses.

[0022] Since the 3D display 4 has motion parallax, the viewer can see an image according to the viewing direction of the 3D display 4. When considering actual long-term viewing of television or movies, the viewer is likely to choose between viewing with a stationary viewpoint and viewing while moving the viewpoint depending on the scene.

[0023] To obtain binocular parallax, the optical viewing zone of the three-dimensional display 4 must be formed so as to encompass both eyes of the observer. The shape of the optical viewing zone of the three-dimensional display 4 is determined according to the shapes and arrangement of the element lenses 43 and element images. FIG. 2(a) illustrates a state in which the square element lenses 43 and element images 91 overlap. In this case, the shape of the optical viewing zone 90 is such that the square element lenses 43 and element images 91 overlap each other. R , left eye E L ) is formed into a square shape so as to encompass both eyes E. FIG. 2(b) illustrates a state in which the honeycomb element lenses 43 and element images 91 are overlapped. In this case, the shape of the optical viewing zone 90 is formed into a honeycomb shape so as to encompass both eyes E. FIG. 2(c) illustrates an example in which the element lenses 43 are rotated 45 degrees with respect to the element images 91 so as to widen the viewing zone. In this case, the shape of the optical viewing zone 90 is formed into a rectangle so as to encompass both eyes E.

[0024] When the wide optical viewing zone 90 shown in Fig. 3(b) is narrowed as shown in Fig. 3(a), the resolution and depth reproduction range of the 3D image are improved. On the other hand, when the wide optical viewing zone 90 shown in Fig. 3(d) is narrowed as shown in Fig. 3(c), the effect of latency causes both eyes E to move out of the optical viewing zone 90, increasing the possibility of observing side lobes. Note that in Fig. 3(c) and Fig. 3(d), both eyes E before movement are illustrated lightly.

[0025] Taking into consideration the movement of the observer, the optical viewing zone 90 should be a size with some margin. When the observer is stationary, the light rays observed within the optical viewing zone 90 are only a small part of the periphery of both eyes E. On the other hand, when the observer is changing his / her posture, the optical viewing zone 90 always tracks the viewpoint position with a delay due to latency, so there is a possibility that the observer may observe all light rays within the optical viewing zone 90.

[0026] Based on this concept, the light rays for which color information is updated are selected within the optical viewing zone 90 according to the moving speed of the viewpoint position, thereby speeding up the generation process of elemental images. Specifically, as shown in FIG. 4(a), when the observer's eyes E are stationary, the color information of the light rays passing near the eyes E is updated, so the number of light rays for which color information is updated is suppressed, and elemental images can be generated at high speed. Also, as shown in FIG. 4(b), when the observer's eyes E are moving, the color information of the light rays passing at a location away from the eyes E is also updated, so the margin of the optical viewing zone can be utilized to the maximum extent, and the observer can be prevented from observing side lobes. In addition, in FIG. 4(a) and FIG. 4(b), the range in which the color information of the light rays is updated within the optical viewing zone 90 is illustrated by dots.

[0027] [Configuration of element image generating device] The configuration of the element image generating device 3 will be described with reference to FIG. As shown in Figure 5, the element image generating device 3 includes a virtual display arrangement unit 30, a viewpoint position calculation unit 31, a movement speed calculation unit 32, a pixel belonging determination unit 33, a light vector calculation unit 34, a light ray update determination unit 35, and a light ray update unit 36.

[0028] The virtual display arrangement unit 30 receives setting values ​​related to the specifications of the three-dimensional display 4 and virtually arranges the three-dimensional display 4 based on the input setting values. The virtually arranged three-dimensional display 4 may be referred to as a virtual display. For example, the setting values ​​include the specifications of the display means 40 (pixel pitch, number of pixels), the specifications of the lens array 42 (lens pitch, focal length, lens structure), and the position and rotation angle of the lens array 42 relative to the display means 40.

[0029] The viewpoint position calculation unit 31 receives an image of the observer's face and calculates the viewpoint position (both eyes E) of the observer from the input face image. For example, the viewpoint position calculation unit 31 detects two-dimensional coordinates of facial features such as the eyes, nose, and mouth from the face image using a method described in Reference Document 1. Then, the viewpoint position calculation unit 31 estimates three-dimensional coordinates of the viewpoint position from the two-dimensional coordinates of the facial features using a method described in Non-Patent Document 3.

[0030] Reference 1: “MediaPipe”, [online], [searched July 25, 2023], Internet<URL:https: / / developers.google.com / mediapipe>

[0031] When the photographing camera 2 is a stereo camera, the viewpoint position calculation section 31 can estimate the three-dimensional coordinates of the viewpoint position from the two-dimensional coordinates of both eyes and depth information, so there is no need to detect the nose and mouth.

[0032] Here, the three-dimensional coordinates of the viewpoint position are described in the camera coordinate system. Therefore, the viewpoint position calculation unit 31 measures the positional relationship (deviation) between the shooting camera 2 and the three-dimensional display 4 in advance, and converts the viewpoint position from the camera coordinate system to the display coordinate system based on the positional relationship. The display coordinate system is a coordinate system that has the screen center of the three-dimensional display 4 as its origin. Thereafter, the viewpoint position calculation unit 31 outputs the calculated viewpoint position to the movement speed calculation unit 32 and the pixel belonging determination unit 33.

[0033] The moving speed calculation unit 32 calculates the moving speed of the viewpoint position from the change in the viewpoint position calculated by the viewpoint position calculation unit 31. That is, the moving speed is calculated from the amount of change per unit time in the viewpoint position input from the viewpoint position calculation unit 31. Thereafter, the moving speed calculation unit 32 outputs the viewpoint position and the moving speed of the viewpoint position to the light ray update determination unit 35.

[0034] The pixel belonging determination unit 33 obtains a projection lens area in which each element lens 43 of the lens array 42 is projected onto the display means from the viewpoint position, and determines that each pixel 41 of the display means 40 included in the projection lens area belongs to the element lens 43 corresponding to the projection lens area. As will be described below, the pixel belonging determination unit 33 determines which element lens 43 of the lens array 42 each pixel 41 of the display means 40 belongs to.

[0035] Here, the pixel belonging determination unit 33 determines that all pixels 41 belong to one of the element lenses 43 constituting the lens array 42. The correspondence between the pixels 41 and the element lenses 43 is determined according to the viewpoint position, and the viewpoint position changes for each frame due to viewpoint tracking. Therefore, the pixel belonging determination unit 33 determines the correspondence between the pixels 41 and the element lenses 43 for each frame. As shown in FIG. 6(a), consider a case where the observer's eyes E are located at the center in front of the three-dimensional display 4. In this case, the pixels 41 included in the area (projection lens area) where a straight line passing through one element lens 43 from the viewpoint position intersects with the display surface of the display means 40 belongs to that element lens 43. As shown in FIG. 6(b), when the eyes E move due to viewpoint tracking, the correspondence between the viewpoint position, the pixels 41, and the element lenses 43 changes, and the projection lens area also moves.

[0036] FIG. 7(a) shows the pixel center x of pixel 41. D 7(b) shows the lens center x of the element lens 43. LIn FIG. 7, dots of different shades are used to make it easier to understand the correspondence between the pixels 41 and the lens elements 43. In FIG. 7(c), some pixels 41 overlap with dots of different shades, and there are pixels 41 that span multiple projection lens regions. In this case, as shown in FIG. 7(d), the pixel affiliation determination unit 33 determines the pixel center X D It is determined that the pixel 41 belongs to the element lens 43 that corresponds to the projection lens region that includes the pixel 41. Here, the method of determining the element lens 43 to which the pixel 41 belongs differs depending on the shape and arrangement of the element lens 43 and the element image, so this will be explained in order.

[0037] <Example 1: Square> As shown in FIG. 2(a), a case where the lens elements 43 and the image elements 91 have a square shape will be described as a first example. The pixel belonging determination unit 33 determines pixel center X D The position of the pixel 41 is divided by the lens pitch of the element lenses 43 projected onto the display means 40 (which is slightly larger than the pitch of the element lenses 43 indicated by the set value). The pixel belonging determination unit 33 then converts this divided value into an integer, thereby determining which element lens 43 the pixel 41 belongs to. Note that this integer value indicates the number of the element lens 43 the pixel 41 corresponds to in the vertical and horizontal directions.

[0038] <Second example: Rotation> As a second example, a case where the lens elements 43 are rotated with respect to the image elements 91 as shown in FIG. 2(c) will be described. In this case, the element lenses 43 are shifted by half a lens in the horizontal direction for each row. The pixel belonging determination unit 33 performs the same process as in the first example, and then determines which element lens 43 the pixel 41 belongs to, taking into account the shift of the element lenses 43. For example, the pixel belonging determination unit 33 determines the pixel center X D To this position, add half the value of the lens pitch of the element lenses 43 projected on the display means 40, divide the result by the lens pitch of the element lenses 43 projected on the display means 40, and convert the result into an integer. Note that for the even-numbered rows, the same process as in the first example can be performed.

[0039] <Example 3: Honeycomb> As shown in FIG. 2(b), a case where the element lenses 43 have an element image 91 in a honeycomb shape will be described as a third example. In this case, the pixel affiliation determination unit 33 performs the same process as in the second example, and then calculates the pixel center x D and the lens center x of each element lens 43 arranged around the pixel 41. L In the example of FIG. 2B, the pixel belonging determination unit 33 calculates the distance between the pixel center x D and the lens center x of the seven element lens 43 L Then, the pixel belonging determination unit 33 determines that the pixel 41 belongs to the lens element 43 for which the distance is the smallest.

[0040] The pixel affiliation determining section 33 outputs, as a determination result, information indicating the correspondence relationship between the pixel 41 and the lens element 43 to the light vector calculating section 34. Note that this determination result indicates the lens element 43 to which the pixel 41 belongs.

[0041] The light vector calculation unit 34 calculates a light vector connecting the center of the element lens 43 and the center of each pixel 41 belonging to the element lens 43. This light vector is calculated based on the lens position x L and the pixel position x of the pixel 41 belonging to the element lens 43. D The straight line connecting these points can be defined as the ray vector (see Figure 8).

[0042] As shown in FIG. 8, the center position x of a certain lens element 43 L =[x L ,y L ,z L ], and the center position x D =[x D ,y D ,z D In this case, the direction vector e of the light vector is expressed by the following equation (1). Note that normalize(·) is a function that normalizes a vector.

[0043]

number

[0044] A light vector x=[x, y, z] is expressed by the following formula (2), where s is a scalar scale coefficient and can be set arbitrarily.

[0045]

number

[0046] The light vector calculation unit 34 uses the above-mentioned equations (1) and (2) to calculate the light vector x for each pixel 41. Thereafter, the light vector calculation unit 34 outputs the calculated light vector x to the light update determination unit 35.

[0047] The light ray update determination unit 35 sets an eyeball peripheral area that is wider when the moving speed of the viewpoint position is fast and narrower when the moving speed of the viewpoint position is slow, and determines to update the color information of the light rays that pass through the set eyeball peripheral area.

[0048] The light ray update determination unit 35 determines which light ray's color information should be updated in that frame using the viewpoint position and the moving speed. That is, the light ray update determination unit 35 updates the color information of only the light ray passing through the eyeball peripheral region for each frame, and does not update the color information of the other light rays for each frame, thereby improving the frame rate. However, if the color information of a light ray is not updated for a long period of time, the color information of the light ray (pixel) may deviate significantly from the original color. For this reason, the light ray update determination unit 35 determines to update the color information of all light rays at a preset frame interval. The frame interval at which the color information of all light rays is updated can be set arbitrarily (for example, 10 frame intervals). Furthermore, the light ray update determination unit 35 may update the color information of all light rays at a preset frame interval after the color information of the light rays was last updated.

[0049] The idea of ​​updating the color information of a light ray will now be described. As shown in Figure 9, a ray of light x passing near both eyes E n Since it is highly likely that the observer will observe the color, it is determined that the color information should be updated. f Since the probability that the observer will not observe the color information is low, it is determined that the color information should not be updated. In other words, the region for determining whether or not to update the color information is called the eyeball peripheral region 92. n The color information of the ray x that does not pass through the peripheral region 92 is updated. f The color information of the right eye E is not updated. R and left eye E L Since each of the left and right eyes has an eyeball peripheral region 92, updating of color information is determined for each of the left and right eyeball peripheral regions 92.

[0050] In this embodiment, the eyeball peripheral region 92 is a spherical region, and the viewpoint position is the center position. The radius r of the eyeball peripheral region 92 is expressed by the following formula (3).

[0051]

number

[0052] Here, f(v) may be an increasing function in which the radius r increases according to the moving speed v of the viewpoint position. The function f(v) may be either a linear function or a nonlinear function, and can be manually set by the user of the element image generating system 1.

[0053] When the moving speed v of the viewpoint position is fast, it is highly likely that light rays will be observed over a wide range, so the eyeball peripheral region 92 is enlarged and the color information of the light rays is updated over a wide range. On the other hand, when the moving speed v of the viewpoint position is slow and the viewpoint position is stationary, it is highly likely that light rays will be observed over a narrow range, so the eyeball peripheral region 92 is made small and the color information of the light rays is updated over a narrow range. When the number of light rays to be updated is reduced, the processing time for generating elemental images is shortened and the frame rate is increased.

[0054] For example, the function f(v) can be defined by the following equation (4): L represents the latency and R represents the minimum radius. L and the minimum radius R are set to arbitrary values ​​in advance. For example, the minimum radius R is set to 20 mm, which is the radius of a sphere that is assumed to be an eyeball. Note that "if" is a conditional statement that returns the value of the previous part if the conditional expression of the latter part is true.

[0055]

number

[0056] As shown in Figure 10, the function f(v) in equation (4) has a latency of T L When the product of the moving speed v and the moving speed v is equal to or smaller than the minimum radius R, the radius r of the eyeball peripheral region 92 becomes the minimum radius R. In addition, in the function f(v) of the formula (4), the latency T L If the product of the moving speed v and the minimum radius R exceeds the minimum radius R, the radius r of the eyeball peripheral region 92 becomes the product.

[0057] The function f(v) may be defined by the following equation (5): Min represents the minimum radius, and R Max represents the maximum radius, and v Threshold represents the threshold value. The minimum radius R Min , maximum radius R Max and threshold v Threshold is set in advance to an arbitrary value.

[0058]

number

[0059] As shown in FIG. 11, in the function f(v) of the formula (5), when the viewpoint position is almost stationary, the radius r of the eyeball peripheral region 92 becomes the minimum radius R Min In addition, in the function f(v) of the formula (5), when the viewpoint position moves to a certain extent, the radius r of the eyeball peripheral region 92 becomes the maximum radius R Max It becomes.

[0060] The function f(v) may be defined by the following formula (6). Here, the coefficient α is set to an arbitrary value in advance. As shown in FIG. 12, the function f(v) in formula (6) is a nonlinear increasing function (quadratic function).

[0061]

number

[0062] Thereafter, the light ray update determination unit 35 outputs the determination result as to whether or not to update the color information of the light ray to the light ray update unit 36 ​​.

[0063] The light ray update unit 36 ​​updates the color information of the light ray according to the determination result of the light ray update determination unit 35. Here, the light ray update unit 36 ​​acquires the color of the intersection point located closest to the viewpoint position among the intersection points between the light ray vector and the subject as the color information of the light ray. In addition, the light ray update unit 36 ​​updates the color information of the light ray determined to be updated by the light ray update determination unit 35, so that the pixel 41 that is likely to be observed by the observer becomes the new color. On the other hand, the light ray update unit 36 ​​holds the color information of the light ray determined not to be updated by the light ray update determination unit 35 as it is in the previous frame, so that the pixel 41 that is unlikely to be observed by the observer holds the same color. As shown in FIG. 8, the color information of the subject is reflected in the elemental images, and a three-dimensional image T of the subject is reproduced.

[0064] [Operation of element image generating device] The operation of the element image generating device 3 will be described with reference to FIG. As shown in FIG. 13, setting values ​​relating to the specifications of the three-dimensional display 4 are input to the virtual display layout unit 30 in step S1. In step S2, the virtual display arrangement unit 30 virtually arranges the three-dimensional display 4 based on the setting values ​​input in step S1.

[0065] In step S3, the viewpoint position calculation section 31 calculates the viewpoint position of the observer from the face image of the observer. In step S4, the movement speed calculation unit 32 calculates the movement speed of the viewpoint position from the change in the viewpoint position calculated in step S3.

[0066] In step S5, pixel belonging determination unit 33 obtains a projection lens area in which each element lens 43 of lens array 42 is projected onto the display means from the viewpoint position. Then, pixel belonging determination unit 33 determines that each pixel 41 of display means 40 included in the projection lens area belongs to the element lens 43 corresponding to the projection lens area. In step S 6 , the light vector calculation unit 34 calculates the light vector connecting the center of the lens element 43 and the center of each pixel 41 belonging to the lens element 43 .

[0067] In step S7, the light ray update determination unit 35 sets an eyeball peripheral region that is wide when the moving speed of the viewpoint position is fast and narrow when the moving speed of the viewpoint position is slow. Then, the light ray update determination unit 35 determines to update color information of the light ray passing through the set eyeball peripheral region. In step S8, the light ray update unit 36 ​​updates the color information of the light ray in accordance with the result of the determination in step S7.

[0068] [Actions and Effects] A light ray that passes through the eyeball peripheral region is likely to be observed by the observer, whereas a light ray that does not pass through the eyeball peripheral region is unlikely to be observed by the observer. Therefore, the element image generating device 3 updates the color information of the light ray that is likely to be observed by the observer every frame, while reducing the update frequency of the color information of the light ray that is unlikely to be observed by the observer. This reduces the load of the color information update process on the element image generating device 3, enabling it to generate element images at high speed.

[0069] Furthermore, since the elemental image generating device 3 uses a ray tracing method, the frame rate does not depend on the number of viewpoints that compose the elemental images, so the frame rate does not decrease even if there are a large number of viewpoints. Furthermore, the elemental image generating device 3 can reproduce the original light rays determined by the lens array 42 and the two-dimensional display means 40. Furthermore, since the elemental image generating device 3 supports viewpoint tracking, the depth reproduction range and viewing zone can be expanded.

[0070] Although the embodiment has been described in detail above, the present invention is not limited to the above-described embodiment, and includes design modifications and the like within the scope of the present invention.

[0071] In the above embodiment, the radius of the eyeball peripheral region is the same for the left and right eyes without distinguishing between the left and right eyes, but the present invention is not limited to this. In other words, the radius of the eyeball peripheral region may be set to a value according to the speed of each of the left and right eyes (the radius of the eyeball peripheral region may differ between the left and right eyes).

[0072] In the above embodiment, the element image generating device is described as an independent piece of hardware, but the present invention is not limited to this. For example, the present invention can be realized by a program for causing hardware resources such as a CPU, memory, and hard disk of a computer to function as the element image generating device. This program may be distributed via a communication line, or may be written on a recording medium such as a CD-ROM or a flash memory and distributed. [Explanation of symbols]

[0073] 1. Elemental image generation system 2. Camera 3. Element Image Generator 4. 3D Display 30 Virtual display layout section 31 Viewpoint position calculation unit 32 Movement speed calculation section 33 Pixel Affiliation Determination Unit 34 Light vector calculation section 35 Ray update determination section 36 Ray update section 40 Display means 41 pixels 42 Lens Array 43 element lens

Claims

1. An element image generating device that generates an element image to be displayed on a three-dimensional display having a display means and a lens array by using a ray tracing method and viewpoint tracking, a viewpoint position calculation unit that receives a face image of an observer and calculates a viewpoint position of the observer from the input face image; a movement speed calculation unit that calculates a movement speed of the viewpoint position from the change in the viewpoint position calculated by the viewpoint position calculation unit; a pixel belonging determination unit that obtains a projection lens area in which each element lens of the lens array is projected from the viewpoint position onto the display means, and determines that each pixel of the display means included in the projection lens area belongs to an element lens corresponding to the projection lens area; a light ray vector calculation unit that calculates a vector of a light ray connecting a center of the element lens and a center of each pixel belonging to the element lens; a light ray update determination unit that sets an eyeball peripheral area that is wide when the moving speed of the viewpoint position is fast and that is narrow when the moving speed of the viewpoint position is slow, and determines to update color information of a light ray that passes through the set eyeball peripheral area; a light ray update unit that updates color information of the light ray in accordance with a result of the determination by the light ray update determination unit; An element image generating device comprising:

2. 2. The elemental image generating device according to claim 1, wherein the light ray update determining unit determines that color information of all the light rays is to be updated at a preset frame interval.

3. 2. The elemental image generating device according to claim 1, wherein the eyeball peripheral region is a spherical region.

4. 4. The elemental image generating device according to claim 3, wherein the eyeball peripheral region is expressed by a function whose radius increases according to the moving speed of the viewpoint position.

5. The eyeball peripheral region is determined based on the moving speed v of the viewpoint position and the preset latency T Ｌ , and using the following formula including a preset minimum radius R: [0010] 4. The elemental image generating device according to claim 3, wherein a radius r is determined.

6. The eyeball peripheral region is determined by the movement speed v of the viewpoint position and a preset minimum radius R Ｍｉｎ , a preset maximum radius R Ｍａｘ , and a preset threshold value v Ｔｈｒｅｓｈｏｌｄ Using the following formula, [0025] 4. The elemental image generating device according to claim 3, wherein a radius r is determined.

7. A program for causing a computer to function as the element image generating device according to any one of claims 1 to 6.

Images