Stereoscopic image generation device and its program

Abstract

To provide a three-dimensional video generation device which can reduce a sense of discomfort due to a size of a three-dimensional model.SOLUTION: A three-dimensional video generation device 4 comprises: a depth compression unit 40 which performs depth compression of a three-dimensional model; an apex coordinate calculation unit 41 which calculates apex coordinates of the three-dimensional model subjected to the depth compression at an initial observation position and apex coordinates of the three-dimensional model subjected to the depth compression at a post-movement observation position; an apex coordinate conversion unit 42 which converts the apex coordinates of the three-dimensional model after the depth compression between the apex coordinates at the initial observation position and the apex coordinates at the post-movement observation position calculated by the apex coordinate calculation unit 41; and an element image rendering unit 43 which generates a three-dimensional video of the three-dimensional model at the apex coordinates converted by the apex coordinate conversion unit 42.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a stereoscopic video generation device and a program thereof.

Background Art

[0002] A spatial image reproduction type display such as an integral type is a display that can perform a stereoscopic display as if there is a subject without special glasses. In such a spatial image reproduction type display, both horizontal and vertical binocular parallax and motion parallax can be presented. Further, in the spatial image reproduction type display, since the focusing adjustment of the eyeball works by increasing the density of light rays, it is possible to avoid inconsistencies in convergence and adjustment.

[0003] On the other hand, in such a spatial image reproduction type display, it is known that blurring occurs when trying to display a scene with a large depth (Non-Patent Document 1). For example, in an integral type display, a planar lens array and a display element for displaying elemental images are arranged in an overlapping manner, and the spatial frequency decreases as the distance from the lens array surface increases, resulting in blurring. In order to solve this blurring hardware-wise, it is necessary to make the pixel pitch of the flat panel display for displaying elemental images as fine as on the order of several microns, which places a burden on system development.

[0004] Therefore, as shown in FIG. 7, a depth compression method for avoiding blurring by compressing the depth range of a 3D scene into a narrower depth range has been proposed (Non-Patent Document 2). This depth compression method converts all vertex coordinates p0 = (x, y, z) constituting the 3D model P0 of the subject into vertex coordinates p´0 = (x´, y´, z´) of the 3D model P´0 after depth compression, as shown in the following equation (1).

[0005]

Equation

[0006] Note that in FIG. 7, a left-handed coordinate system is assumed, with the observation position (the center of the left and right eyes) v0 as the origin, the horizontal direction as the X-axis, the vertical direction as the Y-axis, and the depth direction as the Z-axis. Also, the X-Y plane formed by the X-axis and the Y-axis is parallel to the display surface, and the Z-axis is perpendicular to the display surface.

[0007] The original plane refers to the depth position d of the three-dimensional model P0 before depth compression. Also, z represents the depth from the observation position v0 to the original plane. The target plane refers to the depth position d' of the three-dimensional model P'0 after depth compression. Also, z' represents the depth from the observation position v0 to the target plane. The display surface refers to the display surface of the spatial image reproduction type display. Also, L0 represents the distance (viewing distance) from the observation position v0 to the display surface.

[0008] As shown in FIG. 8, f(z) in Equation (1) is a depth compression function for moving the vertices of the three-dimensional model P from the back side to the front side, and it decreases the depth value from z to z'. Also, the conversion of x' and y' in Equation (1) is a process for keeping the size of the retinal image constant before and after depth compression when observing the three-dimensional model P from the origin. Therefore, when observing the three-dimensional model P monocularly from the origin, there is no visual difference before and after depth compression. Note that in FIG. 8, the display surface is located at z = 1.0.

[0009] On the other hand, when observing the three-dimensional model P compressed in depth at an observation position v away from the origin, the distortion of the shape becomes prominent. Therefore, in order to suppress this shape distortion, a dynamic depth compression method has been proposed in which the origin is updated so as to always coincide with the observation position v (Non-Patent Document 3). In this dynamic depth compression method, as long as monocular vision is performed from the updated origin even when the observation position v is moved, the retinal image remains constant before and after depth compression.

[0010] Strictly speaking, even if the origin coincides with the observation position v, in reality, since binocular observation is performed simultaneously, a slight difference occurs in the retinal image before and after depth compression, and the binocular disparity amount changes. Specifically, since the observation is performed at both eye positions shifted approximately 3 cm to the left and right from the origin, there may be a case where the stereoscopic image is felt to be slightly closer than before compression.

Prior Art Documents

Non-Patent Documents

[0011]

Non-Patent Document 1

Non-Patent Document 2

Non-Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0012] As shown in Fig. 9(a), in the conventional dynamic depth compression method, when moving from the observation position v0 to the observation position v1, the size of the three-dimensional model P´1 on the target plane decreases from w0 to w1.

[0013] Fig. 9(b) shows the three-dimensional models P (P0, P1, P´0, P´1) of the subject drawn on the display surface before and after the movement of the observation position v and before and after depth compression. As shown in Fig. 9(b), before and after depth compression, the size of the three-dimensional model P on the display surface is constant (compared left and right in Fig. 9(b)). That is, the sizes of the three-dimensional models P1, P´1 shown in the upper part of Fig. 9(b) are the same, and the sizes of the three-dimensional models P0, P´0 shown in the lower part of Fig. 9(b) are also the same.

[0014] On the other hand, before and after the movement of the observation positions v0, v1, the size of the three-dimensional model P on the display surface changes (compared up and down in Fig. 9(b)). That is, the sizes of the three-dimensional models P0, P1 shown on the left side of Fig. 9(b) are different, and the sizes of the three-dimensional models P´0, P´1 shown on the right side of Fig. 9(b) are also different. And on the target plane, the size of the three-dimensional model P also changes in the same way.

[0015] Generally, since it is rare for the size of a subject to change in real space, the phenomenon of the size of the three-dimensional model P changing is considered to give the observer a sense of discomfort. Also, when observing the three-dimensional model P through a display frame B with a fixed size, since the size of the display frame and the three-dimensional model P are compared, it is considered that it is easier to notice the change in the size of the three-dimensional model P.

[0016] Here, the retinal image size is a strong depth cue and is important for giving a sense of deep depth to a depth-compressed 3D scene. Based on the prior knowledge that the subject becomes smaller as it gets farther away, the observer perceives the depth of the subject. From this, when the observation position v is brought closer to the display surface, if the size of the 3D model P on the target plane can be reduced so that no discomfort is felt, it is considered that the sense of depth before compression is less likely to be impaired. Similarly, when the observation position v is moved away from the display surface, it is preferable to increase the size of the 3D model P on the target plane so that no discomfort is felt, and to reduce the difference in retinal image size before and after depth compression.

[0017] In summary, in the conventional dynamic depth compression method, the change in the size of the 3D model P that occurs when the observation position v is moved back and forth causes discomfort to the observer. Also, if the size of the 3D model P is constant when the observation position v is moved back and forth, a difference occurs with respect to the retinal image size before depth compression, resulting in a loss of the sense of depth of the subject and causing discomfort to the observer.

[0018] Therefore, an object of the present invention is to provide a stereoscopic video generation device and a program thereof that can reduce discomfort caused by the size of a 3D model.

Means for Solving the Problems

[0019] To solve the above problems, a stereoscopic video generation device according to the present invention is a stereoscopic video generation device that depth-compresses a 3D model of a subject and generates a stereoscopic video of the 3D model, and includes a depth compression unit, a vertex coordinate calculation unit, a vertex coordinate conversion unit, and a stereoscopic video generation unit.

[0020] According to such a configuration, the depth compression unit View of depth-compresses the 3D model at a predetermined The vertex coordinate calculation unit receives the initial observation position and the post-movement observation position moved from the initial observation position, and calculates the vertex coordinates of the three-dimensional model compressed in depth at the initial observation position and the vertex coordinates of the three-dimensional model compressed in depth at the post-movement observation position.

[0021] The vertex coordinate conversion unit converts the vertex coordinates of the three-dimensional model after depth compression between the vertex coordinates at the initial observation position calculated by the vertex coordinate calculation unit and the vertex coordinates at the post-movement observation position. The stereoscopic video generation unit generates a stereoscopic video of the three-dimensional model with the vertex coordinates converted by the vertex coordinate conversion unit.

[0022] In this way, since the stereoscopic video generation device generates a stereoscopic video after changing the size of the three-dimensional model within a range that does not give the observer a sense of discomfort, the sense of discomfort caused by the size of the three-dimensional model can be reduced.

[0023] Note that the present invention can also be realized by a program for causing a computer to function as the above-described stereoscopic video generation device.

Effect of the Invention

[0024] According to the present invention, the sense of discomfort caused by the size of the three-dimensional model can be reduced.

Brief Description of the Drawings

[0025]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Embodiments for Carrying Out the Invention

[0026] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, each of the embodiments described below is for embodying the technical idea of the present invention, and the present invention is not limited to the following unless specifically described. Also, the same means may be denoted by the same reference numerals, and the description may be omitted.

[0027] [Overview of the Three-Dimensional Video Display System] Referring to FIG. 1, the overview of the three-dimensional video display system 1 according to the embodiment will be described. The three-dimensional video display system 1 displays an integral-type three-dimensional video (element images). As shown in FIG. 1, the three-dimensional video display system 1 includes an observation position detection device 2, a parameter setting device 3, a three-dimensional video generation device 4, and a three-dimensional video display device 5.

[0028] The observation position detection device 2 detects the position (observation position v) where the observer is observing the three-dimensional video by a general method. For example, the observation position detection device 2 can use a method of detecting the observation position v from a camera image or a method of measuring the observation position v by motion capture. Then, the observation position detection device 2 outputs the detected observation position v to the three-dimensional video generation device 4.

[0029] Hereafter, the initial observation position, which is the initial value of the observation position v, is denoted as "observation position v0". The observation position v0 is the origin of the virtual space where the three-dimensional model P0 is placed. Also, the observation position after moving from the initial observation position is denoted as "observation position v1". Currently, the position where the observer is observing the stereoscopic video is the observation position v1.

[0030] The parameter setting device 3 sets the first parameter s and the second parameter t described later. For example, the observer of the stereoscopic video display system 1 manually sets the first parameter s and the second parameter t. Then, the parameter setting device 3 outputs the set first parameter s and second parameter t to the stereoscopic video generation device 4.

[0031] The stereoscopic video generation device 4 performs depth compression on the three-dimensional model P0 of the subject and generates a stereoscopic video of the three-dimensional model P0. Then, the stereoscopic video generation device 4 outputs the generated stereoscopic video to the stereoscopic video display device 5. The details of the stereoscopic video generation device 4 will be described later.

[0032] The stereoscopic video display device 5 displays the stereoscopic video input from the stereoscopic video generation device 4. For example, the stereoscopic video display device 5 is a general 3D display that displays integral-type elemental images.

[0033] [Configuration of Stereoscopic Video Generation Device] Hereinafter, the configuration of the stereoscopic video generation device 4 will be described. As shown in FIG. 1, the stereoscopic video generation device 4 includes a depth compression unit 40, a vertex coordinate calculation unit 41, a vertex coordinate conversion unit 42, and an elemental image rendering unit (stereoscopic video generation unit) 43.

[0034] In the present embodiment, the observation position v1 is input to the stereoscopic video generation device 4 from the observation position detection device 2. Also, the first parameter s and the second parameter t are input to the stereoscopic video generation device 4 from the parameter setting device 3. Further, a three-dimensional model (set of vertex coordinates p0) before depth compression is input to the stereoscopic video generation device 4.

[0035] The depth compression unit 40 compresses the three-dimensional model P0 in the depth direction at a predetermined observation position v0. Here, the depth compression unit 40 compresses the three-dimensional model P0 in the depth direction using a general method such as the above-described formula (1). Then, the depth compression unit 40 outputs the three-dimensional models P0 and P'0 before and after depth compression to the vertex coordinate calculation unit 41.

[0036] The vertex coordinate calculation unit 41 receives the observation position v1 moved from the observation position v0, and calculates the vertex coordinates p'0 of the three-dimensional model P'0 depth-compressed at the observation position v0 and the vertex coordinates p'1 of the three-dimensional model P'1 depth-compressed at the observation position v1.

[0037] Specifically, the vertex coordinate calculation unit 41 calculates the vertex coordinates p'0 = (x', y', z') at the observation position v0 using the depth compression formula of the above-described formula (1). Also, the vertex coordinate calculation unit 41 calculates the vertex coordinates p'1 = (x1', y1', z1') at the observation position v1 using the depth compression formula of the following formula (1'). As shown in FIG. 2, the depth compression function f'(z) of the formula (1') has the depth range moved to the front side compared to the depth compression function f(z) of the formula (1) (see FIG. 8).

[0038]

Equation

[0039] Thereafter, the vertex coordinate calculation unit 41 outputs the three-dimensional model P'0 input from the depth compression unit 40 and the calculated vertex coordinates p'0 and p'1 of the three-dimensional model to the vertex coordinate conversion unit 42.

[0040] The vertex coordinate conversion unit 42 is between the vertex coordinates p'0 at the observation position v0 and the vertex coordinates p'1 at the observation position v1 calculated by the vertex coordinate calculation unit 41, and the vertex coordinates p' of the three-dimensional model P'1 after depth compression 1modIt is something to be transformed. In this embodiment, the vertex coordinate conversion unit 42 weights the amount of movement (p'1 - p'0) of the vertex coordinates of the 3D models P'0 and P'1 from the observation position v0 to the observation position v1 with a parameter a including the first parameter s and the second parameter t, and thus obtains the vertex coordinate p' of the 3D model P'1 after depth compression. 1mod is obtained.

[0041] Specifically, the vertex coordinate conversion unit 42 uses the vertex coordinate conversion formulas of the following formulas (2) and (3) to convert the vertex coordinate p'1 of the 3D model P'1 after depth compression into p'. 1mod As shown in FIG. 3, the vertex coordinate conversion unit 42 converts the vertex coordinate p'1 of the 3D model after forward compression into p'. 1mod By doing so, the width w of the 3D model P'1 is controlled within a range that does not give the observer a sense of discomfort. 1mod Furthermore, the vertex coordinate conversion unit 42 changes the height of the 3D model P'1 in the same way as the width w of the 3D model P'1. Here, w0 is the width of the 3D model P'0 at the vertex coordinate p'0, and w1 is the width of the 3D model P'1 at the vertex coordinate p'1. 1mod

[0042]

Equation

[0043] Here, Equation (2) is affected not only by the forward and backward movement of the observation position v1, but also by the horizontal movement and the vertical movement. Therefore, the parameter a of Equation (2) is divided into a depth direction component, a horizontal direction component, and a vertical direction component. As shown in the following Equation (3), the parameter a consists of a first parameter s that weights the horizontal direction component and the vertical direction component of the vertex coordinate movement amount, and a second parameter t that weights the depth direction component of the vertex coordinate movement amount.

[0044]

Equation

[0045] Here, (v x, v y , v z (3) represents the horizontal component, vertical component, and depth component of the movement vector from the observation position v0 to the observation position v1, respectively. Further, the first parameter s is a coefficient that multiplies the amount of movement in the X-Y plane (horizontal and vertical directions) parallel to the display surface. Also, the second parameter t is a coefficient that multiplies the amount of movement in the Z direction (depth direction) perpendicular to the display surface.

[0046] Subsequently, the vertex coordinate conversion unit 42 outputs the three-dimensional model P´1 converted to the vertex coordinate p´ 1mod to the element image rendering unit 43.

[0047] The element image rendering unit 43 generates a stereoscopic image of the three-dimensional model P´1 with the vertex coordinates p´ 1mod converted by the vertex coordinate conversion unit 42. In this embodiment, the element image rendering unit 43 generates an integral-type element image as a stereoscopic image by a general method. Specifically, the element image rendering unit 43 arranges virtual cameras at each viewpoint position in the virtual space, and arranges the three-dimensional model P´1 of the vertex coordinates p´ 1mod in the virtual space. Then, the element image rendering unit 43 generates an element image by photographing the virtual space (3D scene) in which the three-dimensional model P´1 is arranged with each virtual camera. Subsequently, the element image rendering unit 43 outputs the generated element image to the stereoscopic image display device 5.

[0048] <The first parameter s and the second parameter t> Hereinafter, the first parameter s and the second parameter t will be described in detail. When the observation position v1 does not move in the depth direction (Z axis) and moves only in the horizontal and vertical directions (on the X-Y plane), the term of the second parameter t disappears in Equation (3), so the parameter a becomes equal to the first parameter s. On the other hand, when the observation position v1 does not move in the horizontal and vertical directions (on the X-Y plane) and moves only in the depth direction (Z axis), the term of the first parameter s disappears in Equation (3), so the parameter a becomes equal to the second parameter t.

[0049] Fig. 4 shows the relationship between the second parameter t and the width w of the three-dimensional model P´1 on the target plane. 1mod In Fig. 4, the horizontal axis represents the distance (viewing distance L1) from the observation position v1 to the display surface of the stereoscopic video display device 5, and the vertical axis represents the width w of the three-dimensional model P´1 on the target plane. 1mod Here, the depth position of the three-dimensional model P0 before depth compression is set to 10 m from the display surface, and the depth position of the three-dimensional model P´1 after depth compression is set to 10 cm from the display surface. Also, the observation position v1 is set to 40 cm in front of the center of the stereoscopic video display device 5. As shown in Fig. 4, when the second parameter t = 1, it corresponds to the conventional dynamic depth compression method. As the observation position v1 approaches the display surface from the observation position v0, the width w of the three-dimensional model P´1 1mod decreases significantly. As the value of the second parameter t approaches zero, the slope of the straight line becomes gentle, and the decrease in the width w of the three-dimensional model P´1 1mod also becomes gentle. And when the second parameter t = 0, regardless of the observation position v1, the width w of the three-dimensional model P´1 1mod becomes constant.

[0050] The first parameter s and the second parameter t need to be set in advance based on the experimental results of the subjective evaluation experiment. In this embodiment, the first parameter s is divided into a predetermined number of levels (for example, five levels), and the observer is allowed to select the stereoscopic image that feels natural in the paired comparison. At this time, the condition is that the second parameter t is fixed at 0. Specifically, the first parameter s has five levels of {0.00, 0.25, 0.50, 0.75, 1.00}, and the observer is allowed to select one of the two elemental images generated with different values of the first parameter s that feels natural as a stereoscopic image. Since the first parameter s has five levels, there are 10 combinations of two elemental images, and 10 paired comparisons are performed for the first parameter s. Then, using an analysis method such as Bradley-Terry, as shown in Fig. 5(a), a score related to naturalness is calculated for each value of the first parameter s. In Fig. 5(a), the horizontal axis represents the value of the first parameter s, and the vertical axis represents the score related to naturalness. After that, the value of the first parameter s that gives the maximum score related to naturalness is set in the parameter setting device 3 (for example, the first parameter s = 0.25).

[0051] For the second parameter t, it may be set in the same manner as the first parameter s. The second parameter t has five levels of {0.00, 0.25, 0.50, 0.75, 1.00}, and the condition is that the first parameter s is fixed at 0. The observer is allowed to select one of the two elemental images generated with different values of the second parameter t that feels natural as a stereoscopic image. Then, using an analysis method such as Bradley-Terry, as shown in Fig. 5(b), a score related to naturalness is calculated for each value of the second parameter t. After that, the value of the second parameter t that gives the maximum score related to naturalness is set in the parameter setting device 3 (for example, the second parameter t = 0.50).

[0052] As shown in Fig. 5(a), as the value of the first parameter s decreases, the distortion of the shape of the three-dimensional model P'1 increases, and as the value of the first parameter s increases, the three-dimensional model P'1 is greatly displaced vertically and horizontally in space. Further, as shown in FIG. 5(b), the smaller the value of the second parameter t, the larger the size of the image projected by the subject onto the retina, and the larger the value of the second parameter t, the larger the size change of the three-dimensional model P1'.

[0053] [Operation of the stereoscopic image generation device] With reference to FIG. 6, the operation of the stereoscopic image generation device 4 will be described. As shown in FIG. 6, in step S1, the stereoscopic image generation device 4 receives the first parameter s and the second parameter t. In step S2, the stereoscopic image generation device 4 receives the observation position v1 from the observation position detection device 2. In step S3, the stereoscopic image generation device 4 receives the three-dimensional model P0.

[0054] In step S4, the depth compression unit 40 compresses the three-dimensional model P0 in depth at a predetermined observation position v0. In step S5, the vertex coordinates p'0 of the three-dimensional model P'0 compressed in depth at the observation position v0 and the vertex coordinates p'1 of the three-dimensional model P'1 compressed in depth at the observation position v1 are calculated.

[0055] In step S6, the vertex coordinate conversion unit 42 converts the vertex coordinates p'1 of the three-dimensional model P'1 after depth compression between the vertex coordinates p'0 at the observation position v0 and the vertex coordinates p'1 at the observation position v1. In step S7, the element image rendering unit 43 renders the element image of the three-dimensional model P'1 converted to the vertex coordinates p' 1mod Render the element image of the three-dimensional model P'1 converted to.

[0056] [Function and effect] Since the three-dimensional model size is changed before generating the elemental images within a range that does not give an observer a sense of discomfort, the stereoscopic image generation apparatus 4 can reduce the sense of discomfort caused by the size of the three-dimensional model. Thus, the stereoscopic image generation apparatus 4 can realize a stereoscopic image display with less sense of discomfort. Even when the depth reproduction ability is limited as in a conventional integral type display, it is possible to display a deep scene such as a television broadcast, and the sense of discomfort due to the movement of the observation position at that time can be reduced.

[0057] As described above, the embodiments have been described in detail, but the present invention is not limited to the above-described embodiments, and also includes design changes and the like within a range not departing from the gist of the present invention.

[0058] In the above-described embodiment, for simplicity of explanation, the subject is illustrated as a cube, but the subject is not particularly limited. In the above-described embodiment, an example of generating integral type elemental images as the stereoscopic image has been described, but the present invention is not limited thereto. For example, the stereoscopic image generation apparatus may generate a stereoscopic image such as a lenticular method or a binocular stereoscopic method.

[0059] In the above-described embodiment, the stereoscopic image generation apparatus has been described as independent hardware, but the present invention is not limited thereto. For example, the present invention can also be realized by a program that operates hardware resources such as a CPU, a memory, and a hard disk provided in a computer as the above-described stereoscopic image generation apparatus. These programs may be distributed via a communication line, or may be written on a recording medium such as a CD-ROM or a flash memory and distributed.

Explanation of Reference Numerals

[0060] 1 Stereoscopic image display system 2 Observation position detection device 3 Parameter setting device 4 Stereoscopic image generation device 5 Stereoscopic image display device 40 Depth compression unit 41 Vertex coordinate calculation unit 42 Vertex coordinate conversion unit 43 Element image rendering unit (stereoscopic video generation unit)

Claims

1. A stereoscopic video generation device that generates a stereoscopic video by axially compressing a three-dimensional model of a subject, comprising: an axial compression unit that axially compresses the three-dimensional model at a predetermined observation position; a vertex coordinate calculation unit that receives an initial observation position and a post-movement observation position moved from the initial observation position, and calculates vertex coordinates of the three-dimensional model axially compressed at the initial observation position and vertex coordinates of the three-dimensional model axially compressed at the post-movement observation position; a vertex coordinate conversion unit that converts vertex coordinates of the three-dimensional model after axial compression between the vertex coordinates at the initial observation position calculated by the vertex coordinate calculation unit and the vertex coordinates at the post-movement observation position; a stereoscopic video generation unit that generates a stereoscopic video of the three-dimensional model with the vertex coordinates converted by the vertex coordinate conversion unit; A stereoscopic video generation device characterized by comprising the above.

2. The vertex coordinate conversion unit: has a first parameter for weighting a horizontal component and a vertical component of an axial movement amount of vertex coordinates, and a second parameter for weighting an axial direction component of the axial movement amount of vertex coordinates, which are preset; The stereoscopic video generation device according to claim 1, wherein the vertex coordinates of the three-dimensional model after axial compression are obtained by weighting the movement amount of the vertex coordinates of the three-dimensional model from the initial observation position to the post-movement observation position with a parameter including the first parameter and the second parameter.

3. The stereoscopic video generation device according to claim 2, wherein the first parameter and the second parameter are preset based on an experimental result of a subjective evaluation experiment regarding the naturalness of the appearance of the stereoscopic video.

4. The stereoscopic video generation device according to any one of claims 1 to 3, wherein the stereoscopic video generation unit generates elemental images in an integral method as the stereoscopic video.

5. A program for causing a computer to function as the stereoscopic video generation device according to any one of claims 1 to 4.

Images