Viewpoint changing device and its program

Abstract

To provide a viewpoint conversion device and a program thereof that maintain the composition of a blank space when converting the viewpoint of a 3D image.SOLUTION: In a viewpoint conversion device 1, a calculation unit 10 includes a vertex coordinate calculation unit 12 that calculates the coordinates of each vertex vi when a subject is projected onto a normalization screen, a top and bottom point selection unit 13 that selects the top and bottom points of each vertex, a margin calculation unit 14 that calculates an upper margin and a lower margin, and a conversion parameter calculation unit 15 that calculates a conversion parameter α such that the margin ratio rv between the upper margin and the lower margin becomes a preset value.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to a viewpoint conversion device and a program thereof.

Background Art

[0002] Stereoscopic video display devices are expected to diversify into various forms such as not only stationary displays with vertical screens that are popular in homes, but also portable terminals (e.g., smartphones and tablets), tabletop displays, HMDs (Head Mounted Displays), etc. If the same content (stereoscopic video) can be distributed in a video expression suitable for different forms of stereoscopic video display devices, the opportunities for viewing in daily life will increase, and an experience that makes use of the characteristics of each stereoscopic video display device can be expected. Here, a conventional technique for converting the viewpoint of a stereoscopic video has been proposed so as to obtain an appropriate video expression according to the form of each stereoscopic video display device (Patent Document 1 and Non-Patent Documents 1-6).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Non-Patent Documents

[0004]

Non-Patent Document 1

Non-Patent Document 2

Outdoor Tools3

Outdoor Tools 4

Direct Environment 5

Outdoor Configuration6

[0005] The conventional technology described above has a problem in that it does not take into account the composition based on the margin ratio when converting stereoscopic images. The problems of the conventional technology will be explained in detail with reference to Figure 6. Figure 6 illustrates an example in which the viewpoint of a stereoscopic image displayed by a stationary display 90 is converted for display on a tabletop display 91.

[0006] Arrow a represents the horizontal direction of the virtual space reproduced by the tabletop display 91. The dashed line b connects the center of the tabletop display 91 to the position of the viewer 9 and represents the shooting direction of the virtual camera. This virtual camera is a camera that virtually photographs the subject Obj in the virtual space and is usually positioned at the viewpoint (the position of the viewer 9). Hereafter, the virtual camera may be abbreviated as "camera". The angle θ represents the angle between the horizontal direction of the virtual space and the shooting direction of the camera, that is, the angle between arrow a and dashed line b.

[0007] As shown in Figure 6(b), consider the case where the viewpoint of the stereoscopic image is transformed so that the retinal image of the subject Obj is the same between the stationary display 90 and the tabletop display 91 (θ=0). In this case, a discrepancy may occur between the virtual world and the real world in the direction of gravity or horizontally. If such a discrepancy becomes extremely large, it may cause discomfort to the viewer 9.

[0008] As shown in Figure 6(c), consider the case where the viewpoint of the stereoscopic image is transformed between the stationary display 90 and the tabletop display 91 so that the direction of gravity in the virtual world and the real world match (for example, θ = π / 4). In this case, the discrepancy in the angle θ when viewing the subject Obj between the stationary display 90 and the tabletop display 91 becomes large, and the resulting image may not be what the creator of the stereoscopic image intended. Therefore, as shown in Figure 6(d), it is necessary to adjust the angle θ so that the balance between the degree of agreement of the direction of gravity and the similarity of the angle θ is appropriate (for example, θ = π / 8).

[0009] Here, when changing the viewpoint of a stereoscopic image between different types of stereoscopic image display devices, simply rotating the camera around the subject Ojb may not maintain the composition due to the margins. As shown in Figure 7(a), in the stationary display 90, the margin ratio from the outline of the subject Obj to the top and bottom screen frames is approximately 0.775. This margin ratio is the length L from the top of the screen frame to the top edge of the subject. U And the length L from the bottom frame of the screen to the bottom edge of the subject. D This is the ratio.

[0010] In contrast, as shown in Figure 7(b), when the subject Obj is not rotated (θ=0) when displayed on the tabletop display 91, the margin ratio is approximately 0.357. Furthermore, as shown in Figure 7(c), when the subject Obj is rotated by an angle θ=π / 4, the margin ratio is approximately 0.351. Moreover, as shown in Figure 7(d), when the subject Obj is rotated by an angle θ=π / 8, the margin ratio is approximately 0.367. Thus, after the viewpoint transformation of the stereoscopic image, the margin ratio on the tabletop display 91 fluctuates around 0.35, failing to maintain the composition with margins as when displayed on the stationary display 90. In Figure 7, the viewer 9 is looking down at the tabletop display 91 at an angle, so the tabletop display 91 is trapezoidal in shape.

[0011] The object of this invention is to provide a viewpoint transformation device and program that can maintain the composition due to margins when transforming the viewpoint of a stereoscopic image. [Means for solving the problem]

[0012] To solve the above problems, the viewpoint conversion device according to the present invention converts the stereoscopic image displayed by the first stereoscopic image display device to the first stereoscopic image display device Display screen A viewpoint conversion device for converting the viewpoint of a stereoscopic image in order to display it on a second stereoscopic image display device having a display screen at a different angle, comprising a vertex coordinate calculation unit, an uppermost and lowermost point selection unit, a margin calculation unit, and a conversion parameter calculation unit.

[0013] According to this configuration, the vertex coordinate calculation unit pre-sets a normalization screen that is perpendicular to the optical axis of the virtual camera at a unit distance from the virtual camera located at the viewpoint, and when the subject of the 3D image is projected onto the normalization screen... The subject Calculate the coordinates of each vertex. The uppermost and lowermost point selection unit selects the uppermost and lowermost points from the coordinates of each vertex calculated by the vertex coordinate calculation unit.

[0014] The margin calculation unit calculates the upper margin from the top frame of the normalized screen to the top point, and the lower margin from the bottom frame of the normalized screen to the top point. under Calculate the bottom margin up to the point. The conversion parameter calculation unit calculates the amount of movement required to move the subject along the display screen of the second stereoscopic image display device as a conversion parameter, such that the margin ratio between the upper margin and the lower margin becomes a preset value.

[0015] By using these conversion parameters, when the stereoscopic image displayed by the first stereoscopic image display device is displayed on the second stereoscopic image display device, the relationship between the top and bottom points and the margins is adjusted, thereby maintaining the composition created by the margins.

[0016] Furthermore, the present invention can also be realized by a program that causes a computer to function as the aforementioned viewpoint changing device. [Effects of the Invention]

[0017] According to the present invention, when changing the viewpoint of a stereoscopic image, the composition created by the margins can be maintained. [Brief explanation of the drawing]

[0018] [Figure 1] This diagram illustrates a method for moving the center of gravity of a subject along the display screen in an embodiment. [Figure 2] In the embodiment, (a) is a diagram illustrating the abstraction of the shape of the subject, (b) is an explanatory diagram illustrating the projection onto the normalization screen, and (c) is a diagram illustrating the selection of the uppermost and lowermost points. [Figure 3] This is a block diagram showing the configuration of a viewpoint changing device in an embodiment. [Figure 4] A flowchart illustrating the operation of the viewpoint changing device in this embodiment is shown. [Figure 5] This figure shows an example in which the viewpoint of a stereoscopic image is transformed. [Figure 6] Figures (a) to (d) show examples of transforming the viewpoint of stereoscopic images using conventional technology. [Figure 7] (a) to (d) are diagrams illustrating the margins in the prior art. [Modes for carrying out the invention]

[0019] Embodiments of the present invention will be described below with reference to the drawings. However, the embodiments described below are intended to embody the technical concept of the present invention, and unless otherwise specified, the present invention is not limited to these embodiments. In addition, the same reference numerals are used for the same means, and their descriptions may be omitted.

[0020] (Embodiment) [Perspective Shifting Techniques] The following describes a specific method of viewpoint transformation as a prerequisite for explaining the viewpoint transformation device 1 according to the embodiment. To maintain the composition created by the negative space, when changing the viewpoint of the stereoscopic image, in addition to rotating the camera around the center of the subject (Obj) as the axis of rotation, the camera is also translated. As shown in Figure 1, this camera operation is equivalent to fixing the camera at viewpoint c, rotating the subject (Obj), and then translating the subject (Obj). In other words, after rotating the subject (Obj), the center of gravity (v) of the subject (Obj) is... g The viewpoint of the stereoscopic image is changed by moving the display screen (actual screen) 91a of the tabletop display 91 parallel to it. In Figure 1, the display state of the subject Obj before movement is shown in darker colors, and the display state of the subject Obj before movement is shown in lighter colors.

[0021] The angle θ between arrow a, which represents the horizontal direction in the virtual space, and dashed line b, which represents the camera's shooting direction, represents the rotation angle of the subject Obj (i.e., the camera's rotation angle). Furthermore, to prevent the subject Obj's position from changing in the depth direction when changing the viewpoint, the creator will manually set the distance the subject Obj moves.

[0022] In this embodiment, the subject object (Obj) is assumed to be an automobile, but the type of subject object (Obj) is not particularly limited. Also, since the 3D shape of the subject object (Obj) must be known, the target scene is assumed to be a 3D computer graphics (CG) consisting of a mesh structure or point cloud format.

[0023] After the viewpoint change, the centroid v of the subject Obj. g This can be expressed by the following equation (1), using the unknown transformation parameter α.

[0024]

number

[0025] Note, s c d represents the center coordinates of the display screen 91a, and d represents the distance between the display screen 91a and the subject Obj. Also, n represents the unit normal vector to the display screen 91a, and t represents the unit vector parallel to the display screen 91a and pointing upward.

[0026] When the subject Obj has a complex shape, if one tries to accurately obtain the margin according to the change of the center of gravity v of the subject Obj g of the subject Obj, the amount of calculation will become enormous. Therefore, after abstracting the shape of the subject Obj, the margin is obtained. Specifically, as shown in Fig. 2(a), the shape of the subject Obj is abstracted by the bounding box Box, and the coordinates of the vertex v i are obtained from the subject Obj abstracted into a box shape. Here, the coordinates of each vertex v i of the subject Obj after the viewpoint transformation are represented by the following formula (2).

[0027]

Equation

[0028] Note that v i ￣ represents the vector from the center of gravity v g of the subject Obj before the viewpoint transformation to each vertex of the subject Obj abstracted into a box shape (that is, each vertex of the bounding box Box). Here, the bounding box Box refers to a rectangular parallelepiped boundary that surrounds the subject Obj in the virtual space. Also, when the shape of the subject Obj is abstracted by the bounding box Box, the center of gravity v g of the subject Obj may be the average of each vertex v i ￣ of the bounding box Box.

[0029] Substituting formula (1) into formula (2), the coordinates of each vertex v i are represented by the following formula (3).

[0030]

Equation

[0031] If the constant vector in formula (3) is set as h i in formula (4), the coordinates of each vertex v i are represented by the following formula (5).

[0032]

number

number

[0033] As shown in Figure 2(b), each vertex v i Let's consider projecting onto a normalized screen 91b. This normalized screen 91b is a plane perpendicular to the optical axis of the camera at a unit distance (e.g., 1 meter) from the camera located at viewpoint c. i The homogeneous coordinates v' of the projection point when projected onto the normalized screen 91b. i ￣ can be expressed using transformation matrices by the following equations (6) and (7).

[0034]

number

number

[0035] Note that P represents the projection matrix onto the normalized screen 91b, and M represents the transformation matrix from the world coordinate system to the camera coordinate system. Also, H i h is a constant vector i =( h i,x ,h i,y ,h i,z This represents a matrix that translates in the direction of ), and A represents a matrix that scales the vector by a factor of α.

[0036] In Figure 2(b), s1 to s4 represent the four vertices of the display screen 91a. Vertices s'1 to s'4 are the points obtained by projecting vertices s1 to s4 of the display screen 91a onto the normalized screen 91b.

[0037] Homogeneous coordinate v in equation (6) i ´￣=(v´ i,x ￣,v´ i,y ￣,v´ i,z Convert ￣) into a 3D vector, and the vertex v' is represented by equation (8).i Obtain v'. i,w ￣ represents the homogeneous coordinate v i Represents the w component of ´￣.

[0038]

number

[0039] Next, as shown in Figure 2(c), the eight vertices v' projected onto the normalized screen 91b are i From the coordinates, the highest point v' is located as shown in equations (9) to (11) below. top And the lowest point, v', is located at the very bottom. bottom Select this option. Note that k represents the upward unit vector in the normalized screen 91b.

[0040]

number

number

number

[0041] In other words, the highest point v' top is vertex v' i It is the point closest to the upper frame s'1-s'4. Also, the lowest point v' bottom is vertex v' i This is the point closest to the lower frame s'2-s'3. Also, equation (11) assumes a left-handed coordinate system, where the upward direction of the normalized screen 91b is the y-axis of the left-handed coordinate system. In this case, the camera position (viewpoint c) is the horizontal center of the display screen 91a, and the camera does not roll.

[0042] From the upper frame s'1-s'4 of the normalized screen 91b to the top point v' top Upper margin up to m' top This is expressed by the following equation (12). Also, from the lower frame s'2-s'3 of the normalized screen 91b to the lowest point v'bottom The bottom margin up to m' bottom This is expressed by the following equation (13).

[0043]

number

number

[0044] Here, the top and bottom margin ratio r v We define this by equation (14) below. Then we obtain the relationship expressed by equation (15) below.

[0045]

number

number

[0046] By determining the unknown transformation parameter α from equation (15) and substituting this transformation parameter α into equation (1), the predetermined margin ratio r can be obtained. v The centroid v of the subject Obj that satisfies the conditions g You can obtain this.

[0047] [Configuration of the viewpoint changing device] Referring to Figure 3, the configuration of the viewpoint changing device 1 will be explained. The viewpoint conversion device 1 converts the viewpoint of the stereoscopic image displayed by the first stereoscopic image display device in order to display the stereoscopic image on a second stereoscopic image display device that has a display screen at a different angle than the first stereoscopic image display device.

[0048] In this embodiment, the first stereoscopic image display device is a stationary display 90 having a vertical display screen. The second stereoscopic image display device is a tabletop display 91 having a horizontal display screen.

[0049] As shown in Figure 3, the viewpoint transformation device 1 comprises a calculation unit 10 and a drawing unit (viewpoint transformation unit) 20. The calculation unit 10 also comprises a subject abstraction unit 11, a vertex coordinate calculation unit 12, an uppermost and lowermost point selection unit 13, a margin calculation unit 14, and a transformation parameter calculation unit 15.

[0050] Here, the calculation unit 10 receives the position and orientation information of the actual screen and camera {s1, ..., s4}, c, and the position and orientation information of the subject Obj before viewpoint transformation (v g ,v i Enter (￣). The angle θ mentioned above is calculated using the position and orientation information of the actual screen and camera {s1,...,s4},c and the position and orientation information of the subject Obj before viewpoint transformation (v g ,v i It can be calculated from (￣). Furthermore, the calculation unit 10 has parameters (distance d, margin ratio r) that the creator has manually set. v Enter ).

[0051] The object abstraction unit 11 abstracts the shape of the object Obj using a bounding box. As shown in Figure 2(a), the object abstraction unit 11 abstracts the shape of the object Obj into a box shape. For example, the bounding box only needs to be the size that the object Obj is inscribed in.

[0052] The vertex coordinate calculation unit 12 pre-sets the normalization screen 91b and calculates each vertex v' when the object Obj of the 3D image is projected onto the normalization screen 91b. i This calculates the coordinates of each vertex v' when the subject Obj, abstracted by the bounding box Box, is projected onto the normalization screen 91b. i The coordinates are calculated. Specifically, the vertex coordinate calculation unit 12 uses the above-mentioned equations (2) to (8) to calculate the coordinates of each vertex v'. i Calculate the coordinates.

[0053] The uppermost and lowermost point selection unit 13 selects each vertex v' calculated by the vertex coordinate calculation unit 12. i Our highest point v' top and the lowest point v' bottomThis selects the uppermost and lowermost point. Specifically, the uppermost and lowermost point selection unit 13 uses the above-mentioned equations (9) to (11) to select the uppermost point v'. top and the lowest point v' bottom Select this option.

[0054] The margin calculation unit 14 calculates the uppermost point v' from the upper frame s'1-s'4 of the normalized screen 91b. top Upper margin up to m' top And, from the lower frame s'2-s'3 of the normalized screen 91b, under Lower margin m' to the point bottom This calculates the upper margin m' using the above-mentioned formulas (12) and (13). Specifically, the margin calculation unit 14 uses the above-mentioned formulas (12) and (13) to calculate the upper margin m'. top and bottom margin m' bottom Calculate.

[0055] The conversion parameter calculation unit 15 calculates the upper margin m'. top and bottom margin m' bottom The margin ratio is set to a predetermined value r v To achieve this, the amount of movement required to move the subject obj along the display screen 91a is calculated as the conversion parameter α. Specifically, the conversion parameter calculation unit 15 calculates the conversion parameter α by solving equations (14) and (15). After that, the calculation unit 10 outputs the calculated conversion parameter α to the drawing unit 20.

[0056] The drawing unit 20 transforms the viewpoint of the stereoscopic image based on the transformation parameter α. Specifically, the drawing unit 20 substitutes the transformation parameter α input from the calculation unit 10 into equation (1) to obtain the margin ratio r set by the creator. v The centroid v of the subject Obj that satisfies the conditions g The centroid v can be determined. Then, the drawing unit 20 calculates the centroid v g The object Obj, located at point C, is photographed by the camera at viewpoint C to render a stereoscopic image.

[0057] [Operation of the viewpoint changing device] Referring to Figure 4, the operation of the viewpoint changing device 1 will be explained. As shown in Figure 4, in step S1, various parameters are input to the viewpoint changing device 1. In step S2, the subject abstraction unit 11 abstracts the shape of the subject Obj using a bounding box.

[0058] In step S3, the vertex coordinate calculation unit 12 pre-sets the normalization screen 91b and projects the 3D image subject Obj onto the normalization screen 91b, and calculates each vertex v'. i Calculate the coordinates. In step S4, the uppermost and lowermost point selection unit 13 selects each vertex v' calculated in step S3. i Our highest point v' top and the lowest point v' bottom Select this option.

[0059] In step S5, the margin calculation unit 14 calculates the upper margin m'. top and bottom margin m' bottom Calculate. In step S6, the conversion parameter calculation unit 15 calculates the upper margin m'. top and bottom margin m' bottom The margin ratio is set to a predetermined value r v The transformation parameter α is calculated so that this is the case. In step S7, the drawing unit 20 transforms the viewpoint of the stereoscopic image based on the transformation parameter α.

[0060] [Effects / Effects] Figure 5 shows the margin ratio r. v 1, that is, the upper margin m' top and bottom margin m' bottom The diagram illustrates stereoscopic images obtained by changing the angle θ when the settings are made to be uniform. In the example in Figure 5, the distance d = -0.1, 0, 0.1 and the angle θ = 0, π / 8, π / 4, 3π / 8, π / 2. As shown in Figure 5, the viewpoint changing device 1 adjusts the relationship between the uppermost and lowermost points and the margins even when the angle θ is changed, so the composition with margins can be maintained. This makes it easier to create stereoscopic images that reflect the creator's intentions.

[0061] (modified version) Although embodiments have been described in detail above, the present invention is not limited to the embodiments described above, and includes design changes and the like that do not depart from the spirit of the present invention.

[0062] In the embodiment described above, an example was given in which the composition is maintained by the ratio of margins in the vertical direction, but the viewpoint changing device can similarly maintain the composition by the ratio of margins in the horizontal direction.

[0063] In the embodiments described above, the display screens of the first and second stereoscopic image display devices were described as being rectangular in shape, but the shape of the display screen is not particularly limited. For example, even if the display screen has any shape, such as a circle or an ellipse, the viewpoint changing device can similarly maintain the composition based on the margin ratio as long as information about the edges of the display screen is available.

[0064] In the embodiments described above, the stereoscopic image was explained as a still image, but the stereoscopic image may also be a moving image. In this case, the size of the bounding box should be the range in which the subject moves.

[0065] In the embodiment described above, the shape of the subject was abstracted using a bounding box, but it is not necessary to abstract the shape of the subject. In this case, the vertex coordinate calculation unit calculates the coordinates for each vertex of the subject, and the uppermost and lowermost point selection unit selects the uppermost and lowermost points from the coordinates of each vertex. In other words, the number of vertices in equations (9) and (10) should be changed.

[0066] In the above-described embodiment, the viewpoint conversion device was described as rendering a stereoscopic image, but the invention is not limited to this. For example, a viewpoint conversion device located on the distribution side may calculate conversion parameters and transmit these parameters to a rendering device located on the viewer side. In this case, the rendering device on the viewer side will render the stereoscopic image.

[0067] In the embodiments described above, the viewpoint changing device was described as an independent piece of hardware, but the present invention is not limited thereto. For example, the present invention can also be realized by a program that causes hardware resources such as the CPU, memory, and hard disk of a computer to function as the viewpoint changing device described above. This program may be distributed via a communication line, or it may be written to a recording medium such as a CD-ROM or flash memory and distributed. [Explanation of Symbols]

[0068] 1. Viewpoint changing device 10 Arithmetic section 11 Subject abstraction section 12. Vertex coordinate calculation unit 13. Selection section for the highest and lowest points 14. Margin Calculation Section 15. Calculation unit for conversion parameters 20 Drawing section (viewpoint transformation section)

Claims

1. A viewpoint conversion device for converting the viewpoint of a stereoscopic image, in order to display the stereoscopic image displayed by a first stereoscopic image display device on a second stereoscopic image display device having a display screen at a different angle from the display screen of the first stereoscopic image display device, A vertex coordinate calculation unit pre-sets a normalization screen perpendicular to the optical axis of a virtual camera at a unit distance from the virtual camera located at the viewpoint, and calculates the coordinates of each vertex of the subject when the subject of the stereoscopic image is projected onto the normalization screen. The vertex coordinate calculation unit selects the highest and lowest points among the vertices calculated by the vertex coordinate calculation unit, A margin calculation unit that calculates the upper margin from the upper frame of the normalization screen to the uppermost point and the lower margin from the lower frame of the normalization screen to the lowermost point, A conversion parameter calculation unit calculates the amount of movement to move the subject along the display screen of the second stereoscopic image display device as a conversion parameter, such that the margin ratio between the upper margin and the lower margin becomes a predetermined value. A viewpoint changing device characterized by being equipped with the following features.

2. The system further comprises a subject abstraction unit that abstracts the shape of the subject using a bounding box, The viewpoint transformation device according to claim 1, characterized in that the vertex coordinate calculation unit calculates the coordinates of each vertex when the subject abstracted by the bounding box is projected onto the normalization screen.

3. The viewpoint conversion device according to claim 1, characterized in that the first stereoscopic image display device is a stationary display having a vertical display screen.

4. The viewpoint changing device according to claim 1, characterized in that the second stereoscopic image display device is a tabletop type display having a horizontal display screen.

5. The viewpoint conversion device according to claim 1, further comprising a viewpoint conversion unit that converts the viewpoint of the stereoscopic image based on the conversion parameters.

6. A program for a computer to function as a viewpoint changing device according to any one of claims 1 to 5.

Images