Viewpoint calculation device and program thereof

Abstract

To calculate an optimum viewpoint between stereoscopic display apparatuses having different display screen directions.SOLUTION: A stereoscopic video conversion apparatus 1 comprises a viewpoint recommendation unit 11 for recommending a second viewpoint when a stereoscopic video that is displayed by a first stereoscopic display apparatus at a first viewpoint is displayed by a second stereoscopic display apparatus and recommending the second viewpoint that minimizes an objective function expressed as a weighted sum of a difference in a direction of the ground in the stereoscopic video from the first and second viewpoints and a difference in the appearance of a subject in the stereoscopic video from the first and second viewpoints.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a viewpoint calculation device and a program therefor that can calculate an optimal viewpoint between stereoscopic display devices having display screens with different orientations. [Background technology]

[0002] Conventionally, as shown in FIG. 8, there has been a stationary stereoscopic display device 9 as found in an ordinary home. F In addition, tabletop stereoscopic display devices9 T There are various types of 3D display devices, including tablets and head-mounted displays. As the number of types of 3D display devices increases, opportunities to enjoy 3D images will increase and new experiences suited to each 3D display device can be expected.

[0003] Here, the stationary stereoscopic display device 9 F The vertical display screen is 90°, just like a typical display. F Hereinafter, the term "stationary stereoscopic display device" may be abbreviated to "display." In addition, a tabletop stereoscopic display device9 T The table top is 90 cm wide and the display screen is 90 cm wide. T In other words, it is a tabletop stereoscopic display device9. T The horizontal display screen is 90 T Hereinafter, "tabletop type stereoscopic display device" may be abbreviated to "tabletop."

[0004] For example, if the viewpoint is display 9 F 3D image set for table top 9 T In this 3D image, a man and a woman are facing each other, with the subject at 100 1 ,100 2 As shown in FIG. F The vertical display screen is 90 FSince the stereoscopic image is displayed in the real world, the ground 101 appears horizontally, just like in the real world, and no unnaturalness occurs. T The horizontal display screen is 90 T Display 9 F Since the tabletop 9 displays a stereoscopic image for a tabletop, unlike the real world, the ground 101 appears vertical, which creates an unnatural feeling. T In this case, the ground direction in the virtual world is significantly different from that in the real world, which creates an unnatural appearance. As a result, it will be necessary to create 3D images according to the type of 3D display device, which is expected to require a great deal of effort. In addition, Table Top 9 T The space of the three-dimensional images displayed on a screen is sometimes referred to as the "virtual world."

[0005] Also, Table Top 9 T Consider a case where the viewpoint of the stereoscopic image is changed so that the ground 101 appears horizontal, and the direction of the ground in the virtual world is made to completely match the direction of the ground in the real world. In this case, as shown in FIG. F In the stereoscopic image displayed by the 1 ,100 2 This is a side view of the subject, so 1 ,100 2 The expression on the face is easy to understand. On the other hand, Table Top 9 T In the stereoscopic image displayed by the 1 ,100 2 The composition is converted to a top-down shot, and the subject is 100 1 ,100 2 It is difficult to understand the expression of the face. T So, subject 100 1 ,100 2 Sometimes the intended content cannot be conveyed through the original 3D image, such as making it difficult to understand facial expressions.

[0006] Furthermore, consider the case where the viewpoint is randomly changed. In this case, as shown in FIG. F In the stereoscopic image displayed by the 1100 More Female Subjects 2 The subject is in the foreground, and the female subject is 100 2 On the other hand, Table Top 9 T In the stereoscopic image displayed by the 1 100 female subjects 2 The male subject was replaced by the male subject 100. 1 In this way, the 100 most noticeable subjects 1 ,100 2 may be swapped, resulting in a significant change in the content of the original 3D image.

[0007] In response to this, a method for converting depth information of stereoscopic images between different types of stereoscopic display devices has been proposed (Non-Patent Document 1). Also, a method for searching for an optimal viewpoint for a certain subject 100 has been proposed (Non-Patent Document 2, Patent Document 1). [Prior art documents] [Patent documents]

[0008] [Patent Document 1] Patent No. 4272966 [Non-patent literature]

[0009] [Non-Patent Document 1] Manuel Lang, Alexander Hornung, Oliver Wang, Steven Poulakos, Aljoscha Smolic, Markus Gross, Nonlinear Disparity Mapping for Stereoscopic 3D, ACM SIGGRAPH 2010. [Non-Patent Document 2] Mateu Sbert, Dimitri Plemenos, Miquel Feixas, and Francisco Gonzalez,Viewpoint Quality: Measures and Applications, In Proceedings of the First Eurographics conference on Computational Aesthetics in Graphics 2005. Summary of the Invention [Problem to be solved by the invention]

[0010] However, the method described in Non-Patent Document 1 only converts the depth information of a stereoscopic image, and does not convert the viewpoint of the stereoscopic image. In addition, the methods described in Non-Patent Document 2 and Patent Document 1 are limited to a specific stereoscopic display device, and are not compatible with various types of stereoscopic display devices.

[0011] Therefore, an object of the present invention is to provide a viewpoint calculation device and a program therefor that are capable of calculating an optimal viewpoint between stereoscopic display devices having different display screen orientations. [Means for solving the problem]

[0012] In order to solve the above problems, the viewpoint calculation device of the present invention is a viewpoint calculation device that calculates a second viewpoint when a stereoscopic image displayed at a first viewpoint by a first stereoscopic display device having a display screen in a predetermined direction is displayed on a second stereoscopic display device having a display screen in a direction different from that of the first stereoscopic display device, and is configured to include a viewpoint calculation unit.

[0013] According to this configuration, the viewpoint calculation unit calculates the second viewpoint that minimizes an objective function expressed as a weighted sum of the difference in ground direction in the stereoscopic images from the first viewpoint and the second viewpoint, and the difference in how the subject appears in the stereoscopic images from the first viewpoint and the second viewpoint. In this way, the viewpoint calculation device converts the first viewpoint of the first stereoscopic display device into an optimal second viewpoint of the second stereoscopic display device while maintaining the content of the original stereoscopic video as much as possible.

[0014] The present invention can also be realized by a program for causing a computer to function as the viewpoint calculation device described above. Effect of the Invention

[0015] According to the present invention, it is possible to calculate an optimal viewpoint between stereoscopic display devices having display screens with different orientations. [Brief description of the drawings]

[0016] [Figure 1] 1 is a block diagram showing a configuration of a stereoscopic video conversion device according to an embodiment. [Diagram 2] 10A and 10B are explanatory diagrams illustrating differences in ground directions of stereoscopic images in an embodiment. [Diagram 3] 1A and 1B are explanatory diagrams illustrating the area of ​​each surface constituting a subject in an embodiment, where FIG. 1A shows the subject viewed from the front side, and FIG. [Figure 4] 4 is an explanatory diagram illustrating the area of ​​each face of a subject projected onto a normalized image coordinate system in an embodiment. FIG. [Diagram 5] FIG. 2 is an explanatory diagram for explaining an example of a subjective evaluation experiment in the embodiment. [Figure 6] 11 is a graph showing a score for each second viewpoint candidate in the embodiment. [Figure 7] 4 is a flowchart showing the operation of the three-dimensional video conversion device according to the embodiment. [Figure 8] FIG. 1 is an explanatory diagram illustrating a state in which the direction of the ground in the virtual world is significantly different from the direction of the ground in the real world in the prior art. [Figure 9] FIG. 1 is an explanatory diagram illustrating a state in which the direction of the ground in the virtual world completely matches the direction of the ground in the real world in the prior art. [Figure 10]FIG. 1 is an explanatory diagram illustrating the replacement of a subject in the prior art. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, each embodiment described below is intended to embody the technical idea of ​​the present invention, and unless otherwise specified, the present invention is not limited to the following. In addition, the same symbols are used for the same means, and descriptions thereof may be omitted.

[0018] (Embodiment) [Configuration of 3D image conversion device] The configuration of a stereoscopic image conversion device (viewpoint calculation device) 1 according to the embodiment will be described with reference to FIG. The stereoscopic image conversion device 1 is a first stereoscopic display device having a display screen in a predetermined direction, and the first stereoscopic display device is a first viewpoint v s A stereoscopic image displayed at the second viewpoint v when the stereoscopic image displayed at the first stereoscopic display device is displayed at a second stereoscopic display device having a display screen in a different direction from that of the first stereoscopic display device. t The 3D image conversion device 1 then recommends (calculates) the recommended second viewpoint v t The stereoscopic image is rendered and displayed on the second stereoscopic display device. The direction of the display screen is the direction indicated by the normal line of the display screen. t The recommendation is made in a prescribed manner from the second viewpoint v t The purpose of the system is to calculate and present the above information to the outside world.

[0019] In this embodiment, the first stereoscopic display device has a display screen 90 in the vertical direction. F Display 9 F (Figure 8). Therefore, the first viewpoint v s Display 9 F This is the perspective. In addition, the second stereoscopic display device is a display screen 90 in the horizontal direction. T Table top with 9 T Therefore, from the second viewpoint v t Table Top 9 TThis is the perspective.

[0020] As shown in FIG. 1, the stereoscopic image conversion device 1 includes a weight calculation unit 10, a viewpoint recommendation unit (viewpoint calculation unit) 11, and a drawing unit 12.

[0021] The weight calculation unit 10 calculates the weight w 0 ,w 1 Calculate the weight w 0 ,w 1 to viewpoint recommendation unit 11. Here, weight calculation unit 10 receives preset second viewpoint candidates and subjective evaluation experiment results regarding the appearance of subject 100 for the second viewpoint candidates. Weight calculation unit 10 then calculates a score for each second viewpoint candidate from the subjective evaluation experiment results using an analysis method such as the Bradley-Terry method. Furthermore, weight calculation unit 10 uses a pseudo inverse matrix to calculate weights w 0 ,w 1 After that, the weight calculation unit 10 calculates the calculated weight w 0 ,w 1 to the viewpoint recommendation unit 11. 0 ,w 1 The specific calculation method will be described later.

[0022] The viewpoint recommendation unit 11 is a display 9 F First View of v s and the weight w calculated by the weight calculation unit 10. 0 ,w 1 and a stereoscopic 3DCG scene (three-dimensional model) are input. Then, the viewpoint recommendation unit 11 selects a first viewpoint v s and the second viewpoint v t Difference in ground direction in the stereoscopic image from the first viewpoint v s and the second viewpoint v t The objective function, which is expressed as the weighted sum of the difference in appearance of the subject 100 (FIG. 8) in the stereoscopic image with the second viewpoint v, is minimized. t Then, the viewpoint recommendation unit 11 recommends (calculates) the recommended second viewpoint v t is output to the rendering unit 12. Note that the second viewpoint v tThe specific recommendation method will be described later.

[0023] The rendering unit 12 renders the second viewpoint v recommended by the viewpoint recommendation unit 11. t Specifically, the rendering unit 12 renders a stereoscopic image from the second viewpoint v t The rendering unit 12 renders the stereoscopic image by capturing a stereoscopic 3DCG scene with a virtual camera placed at the table top 9. T Output to.

[0024] <Recommendation of the second viewpoint> Hereinafter, the second viewpoint v by the viewpoint recommendation unit 11 t Here, we will explain the recommendation of weight w 0 ,w 1 is assumed to have already been calculated.

[0025] In a 3DCG scene, display 9 F First viewpoint decided for v s In this case, the viewpoint recommendation unit 11 calculates an objective function f(v s ,v t ) is minimized by setting the table top T Second viewpoint for v t I recommend it.

[0026]

number

[0027] G(v t ) is the first viewpoint v s and the second viewpoint v t This shows the difference in the ground direction in a stereoscopic image between the real world and the display 9. Here, the ground direction is the direction indicated by the normal to the ground. F is the first viewpoint v s The ground of the 3D image displayed in G r The ground in the real world G r The direction of g rIn addition, the ground in the virtual world, that is, the table top 9 T is the second viewpoint v t The ground of the 3D image displayed in G t The ground G in the virtual world t The direction of g t Therefore, the angle G(v t ) is the real-world ground plane G r Direction of g r and the ground G in the virtual world t Direction of g t It becomes the angle with. w 0 The angle G(v t ) weight.

[0028] R(v s ,v t ) is the first viewpoint v s and the second viewpoint v t This difference in appearance of the object 100 in a 3D image is expressed as R(v s ,v t ) is expressed by information on M faces constituting the three-dimensional model of subject 100, as shown in the following equations (2) and (3). w 1 is the difference in appearance of the subject 100 R(v s ,v t ) weight.

[0029]

number

[0030] R(v s ,v t ) is a feature vector c(v s ) and feature vector c(v t) Furthermore, c(v) in equation (3) is a feature vector related to the area of ​​each face constituting subject 100 at viewpoint v. Specifically, c(v) in equation (3) is a feature vector listing the proportion of the area of ​​each face constituting subject 100 to the total area of ​​subject 100 at viewpoint v. In other words, equation (3) expresses how the appearance of each face constituting subject 100 changes at viewpoint v.

[0031] Here, A i represents the area of ​​each face (each polygon) constituting the subject 100, and is a unique value for the subject 100 regardless of the viewpoint v (where 1≦i≦M). As shown in FIG. 3, if the subject 100 is a cube and the three-dimensional model of the subject 100 is expressed by polygons, the subject 100 is composed of 12 polygons and the area of ​​each polygon is A 1 ~A 12 3(a) shows the subject 100 as viewed from the front side, and FIG. 3(b) shows the subject 100 as viewed from the rear side.

[0032] Also, a i represents the area of ​​each face (polygon) constituting the subject 100 projected onto a predetermined normalized image coordinate system. i represents the area of ​​each surface constituting the subject 100 when viewed from the second viewpoint v. As shown in FIG. t The virtual camera C is placed on the table top 9. T The normalized image coordinate system is the image coordinate system (two-dimensional coordinate system) of the normalized image P.

[0033] a T and A T As shown in the following equations (4) and (5), the area a i and area A i It represents the sum of each.

[0034]

number

[0035] Although the subject 100 has been described as being represented by triangular polygons, it may be represented by quadrangular polygons. In this case, the subject 100 in FIG. 3 is composed of six polygons, and the area of ​​each polygon is A 1 ~A 6 ,a 1 ~a 6 This is (not shown).

[0036] The viewpoint recommendation unit 11 calculates the objective function f(v s ,v t ) is the smallest for the second viewpoint v t For example, the viewpoint recommendation unit 11 can obtain the objective function f(v s ,v t ) is the smallest for the second viewpoint v t In addition, the viewpoint recommendation unit 11 preliminarily sets a plurality of second viewpoint candidates, and calculates the objective function f(v s ,v t ) is the smallest for the second viewpoint v t You may ask for:

[0037] <Calculating weights> Hereinafter, the weights w calculated by the weight calculation unit 10 will be referred to as 0 ,w 1 The calculation of is explained below. In equation (1), the weight w 0 ,w 1 is the difference in ground direction G(v t ) and the difference in the appearance of the subject R(v s ,v t ) are expressed as the weights w 0 ,w 1 Since is an unknown quantity, it needs to be found in advance through a subjective evaluation experiment.

[0038] For example, as shown in FIG. F is the first viewpoint v s The subject is shown the 3D image displayed by the second viewpoint candidate v t0 ,v t1In addition, we convert the two second viewpoint candidates v t0, v t1 Of the 3D images, Table Top 9 T The second viewpoint displayed by v t The subjects were asked to select which of the three-dimensional images they preferred.

[0039] In this case, the second viewpoint candidate (v t0 ,v t1 ,…,v tN ) is N, pairwise comparison is performed N(N-1) times. As shown in FIG. 6, the weight calculation unit 10 uses an analysis method such as the Bradley-Terry method to calculate the second viewpoint candidate (v t0 ,v t1 ,…,v tN ) Score S N In Figure 6, the vertical axis is the score S N The horizontal axis represents the second viewpoint candidate (v t0 ,v t1 ,…,v tN ) (where N=5).

[0040] In addition, the second viewpoint candidate (v t0 ,v t1 ,…,v tN ) may be sampled uniformly from a sphere that encloses 100 of the object. t0 ,v t1 ,…,v tN ) can reduce the amount of computation by narrowing the area of ​​the sphere sampled, such as a hemisphere or a quarter sphere, rather than the entire sphere surrounding 100 of the object.

[0041] Here, the above-mentioned formula (1) can be replaced with the following formula (6) using the score S.

[0042]

number

[0043] Then, the weight calculation unit 10 calculates the N scores (S 1 ,S2 ,…,S N ) and the angle G(v t ) and the difference in appearance of the subject 100 R(v s ,v t ) and the weight w 0 ,w 1 The pseudo-inverse matrix X + It is calculated by:

[0044]

number

[0045] The weight calculation unit 10 calculates the weight w 0 ,w 1 For example, the weight calculation unit 10 may calculate the weight w for each 3DCG scene. 0 ,w 1 By calculating t In addition, for example, the weight calculation unit 10 can accurately estimate the weight w 0 ,w 1 Calculate the weights w 0 ,w 1 By sharing the weight w 0 ,w 1 The amount of calculation can be reduced.

[0046] Then, the viewpoint recommendation unit 11 calculates the weight w 0 ,w 1 Putting this into equation (1), we obtain the objective function (v s ,v t ) to find the second viewpoint candidate (v t0 ,v t1 ,…,v tN ) to find the optimal second viewpoint v t Recommend (calculate) the following.

[0047] [Operation of 3D image conversion device] The operation of the stereoscopic video conversion device 1 will be described with reference to FIG. As shown in FIG. 7, in step S1, the weight calculation unit 10 calculates a second viewpoint candidate (v t0 ,v t1 ,…,v tN ) Score S N Furthermore, the weight calculation unit 10 calculates the score S N And the angle G(v t ) and the difference in appearance of the subject 100 R(v s ,v t ) and the weight w 0 ,w 1 Calculate.

[0048] In step S2, the viewpoint recommendation unit 11 selects the first viewpoint v s and the second viewpoint v t The difference in the ground direction in the stereoscopic image with G(v t ) and the first viewpoint v s and the second viewpoint v t Difference in appearance of subject 100 in a 3D image with s ,v t ) and the weighted sum of the objective function f(v s ,v t ) is the smallest for the second viewpoint v t Recommend (calculate) the following.

[0049] In step S3, the rendering unit 12 renders the second viewpoint v t Specifically, the rendering unit 12 renders a stereoscopic image from the second viewpoint v t The stereoscopic image is rendered by capturing a stereoscopic 3DCG scene with a virtual camera placed in the

[0050] [Actions and Effects] As described above, the stereoscopic image conversion device 1 converts the content of the original stereoscopic image into a stereoscopic image on the display 9 in such a way that the content of the original stereoscopic image is maintained as much as possible. F The best first viewpoint is on the table top 9 T This allows the stereoscopic image conversion device 1 to convert the image to the optimal second viewpoint. TSince there is no need to separately create a stereoscopic image suitable for the tabletop 9, the labor required for the creation of the stereoscopic image can be significantly reduced. T Since a stereoscopic image is generated from the second viewpoint, the intended content of the original stereoscopic image can be more easily conveyed to the viewer, and the content of the original stereoscopic image can be prevented from being significantly changed.

[0051] (Modification) Although the embodiment has been described in detail above, the present invention is not limited to the above-described embodiment, and includes design modifications and the like within the scope of the present invention.

[0052] In the above embodiment, the first stereoscopic display device is a stationary stereoscopic display device, and the second stereoscopic display device is a tabletop stereoscopic display device, but the present invention is not limited to this. For example, the second stereoscopic display device may be a terminal whose display screen direction changes according to the user's posture, such as a tablet terminal, a smartphone, or a head-mounted display. In this case, the second stereoscopic display device may measure the posture of the device body (i.e., the direction of the display screen) using a built-in acceleration sensor, and output the measured direction of the display screen to the stereoscopic image conversion device.

[0053] In the above embodiment, the stereoscopic image conversion device is described as rendering a stereoscopic image, but is not limited thereto. In other words, the stereoscopic image conversion device may output information of the second viewpoint to the outside without rendering a stereoscopic image of the second viewpoint.

[0054] In the above embodiment, the weight calculation unit uses the Bradley-Terry method, but is not limited to this. For example, the weight calculation unit may use an analysis method such as the Thurston method.

[0055] In the above embodiment, the weight calculation unit calculates the weight, but the present invention is not limited to this. For example, the weight may be preset in the viewpoint recommendation unit based on an empirical rule.

[0056] In the above embodiment, the stereoscopic image conversion device is described as an independent hardware, but the present invention is not limited to this. For example, the present invention can be realized by a program for making hardware resources such as a CPU, memory, and hard disk of a computer function as the stereoscopic image conversion device. This program may be distributed via a communication line, or may be written and distributed on a recording medium such as a CD-ROM or a flash memory. [Explanation of symbols]

[0057] 1. 3D image conversion device (viewpoint calculation device) 9 F Stationary stereoscopic display device 9 T Tabletop stereoscopic display device 10 Weight calculation section 11 Viewpoint recommendation unit (viewpoint calculation unit) 12 Drawing section 90 F display screen 90 T display screen

Claims

1. A viewpoint calculation device that calculates a second viewpoint when a stereoscopic image displayed at a first viewpoint by a first stereoscopic display device having a display screen in a predetermined direction is displayed on a second stereoscopic display device having a display screen in a direction different from that of the first stereoscopic display device, a viewpoint calculation unit that calculates the second viewpoint such that an objective function expressed as a weighted sum of a difference in ground direction in the stereoscopic image between the first viewpoint and the second viewpoint and a difference in how a subject appears in the stereoscopic image between the first viewpoint and the second viewpoint is minimized; A viewpoint calculation device comprising:

2. The viewpoint calculation unit calculates the objective function f(v ｓ , v ｔ ) is the first viewpoint v ｓ and the second viewpoint v ｔ The angle G (v ｔ ), the angle G(v ｔ ) weight w ０ , the first viewpoint v ｓ and the second viewpoint v ｔ The difference R(v ｓ , v ｔ ), and the difference in appearance of the subject R(v ｓ , v ｔ ) weight w １ In the following formula (1) [0010] 2. The viewpoint calculation device according to claim 1, wherein the viewpoint calculation device is represented as a viewpoint calculation device.

3. The viewpoint calculation unit calculates the difference in appearance of the subject R(v ｓ , v ｔ ) is the area A of each surface constituting the subject. ｉ , the area A ｉ The sum of A Ｔ , the area a of each surface constituting the subject when viewed from a viewpoint v ｉ , and the area a ｉ The sum of a Ｔ In the following formulas (2) and (3) [0025] 3. The viewpoint calculation device according to claim 2, wherein the viewpoint is represented as an image.

4. A subjective evaluation experiment result regarding the appearance of the object is input for a second viewpoint candidate that has been set in advance, and a score for each of the second viewpoint candidates is calculated from the subjective evaluation experiment result by a predetermined analysis method, and the score and the angle G(v ｔ ) and the difference in appearance of the subject R(v ｓ , v ｔ ) and the weight w ０ , w １ 4. The viewpoint calculation device according to claim 2, further comprising a weight calculation unit for calculating a weight of the viewpoint.

5. 5. The viewpoint calculation device according to claim 1, further comprising a rendering unit that renders the stereoscopic video image at the second viewpoint calculated by the viewpoint calculation unit.

6. the first stereoscopic display device is a stationary stereoscopic display device having a vertical display screen, 6. The viewpoint calculation device according to claim 1, wherein the second stereoscopic display device is a table-top type display device having a horizontal display screen.

7. A program for causing a computer to function as the viewpoint calculation device according to any one of claims 1 to 6.

Images