Method for determining a three-dimensional projection function and program for executing this method

Abstract

To provide a method for determining three-dimensional projection functions for estimating projection functions from an image including a graphic method in which parallel lines in a space do not converge to one vanishing point, and also to provide and a drawing method in which a 3D model can be projected on a background image without any sense of incongruity.SOLUTION: Disclosed is a method for determining three-dimensional projection functions from a background image such as an animation, a drama, a game, a cartoon, or the like. This method includes at least the steps of: displaying a virtual figure on the background image; causing a user to correct the displayed figure according to the background image; and estimating parameters for controlling the three-dimensional projection functions based on equations established from similarity relations with coordinates of vertexes of the corrected figure. Also, a 3D model is rendered on the background image using the three-dimensional projection functions determined in the method for determining the three-dimensional projection functions.SELECTED DRAWING: Figure 15

Description

Technical Field

[0001] The present invention relates to a program for executing a method for determining a three-dimensional projection function from background images such as animations, dramas, games, and comics. Method and and This methods.

Background Art

[0002] In recent animation production, both hand-drawn and 3DCG are often used. Original drawings depicting background paintings, characters, etc., and 3D models are produced separately, and the final screen is completed by synthesizing them. At the previous stage, there is a process called layout in which, based on the storyboard, the camera's field of view, movement, and the placement of characters in that cut are specified. By sharing this specification among the person in charge of the background painting, the person in charge of the original drawing, and the person in charge of the 3D model, the final screen can be created without causing any deviation.

[0003] The background painting is produced based on the background original drawing created in the layout process. If the projection function can be obtained from the projection drawing method used in the background original drawing, it will be possible to project the characters created as 3D models onto the hand-drawn background painting without a sense of incongruity, which is expected to lead to support for animation production.

[0004] Background paintings depicting the inside of buildings, cityscapes, etc. often follow the Manhattan world assumption. The Manhattan world assumption is the assumption that many of the line segments that make up artificial objects are parallel or orthogonal to each other in three-dimensional space. There are multiple studies on obtaining the projection function and camera pose for the projected background painting based on the Manhattan world assumption.

[0005] For example, Zhang et al. proposed a method in Non-Patent Document 1 that uses the Manhattan world constraint that line segments are orthogonal or parallel to each other at the model estimation stage based on the RANSAC method, and showed that it is superior to existing studies in both accuracy and computational efficiency. Furthermore, Zhang et al. proposed a method in Non-Patent Document 2 to map the line segments projected onto the screen onto the unit sphere so that the solutions of the model estimation equations can be obtained for line segments parallel to the screen, and solve the model estimation equations on the unit sphere. Also, a method for estimating the vanishing point together with the focal length was shown for photographs taken by perspective projection even when the focal length is unknown. In the case of general perspective projection, since the angle of view or the focal length is the only parameter of the projection function, the estimation of the focal length directly leads to the estimation of the projection function.

[0006] These studies target background paintings using one-point perspective, two-point perspective, and three-point perspective that can be interpreted as general perspective projections. In the background paintings using these perspective methods, the extension lines of line segments parallel in 3D space converge to one vanishing point. On the other hand, in hand-drawn animation production, due to the characteristic that the artist can draw freely without being restricted by 3D constraints, expressions that cannot be interpreted by these perspective methods may be used. For example, there are background paintings drawn using a drawing method where the vanishing point is not determined to be one.

[0007] Also, studies have been conducted to reconstruct a pseudo 3D model by the user giving auxiliary input to a single picture and generate an animation with freely movable camera position and angle. However, since these methods do not estimate the projection function of the background painting, when projecting a 3D model of a character onto the background painting, a projection function different from that of the background painting will be used for the projection of the 3D model of the character.

[0008] In Non-Patent Document 3, "Tour Into the Picture" by Horry et al. proposed a method for pseudo-reconstructing a three-dimensional scene model from a single image by having the user provide auxiliary input, as well as an interface therefor. In this method, the input image is divided into a background and foreground objects other than the background, a pseudo three-dimensional scene model is reconstructed, and an animation is generated by moving the position and orientation of the camera. The interface proposed by Horry et al. is called Spidery Mesh and is used to determine the parameters for reconstructing the scene model. The user can perform three operations on Spidery Mesh: deformation of the inner rectangle, translation of the inner rectangle, and translation of the vanishing point. The reconstructed scene model of the background part has a shape consisting of five faces: left, right, top, bottom, and back, as shown in FIG. 1.

[0009] In Non-Patent Document 4, Kubo et al. focused on the feature of ukiyo-e that the intersections of parallel lines in the depth direction converge to different vanishing points on the left and right sides of the screen, and called such a drawing method the ukiyo-e-like drawing method. They proposed a method that can reconstruct a pseudo 3D scene model from the background paintings drawn using this drawing method. In this method, a scene model consisting of five planes, namely left, right, top, bottom, and back, is reconstructed from a single background painting by the method shown in FIG. 2. In FIG. 2, point E is the position of the camera, points A, B, C, and D are the values input by the user from the interface, and points A', B', C', and D' are the vertices of the right side of the box of the scene model. At this time, the coordinates of points A', B', C', and D' in the three-dimensional space can be obtained from the constraint that points A', B', C', and D' are respectively on the extension lines of EA, EB, EC, and ED, and the constraint that the figure A', B', C', D' is a rectangle. Similarly, the coordinates of the vertices can be obtained for the left side of the box, and the scene model can be reconstructed. When reconstructing a scene model from a background painting using a general perspective drawing method by using this method, as shown in FIG. 3(a), the upper, lower, left, and right faces of the box are in an orthogonal or parallel relationship to each other, and it has a shape like a rectangular parallelepiped. On the other hand, the scene model reconstructed from a background painting using the ukiyo-e-like drawing method has a shape in which the left and right faces and the upper and lower faces are not parallel to each other, as shown in FIG. 3(b).

[0010] By placing the 3D model of the character in the box of the scene model reconstructed using these methods and projecting it by the perspective projection function, the hand-drawn background painting and the 3D model character can be synthesized. However, in these methods, since the estimation of the projection function used for projecting the background painting is not performed, the perspective projection function for projecting the 3D model is different from the projection function corresponding to the drawing method used in the background painting. Therefore, for example, when creating an animation in which the 3D model moves in the depth direction, the 3D model becomes smaller as it moves away from the screen, but the sense of perspective does not match the sense of perspective of the background painting, and the composition may be unnatural.

Prior Art Documents

Non-Patent Documents

[0011] [Non-Patent Document 1] Lilian Zhang and Reinhard Koch. Vanishing points estimation and line classification in a manhattan world. In Kyoung Mu Lee, Yasuyuki Matsushita, James M. Rehg, and Zhanyi Hu, editors, Computer Vision - ACCV 2012, pp. 38-51, Berlin, Heidelberg, 2013. Springer Berlin Heidelberg. [Non-Patent Document 2] Lilian Zhang, Huimin Lu, Xiaoping Hu, and Reinhard Koch. Vanishing point estimation and line classification in a manhattan world with a unifying camera model. Int. J. Comput. Vision, Vol. 117, No. 2, p. 111-130, April 2016. [Non-Patent Document 3] Y. HORRY. Tour into the picture: Using a spidery mesh interface to make animation from a single image. Proc. SIGGRAPH ’97 (Los Angels, California, August 3-8,1997), pp. 225-232, 1997. [Non-Patent Document 4] Yuka Kubo, Zhao Jie, and Koichi Hirota. A method for transformation of 3d space into ukiyo-e composition. In ACM SIGGRAPH ASIA 2008 Artgallery: Emerging Technologies, SIGGRAPH Asia ’08, p. 29-35, New York, NY, USA, 2008. Association for Computing Machinery.

Non-Patent Document 5

Non-Patent Document 6

Non-Patent Document 7

Summary of the Invention

Problems to be Solved by the Invention

[0012] The present invention has been made in consideration of the above-mentioned problems, and provides a method for determining a three-dimensional projection function that estimates a projection function from an image that includes a projection in which parallel lines in space do not converge to a single vanishing point, and a drawing method that can project a 3D model onto a background image without creating an awkward appearance. [Means for solving the problem]

[0013] One aspect of the present invention is a method for determining a three-dimensional projection function from a background image of an animation, a play, a game, a cartoon, etc., the background image being a projection function of a three-dimensional image in a space. extends in the same depth direction The image includes a projection method in which parallel lines do not converge to a single vanishing point, and includes at least a step of displaying a virtual figure on a background image, a step of a user correcting the displayed figure to match the background image, and a step of estimating parameters for controlling a three-dimensional projection function based on an equation formulated from the coordinates of the vertices of the corrected figure and a similarity relationship. and the parameters are four parameters: τ that controls the viewing angle, α that controls the shear in the x direction, β that controls the shear in the y direction, and s that expands and contracts the space in the y direction according to the distance from the projection reference plane .

[0014] Another aspect of the present invention is a method for determining a three-dimensional projection function from a background image of animation, theater, games, manga, etc., which includes at least a step in which a user corrects an object on the background image to match the background image, a step in which the corrected object corresponds to a virtual figure, and a step in which parameters that control the three-dimensional projection function are estimated based on an equation formulated from the coordinates of the vertices of the virtual figure and a similarity relationship.

[0015] In one embodiment of the present invention, the virtual figure satisfies all of (1) to (3). (1) It has at least four vertices that satisfy the following conditions: (1-1) The ratio of the lengths of the sides of the figure connecting each of the four vertices is known, and the angle between two adjacent sides is also known (see Figure 8(A)). Or, the ratio of the lengths of the sides of the figure connecting all the points is known (see Figure 8(B)). (1-2) They are on the same plane parallel to the ground (xz plane) on the background image. (2) It has three pairs of two vertices that satisfy the following conditions. However, if the figure in (1) includes sides parallel to the screen, two pairs are sufficient. (2-1) For all pairs, the line connecting the two vertices is a vertical line. (2-2) For each pair, the vertical line connecting the two vertices satisfies the following conditions. (2-2-1) They all have the same length. (2-2-2) There are vertices of the figure in (1) on the extension line, and it is known which extension line of which vertical line intersects which vertex. (2-2-3) They all have different depths in the z direction. (2-2-4) When the proviso in (2) is not satisfied, the length of the vertical line is known, or the depth in the z direction of any one of the vertical lines is known. (3) The ratio of the length of each side of the figure in (1) to the length of the vertical line in (2) is known.

[0016] Also, in one aspect of the present invention, the figure can be a cube.

[0017] Also, in one aspect of the present invention, the three-dimensional projection function is a transformation by a parameter s that expands and contracts the space in the y-axis direction (x,y,z,1) t →(x,y+y(z+λ)s,z,1) t and a projection matrix P represented by the following formula

Equation

[0018] In addition, in one aspect of the present invention, when each vertex of the lower surface of the cube is A, E, F, B in clockwise order, and each vertex of the upper surface is C, G, H, D in clockwise order as well, in the step where the user corrects the figure, the figure is automatically corrected according to the user's operation so as to satisfy (Condition 1) or both (Condition 1) and (Condition 2). (Condition 1) AC, BD, EG, and FH are vertical lines. (Condition 2) When the intersection points of the extension lines of AE and BF, CG and DH, DH and BF, and CG and AE are denoted as P1, P2, P3, and P4 respectively, P1P2 is a vertical line and P3P4 is a horizontal line.

[0020] Another aspect of the present invention is a program for causing a computer to execute the method for determining the above-described three-dimensional projection function. the method It is a program to be executed by a computer.

Advantages of the Invention

[0021] As described above, according to the present invention, the projection function can be estimated from an image including a drawing method in which parallel lines in space do not converge to one vanishing point, and a 3D model can be projected onto the background image without a sense of incongruity.

Brief Description of the Drawings

[0022]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

[0023] Hereinafter, preferred embodiments of the present invention will be described in detail. Note that the embodiments described below do not unduly limit the content of the present invention described in the claims, and not all of the configurations described in the embodiments are necessarily essential as the solution means of the present invention. The embodiments described below do not unduly limit the content of the present invention described in the claims, and not all of the configurations described in the embodiments are necessarily essential as the solution means of the present invention. Not all of the configurations described in the embodiments are necessarily essential as the solution means of the present invention.

[0024] <Integrated Projection Method> In order to realize, by 3DCG, an expression using a drawing method that cannot be explained by a general perspective drawing method, mainly two methods have been proposed: a method of deforming the 3D model itself and a method of changing the projection function. As a method related to the present invention, a method of changing the projection function will be described.

[0025] In Non-Patent Documents 5 and 6 above, Yoshimura et al. proposed an integrated projection method that combines perspective projection, inverse perspective projection, orthographic projection, and oblique projection as a method for realizing a projection having two characteristics of linearity and verticality in addition to the characteristic that the left and right sides of the screen each have a vanishing point, which was described by Kubo et al. (Non-Patent Document 4) as a characteristic of a ukiyo-e-like drawing method. Linearity refers to the property that a straight line in three-dimensional space is preserved as a straight line after projection, and verticality refers to the property that a vertical line in three-dimensional space is preserved as a vertical line after projection.

[0026] A drawing method that can be interpreted as a pseudo multi-viewpoint projection is also used in the background paintings of modern hand-drawn animations. Therefore, an object of the present invention is to estimate the projection function of the integrated projection method from a background painting using such a drawing method.

[0027] In the integrated projection method proposed by Yoshimura et al., pseudo multi-viewpoint projection is realized by enabling independent movement of the four upper, lower, left, and right sides of the view volume. In this integrated projection method, a plane called the projection reference plane is used as the reference for projection. The projection reference plane is parallel to the screen and exists between the near clipping plane and the far clipping plane of the view volume. In the projection function, four parameters are used: τ for controlling the camera's field of view, α for controlling the shear in the x direction, β for controlling the shear in the y direction, and s for stretching and shrinking the space in the y direction according to the distance from the projection reference plane. When the user changes these four values, the shape of the view volume is changed so that the position of the point intersecting the projection reference plane does not change on the screen.

[0028] In the method of Yoshimura et al., before the projection transformation by the projection matrix, transformation by the parameter s is performed. In this transformation, the point (x, y, z, 1) in the camera coordinate system t is moved to (x, y + y(z + λ)s, z, 1). t This is a transformation such that the space stretches and shrinks by s in the y direction as the value of |z + λ| increases (gets farther from the projection reference plane).

[0029] In the projection matrix, the parameters τ, α, β are used. These parameters are defined as follows. For the view volume sheared in the x direction as shown in Fig. 4(a), α is defined such that the shear angle is arctan(α). Similarly, for the shear in the y direction, β is defined such that the shear angle is arctan(β). τ is defined such that the field of view angle is 2 arctan(τ) as shown in Fig. 4(b). That is, when the field of view angle is FoV, τ = tan(FoV / 2).

[0030] In the method proposed by Yoshimura et al., different projection matrices are used for the cases of 0 < τ, τ = 0, and τ < 0. For 0 < τ, it is a projection that extends the perspective projection, for τ = 0, it is a projection that extends the parallel projection, and for τ < 0, it is a projection that extends the inverse perspective projection.

[0031] In the background paintings of hand-drawn animations targeted by the present invention, since expressions such as parallel projection and reverse perspective projection are rarely used, the estimation of the viewing angle FoV is performed in the range of 0 < τ. Therefore, the case of 0 < τ will be described here.

[0032] When 0 < τ, the projection matrix P is defined as follows, where W is the horizontal width of the screen, H is the vertical height of the screen, N is the distance from the origin of the screen, F is the distance from the origin of the far clipping plane of the view volume, and λ is the distance from the origin of the projection reference plane.

Equation

[0033] In the projection by this projection matrix P, when α, β, and τ are changed, in addition to the change in the normal shear amount and the viewing angle, as shown in FIG. 5 ((a) shows the change in the shear amount, (b) shows the change in the focal length), corrections are added so that the position and size of the projection reference plane do not change. The matrix on the left side of the projection matrix P changes the viewing angle, and the matrix on the right side corrects the change in the shear amount and the changes in the viewing angle and the shear amount.

[0034] <Method for determining three-dimensional projection function> A method for determining a three-dimensional projection function according to the present invention will be described. One aspect of the present invention is a method for determining a three-dimensional projection function from background images such as animations, dramas, games, and comics, including at least a step of displaying a virtual figure on the background image, a step of the user correcting the displayed figure to match the background image, and a step of estimating parameters for controlling the three-dimensional projection function based on equations established from the coordinates of the vertices of the corrected figure and the similarity relationship.

[0035] Alternatively, another aspect of the present invention is a method for determining a three-dimensional projection function from a background image such as an animation, a drama, a game, a comic, etc., including at least a step of a user correcting an object on the background image to match the background image, a step of associating the corrected object with a virtual figure, and a step of estimating parameters for controlling the three-dimensional projection function based on an equation established from the coordinates of the vertices of the virtual figure and a similarity relationship. In this case, the user corrects (deforms) an object (for example, a character, etc.) on the background image, and then associates the corrected object with a virtual figure. One example of the association method includes assuming a bounding box that includes the object, but other methods may also be used. In the following description, an example in which the user corrects a virtual figure displayed on the background image will be described, but according to this technique, it is not always necessary for the user to directly correct the virtual figure.

[0036] The user assumes a figure in the three-dimensional space drawn on the background image for which the projection function is to be estimated, and draws the projected figure of the figure on the screen. From the projected image of the drawn figure, four values, τ, α, β, s, for controlling the projection of the integrated projection method are estimated, and the projection function is determined.

[0037] The interfaces of the method according to the present invention are shown in FIGS. 6 and 7. In one aspect of the present invention, when estimating the projection function, a virtual figure (FG) is displayed on the background image (BG) (FIG. 6). The virtual figure (FG) may be automatically displayed on the background screen (BG) in advance, or the user may be able to set the initial position. Thereafter, the user moves the positions of the vertices displayed on the screen and corrects the figure (FG) to match the background image (BG) (for example, along the contour in the depth direction) (FIG. 7). In one aspect of the present invention, the projection function is estimated each time the user moves a vertex, and a radial broken line (AL) using the estimated projection function may be projected onto the screen so that the user can easily recognize the estimated projection function. The user moves the vertices of the figure (FG) until the intended projection function is obtained and interactively adjusts the parameters of the projection function. After the projection function is estimated, the virtual figure (FG) and the radial broken line (AL) can be deleted from the background screen (BG) (set to a non-display state).

[0038] The virtual figure used in the method for determining the three-dimensional projection function according to the present invention is characterized by satisfying all of (1) to (3). (1) It has at least four vertices that satisfy the following conditions. (1-1) The ratio of the lengths of the sides of the figure connecting each of the four vertices is known, and the angle formed by two adjacent sides is also known (see FIG. 8(A)). Alternatively, the ratio of the lengths of the sides of the figure connecting all the points is known (see FIG. 8(B)). (1-2) It lies on the same plane parallel to the ground (xz plane) on the background image. (2) It has three pairs of two vertices that satisfy the following conditions. However, if the figure in (1) includes a side parallel to the screen, two pairs are sufficient. (2-1) For all pairs, the line connecting the two vertices is a vertical line. (2-2) For each pair, the vertical line connecting the two vertices satisfies the following conditions. (2-2-1) Their lengths are equal. (2-2-2) There is a vertex of the figure in (1) on the extension line, and it is known which vertical line extension intersects which vertex. (2-2-3) The depths in the z direction are all different. (2-2-4) When the conditions of the proviso (2) are not satisfied, the length of the vertical line is known, or the depth in the z direction of any one of the vertical lines is known. (3) The ratio of the length of each side of the figure in (1) to the length of the vertical line in (2) is known.

[0039] Examples of figures satisfying such conditions are shown in FIGS. 8(C) and (D). In FIGS. 8(C) and (D), the sides connecting the vertices satisfying the above condition (1) are drawn as dotted lines, and the vertical lines connecting the two vertices for each pair of two vertices satisfying the condition (2) are drawn as solid lines. Hereinafter, as an example of a figure satisfying such conditions, a cube will be taken as a premise for explanation. However, for any figure satisfying the above conditions, such as a rectangular parallelepiped other than a cube, it is also possible to estimate the parameters for controlling the three-dimensional projection function based on the equations established from the vertex coordinates and similarity relationships of the figure.

[0040] The user assumes a cube in a three-dimensional space that satisfies the following conditions and draws its projection image on the screen. · The cube is placed horizontally with respect to the ground (xz plane). The size, position, and rotation angle with respect to the screen of the cube are free. An example of a cube in a three-dimensional space that satisfies the conditions is shown in FIG. 9.

[0041] The projection image by the integrated projection method proposed by Yoshimura et al. has the property that the vertical line is maintained before and after projection, and the property that the intersection points (vanishing points) of the parallel lines in the space after projection are arranged horizontally and vertically. When the vertices of the cube are placed as A, ···, H as shown in FIG. 9, when the projection image of the cube is drawn correctly, the following two constraints are satisfied. 1. AC, BD, EG, FH are vertical lines 2. Let the intersection points of the extension lines of AE and BF, CG and DH, DH and BF, and CG and AE be P1, P2, P3, and P4 respectively. Then, P1P2 is a vertical line and P3P4 is a horizontal line. To appropriately estimate the projection function, it is desirable that the input figure satisfies these constraints. Therefore, a correction method for two user input images is introduced. The figure corrected by correction method 1 satisfies constraint 1, and the figure corrected by correction method 2 satisfies constraint 1 and constraint 2. Also, to avoid deformation that the user does not intend as much as possible, when correcting, move the adjacent points of the point moved by the user.

[0042] As an example, when the user moves vertex A to the position of point A’, it is shown in Fig. 10. However, in Fig. 10, the solid line is the figure before correction, the dashed line is the figure after correction by correction method 1, and the dash-dotted line is the figure after correction by correction method 2.

[0043] The two corrections are performed as follows.

[0044] (Correction method 1) Move vertex C parallel to the x-axis to the position of point C’ so as to satisfy constraint 1, that is, C′ x = A′ x .

[0045] (Correction method 2) In addition to correction method 1, adjust the y-coordinates of the other two points B and E of the adjacent points of the vertex moved by the user so as to satisfy constraint 2, that is, P 1x = P 2x ∧P 3y = P 4y . At this time, the coordinates of (B′ y , E′ y ) after correction are obtained as follows. )

[0046] P1, P2, P3, and P4 can be expressed as follows using real numbers t1, ···, t8 and points A to H.

Equation

[0047] By solving Equation (3.1) for t1, ···, t8, the coordinates of P1, P2, P3, and P4 can be expressed in terms of the coordinates from A to H. Let this be condition P 1x =P 2x Substituting into gives a linear equation in B′ y , E′ y Substituting condition P 3y =P 4y into gives a quadratic equation in B′ y , E′ y Solving these equations yields two sets of solutions for (B′ y , E′ y ), so we adopt one of the solutions. The same correction is made when the user moves vertices other than point A.

[0048] A method for estimating four values that control the projection function from the coordinates of each vertex of the input cube is shown. The horizontal width W and vertical height H of the screen are determined from the input background image. The distance N from the origin of the screen and the distance λ from the origin of the projection reference plane are fixed values. Also, in the view volume of the integrated projection method, the projection reference plane is at the same position as the screen, i.e., λ = N.

[0049] Also, among the values that control the function, since τ is obtained as τ = W / 2f using the focal length f of the camera and the horizontal width W of the screen, we will find the focal length f instead of τ below. Below, for point A, the coordinates on the screen after projection are represented as (A x , A y ), and the coordinates of points B, ···, H are represented similarly.

[0050] The view volume in the integrated projection method for a projection with a focal length of f and shear amounts of α and β in the x and y directions, respectively, can be represented as in Figure B of Figure 11. However, in Figure 11, λ = N is assumed. This is obtained by translating the frustum of a perspective projection (Figure G of Figure 11) with a focal length of f and shear amounts of α and β in the x and y directions, respectively, parallel to the x, y, and z axes as shown by arrow 1 so that the distance from the origin of the projection reference plane is λ and the center of the projection reference plane intersects the z - axis.

[0051] In the method according to the present invention, consider a figure obtained by translating the view volume (figure B in Fig. 11) in the integrated projection method in the opposite direction to the position of figure G in Fig. 11 as shown by arrow 2, and assume that the cube in the three-dimensional space corresponding to the input cube is also at the position of the frustum (figure G). Also, let the coordinates of vertex A of the cube in the three-dimensional space be (A * x , A * y , A * z ), and represent the coordinates of points B, ···, H in the same way.

[0052] (Estimation of f, α, s) First, obtain the focal length f of the camera, the shear amount α in the x direction, and the value s for stretching and shrinking the spatial y-axis direction according to the distance from the projection reference plane.

[0053] The square (figure G) in Fig. 12 is a view of the cube in the three-dimensional space corresponding to the input cube seen from the y direction. The input cube is reduced while remaining similar so that point A touches the screen as in the square (figure B) in Fig. 12. At this time, since the x component of the projection image does not change before and after reduction, the square (figure B) can be considered as the view of the cube in the three-dimensional space seen from the y direction instead of the square (figure G) corresponding to the input cube.

[0054] Let the length of one side of the input cube be L, the length of one side of the square (figure B) be l, and the z coordinate of point A of the input cube be A * z . Then, the following holds from Fig. 12.

Equation

[0055] Also, let the rotation angle of the cube (the angle formed by side AB and the x-axis) be θ (0 ≤ θ < π / 2), then , the square (figure B) becomes as shown in Fig. 13. At this time, from Fig. 13, the following holds from the similarity relationship of the figures.

Equation

[0056] Considering points E and F in the same way, the following system of simultaneous equations holds for the unknowns f, α, and θ.

Equation

[0057] Also, for the cube in three-dimensional space corresponding to the input cube, the views of sides AC, BD, and FH from the x-direction are shown in Fig. 14. However, after performing the transformation by s on points A, C, B, D, F, and H, sides BD and FH are represented by thick lines, and the y-coordinates of points A, C, B, D, F, and H after the transformation by s are denoted as A ** y , C ** y , B ** y , D ** y , F ** y , H ** y respectively.

[0058] From Fig. 14, for side AC, the following holds due to the similarity relationship of the figures.

Equation

[0059] Also, at this time, C ** y - A ** y = L(1 + (f + A * z ))s), so it can be transformed as follows.

Equation

[0060] Considering sides BD and FH in the same way, the following holds.

Equation

[0061] At this time, B * z = A * z -Lsinθ and F * z = A * z -L(cosθ + sinθ), so it can be transformed as follows.

Mathematics

[0062] Here, for the rotation θ of the cube, it is determined whether θ = 0, and a case-by-case analysis is performed. When θ = 0, the sides AB, CD, EF, and GH are horizontal lines (parallel to the x-axis) on the screen. Therefore, by examining the slopes of these sides of the input figure, it is possible to determine whether θ = 0.

[0063] (i) When θ = 0 Since side AB touches the screen, l = A x - B y That is. Therefore, l = A x - B y And Substitute θ = 0 into Equation (3.4) and solve for f and α, a unique solution can be obtained.

[0064] Next, find s. Substitute θ = 0 into Equations (3.2), (3.6), and (3.9), and solve them as a system of simultaneous equations for the unknowns s, A * z , L, and a unique solution can be obtained.

[0065] (ii) When θ > 0 From Equation (3.2), l = -fL / A * z That is. Substitute this into Equation (3.4), and the following holds.

Mathematics

[0066] For the six equations of Equation (3.10) and the aforementioned Equations (3.6) and (3.9), by performing optimization with these as the objective functions, f, α, and s can be obtained. At this time, the unknowns are a total of five, which are four of f, α, s, θ plus either L or A. * z For the optimization, the BOBYQA method, which is a constrained non-linear optimization method, is used.

[0067] From the above, the projection parameters f, α, and s can be estimated.

[0068] (Estimation of β) From Figure 14, let the y-coordinate of point A in three-dimensional space be A * y , and the y-coordinate after conversion by s be A ** y . Then A * y can be expressed as follows.

Equation

[0069] Similarly, let the y-coordinate of point F in three-dimensional space be F * y . Then F * y can be expressed as follows.

Equation

[0070] Therefore, since the cube is placed horizontally in three-dimensional space, that is, A * y = F * y under the condition, the following holds.

Equation

[0071] The estimated f, α, s and the known θ, L, A * zUsing this equation, it can be solved for β.

[0072] FIG. 15 is a flowchart showing the processing of the method for determining a three-dimensional projection function according to an embodiment of the present invention described so far. By such a process, the four values f, α, β, and s necessary for the projection function can be obtained from the coordinates of the vertices of the input cube.

[0073] <Drawing method> If a three-dimensional projection function is estimated by the method for determining a three-dimensional projection function according to the present invention described above, a 3D model can be drawn on a background image using the projection function. As a result, a character created as a 3D model can be projected onto a hand-drawn background image without a sense of incongruity.

[0074] <Program> Each step described in the method for determining a three-dimensional projection function and the drawing method according to the present invention described above can be calculated and realized by executing a program stored in a memory. These processes can be executed by a computer having a general hardware configuration. The program can be stored in a storage medium such as a RAM, ROM, USB memory, or DVD.

Example

[0075] Hereinafter, the present invention will be described more specifically using examples, but the present invention is not limited to the following examples at all.

[0076] (Example 1) Three users each estimated a projection function 10 times using the interface of the method for determining a three-dimensional projection function according to an embodiment of the present invention, and evaluated the error and standard deviation of the four control values of the projected projection function. At that time, the figure used for estimating the projection function was a cube. However, among the control values, since the error range varies greatly depending on the true value of the focal length f, the angle of view FoV was used as the evaluation target instead of f. FoV can be expressed as FoV = 2 arctan (W / 2f) using the focal length f and the horizontal width W of the screen.

[0077] As the input image, the image with a resolution of 800×600 [pixel] shown in FIG. 16 was used. This is an image projected by the projection function of the integrated projection method where the control values are FoV = 90°, α = 0.39, β = -0.06, and s = 0.05.

[0078] Table 1 shows the average error and standard deviation from the true values used for the projection of the original background painting for FoV calculated from the estimated α, β, s, and f. Also, FIG. 17 shows the images projected using each projection function estimated by the input of User 1 with respect to the dashed lines arranged in the depth direction in the space.

[0079] [Table 1]

[0080] Although the projected images using the estimated projection functions approximately match, there are also variations. From Table 1, it can be seen that these estimated projection functions produce projections with a larger standard deviation of the shear amount α in the horizontal direction of the screen compared to the shear amount β in the vertical direction of the screen. This is manifested in FIG. 17 as a tendency for the vanishing points of the parallel lines extending into the depth in the vertical direction of the screen to be almost constant, while the vanishing points of the parallel lines extending into the depth in the horizontal direction of the screen show a tendency to be projected with fluctuations.

[0081] (Example 2) By synthesizing the background painting and the 3D model, a plurality of images were created and evaluated such that the 3D model appears to move on the floor depicted in the background painting. A humanoid model (see Non-Patent Document 7) was used as the 3D model. Also, the image shown in FIG. 16 was used as the background painting.

[0082] The images synthesized with the background painting and the 3D model were created in the following three ways. 1. Project the 3D model onto the background painting using the true projection function used for the projection of the background painting. 2. Using the projection function estimated by the method for determining a three-dimensional projection function according to an embodiment of the present invention, a 3D model is projected onto the background image. As the projection functions, three of the projection functions estimated by the experiment conducted in Example 1 are used. Let these projection functions be (a), (b), and (c) respectively, and Table 2 shows the errors between the control values of the projection functions and their true values. 3. Reproduce the pseudo 3D scene model proposed by Kubo et al. according to Non-Patent Document 4, and project the 3D model using a general perspective projection function. However, the viewing angle of the perspective projection is set to 90°. Also, as shown in FIG. 18, for the scene model, a texture obtained by dividing the background image is created and pasted inside a rectangular parallelepiped 3D model, and the left and right side faces are rotated to reproduce it.

[0083] In Methods 1 and 2, the 3D model moves in the depth direction. Also, the x and y coordinates of the 3D model were manually adjusted while visually checking the projection image. In Method 3, the y coordinate was determined so that a humanoid 3D model is placed on the floor of the scene model reconstructed from the background image, and the x and z coordinates were manually adjusted to move along the pattern of the floor.

[0084] The image created by Method 1 is shown in FIG. 19(A), the images created by Method 2 are shown in FIGS. 19(B), (C), and (D), and the image created by Method 3 is shown in FIG. 19(E).

[0085] From Table 2 and FIGS. 19(A) to (D), it can be seen that even when there is an error in the estimation of the projection function using the proposed method, a result image close to the projection of the 3D model when using the true projection function can be obtained. Also, looking at FIG. 19(E), which is the result using the scene model proposed by Kubo et al., when comparing the size of the building drawn on the background image and the size of the projection image of the humanoid 3D model, it can be seen that as the humanoid 3D model moves in the depth direction, the relative size of its projection image with respect to the building changes. On the other hand, in FIGS. 19(B) to (D), which are images created by the method for determining a three-dimensional projection function according to an embodiment of the present invention, the ratio of the size of the projection image of the humanoid 3D model to the size of the building is almost constant.

[0086]

Table 2

[0087] Although an embodiment of the present invention has been described in detail as above, those skilled in the art will easily understand that many modifications can be made without substantially departing from the novel matters and effects of the present invention. Therefore, all such modified examples are intended to be included within the scope of the present invention.

[0088] For example, in the specification or drawings, a term described at least once together with a broader or synonymous different term can be replaced with that different term anywhere in the specification or drawings. Also, the method for determining the three-dimensional projection function, the drawing method, and the configuration and operation of the program for executing these methods are not limited to those described in one embodiment of the present invention, and various modified implementations are possible.

Claims

1. A method for determining a three-dimensional projection function from a background image such as an animation, a drama, a game, or a comic, comprising: The background image is an image including a drawing method in which parallel lines extending in the same depth direction in space do not converge to one vanishing point. At least, A step of displaying a virtual figure on the background image; A step of a user correcting the displayed figure to match the background image; Based on an equation established from the coordinates of the vertices of the corrected figure and a similarity relationship, a step of estimating parameters for controlling the three-dimensional projection function is included, The parameters are four parameters of τ for controlling the angle of view, α for controlling the shear in the x direction, β for controlling the shear in the y direction, and s for expanding and contracting the space in the y direction according to the distance from the projection reference plane, and the method for determining the three-dimensional projection function is characterized by this.

2. A method for determining a three-dimensional projection function from a background image such as an animation, a drama, a game, or a comic, comprising: At least, A step of a user correcting an object on the background image to match the background image; A step of corresponding the corrected object to a virtual figure; Based on an equation established from the coordinates of the vertices of the virtual figure and a similarity relationship, a step of estimating parameters for controlling the three-dimensional projection function is included, and the method for determining the three-dimensional projection function is characterized by this.

3. The method for determining a three-dimensional projection function according to claim 1 or claim 2, wherein the virtual figure satisfies all of (1) to (3). (1) It has at least four vertices that satisfy the following conditions. (1-1) The ratio of the lengths of the sides of the figure connecting each of the four vertices is known, and the angle formed by two adjacent sides is also known. Or, the ratio of the lengths of the sides of the figure connecting all the points is known. (1-2) It is on the same plane parallel to the ground (xz plane) on the background image. (2) It has three pairs of two vertices that satisfy the following conditions. However, if the figure in (1) includes sides parallel to the screen, two pairs are sufficient. (2-1) For all pairs, the line connecting the two vertices is a vertical line. (2-2) For each pair, the vertical line connecting the two vertices satisfies the following conditions. (2-2-1) They all have the same length. (2-2-2) There are vertices of the figure in (1) on the extension line, and it is known which extension line of which vertical line intersects which vertex. (2-2-3) The depths in the z direction are all different. (2-2-4) When the condition of the proviso in (2) is not satisfied, the length of the vertical line is known, or the depth in the z direction of any one of the vertical lines is known. (3) The ratio of the length of each side of the figure in (1) to the length of the vertical line in (2) is known.

4. The method for determining the three-dimensional projection function according to claim 3, wherein the figure is a cube.

5. The three-dimensional projection function is a transformation by a parameter s that expands and contracts the space in the y-axis direction (x, y, z, 1) ｔ → (x, y + y(z + λ)s, z, 1) ｔ and a projection matrix P represented by the following formula [Equation 1] (However, W is the horizontal width of the screen, H is the vertical height, N is the distance from the origin of the screen, F is the distance from the origin of the far clip plane of the view volume, and λ is the distance from the origin of the projection reference plane) consisting of estimating four parameters: τ for controlling the viewing angle, α for controlling the shear in the x direction, β for controlling the shear in the y direction, and s for expanding and contracting the space in the y direction according to the distance from the projection reference plane, the method for determining the three-dimensional projection function according to any one of claims 1 to 4.

6. When each vertex of the lower surface of the cube is labeled A, E, F, B in clockwise order, and each vertex of the upper surface is labeled C, G, H, D in clockwise order as well, In the step where the user corrects the figure, the figure is automatically corrected according to the user's operation so as to satisfy (Condition 1), or both (Condition 1) and (Condition 2). The method for determining a three-dimensional projection function according to claim 4, characterized in that. (Condition 1) AC, BD, EG, FH are vertical lines. (Condition 2) Let the points where the extension lines of AE and BF, CG and DH, DH and BF, and CG and AE intersect be P １ , P ２ , P ３ , P ４ respectively. Then P １ P ２ is a vertical line, and P ３ P ４ is a horizontal line.

7. A program for causing a computer to execute the method for determining a three-dimensional projection function according to any one of claims 1 to 6.

Images