Viewpoint tracking image generating device and its program

Abstract

To provide a viewpoint following video generation device that does not require a motion capture device to measure the position of a display device in real space.SOLUTION: A viewpoint following video generation device 7 includes: three-dimensional position / distance input means 72 for inputting three-dimensional positions / distances for three or more measurement points; real space / virtual space correspondence relation calculation means 75 for estimating a three-dimensional position of a screen in virtual space based on the three-dimensional positions / distances; and viewpoint following video generation means 76 for arranging a virtual camera at a viewpoint position and generating a video corresponding to the three-dimensional position of the screen in the virtual space.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a viewpoint tracking image generating device and a program therefor. [Background technology]

[0002] Viewpoint tracking image display technology provides a three-dimensional expression with motion parallax by appropriately displaying an image according to the viewpoint position of the observer in front of the screen. Viewpoint tracking images can be generated by extracting an appropriate image from a CG scene using the relationship between the viewpoint position and the screen position. Specifically, a virtual camera is placed in the CG scene, and the image is rendered based on the camera parameters (direction and angle of view of the virtual camera) calculated from the viewpoint position of the observer in real space and the position of the screen (for example, Patent Document 1).

[0003] To achieve high-quality 3D expression, the screen position and the observer's viewpoint position in the real space must correspond accurately to the virtual screen position and virtual camera position in the virtual space (CG scene). The virtual screen corresponds to the frame (window frame and angle of view) when shooting a CG scene with a virtual camera. The virtual space is an artificial 3D space in which the positions of the camera, screen, and other markers are expressed by coordinate values.

[0004] Here, a motion capture device is used to measure the viewpoint position and screen position in real space. For example, a motion capture device is a device that irradiates a marker with infrared light, captures the reflection with multiple cameras, and calculates the 3D position of the marker. By attaching markers to the position of the observer's eyes and the corners of the screen, the viewpoint position and screen position can be measured, respectively. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent No. 6581514 Summary of the Invention [Problem to be solved by the invention]

[0006] However, the motion capture device described above needs to use multiple cameras to cover the range where the markers are located (the range where the screen or observer moves). For example, if the screen is huge or located far from the observer, the motion capture device needs to widen the measurement range of the viewpoint position and screen position. In order to widen the measurement range, it is necessary to increase the number of cameras, which leads to an increase in the size of the device and increases the cost. For example, a motion capture device needs 12 or more cameras.

[0007] Here, when the screen is huge, it is assumed that the screen itself does not move. In this case, if the position of the stationary screen can be known by some means, it is sufficient to measure only the observer's viewpoint position. As a result, since it is not necessary to measure the position of the screen in real space, the measurement range of the motion capture device is narrowed, the number of cameras can be reduced, and the device size and cost can be suppressed.

[0008] An object of the present invention is to provide a point tracking image generation device and a program therefor that do not require the position of the display device in real space to be measured by a motion capture device. [Means for solving the problem]

[0009] In order to solve the above problems, the viewpoint tracking image generation device of the present invention is a viewpoint tracking image generation device that generates an image that tracks the viewpoint position based on the viewpoint position of a person, the three-dimensional position of a rangefinder that measures the distance to a target displayed on a display device, and the distance from the rangefinder to the target, and is configured to include a position / distance input means, a display device position estimation means, and an image generation means.

[0010] According to this configuration, the position / distance input means inputs the distances from the range finder to the target in the real space and the positions of the range finder in the virtual space for three or more measurement points. The display device position estimating means estimates the position of the display device in the virtual space based on the distance from the range finder to the target in real space and the position of the range finder in the virtual space. The image generating means places a virtual camera at a viewpoint position and generates an image corresponding to the position of the display device in the virtual space.

[0011] In other words, since the viewpoint tracking image generation device estimates the position of the display device from the distance from the display device to the range finder and the position of the range finder, there is no need to measure the position of the display device in real space, and the measurement range of the motion capture device can be narrowed.

[0012] The present invention can also be realized by a program for causing a computer to function as the viewpoint tracking image generating device described above. Effect of the Invention

[0013] According to the present invention, there is no need to measure the position of the display device in real space using a motion capture device, and the measurement range of the motion capture device can be narrowed. [Brief description of the drawings]

[0014] [Figure 1] 1 is a schematic configuration diagram of a viewpoint tracking video display system according to an embodiment. [Diagram 2] (a) is a close-up of the glasses, and (b) is a close-up of the rangefinder. [Diagram 3] 1 is a block diagram showing a configuration of a viewpoint tracking image production device according to an embodiment. [Figure 4] FIG. 4 is an explanatory diagram for explaining measurement by a distance measuring device in the embodiment. [Diagram 5] FIG. 2 is an explanatory diagram illustrating a correspondence relationship between a real space and a virtual space in the embodiment. [Figure 6]FIG. 4 is an explanatory diagram for explaining a screen position estimation method in the embodiment. [Figure 7] 4 is a flowchart showing the operation of the viewpoint tracking image production device according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, each embodiment described below is intended to embody the technical idea of ​​the present invention, and unless otherwise specified, the present invention is not limited to the following. In addition, the same symbols are used for the same means, and descriptions thereof may be omitted.

[0016] (Embodiment) [Configuration of viewpoint tracking video display system] The configuration of a viewpoint tracking video display system 1 according to an embodiment will be described with reference to FIGS. The viewpoint tracking video display system 1 displays an image (viewpoint tracking video) that tracks the viewpoint of an observer α. As shown in Fig. 1, the viewpoint tracking video display system 1 includes a screen (display device) 2, glasses 3, a range finder 4, a camera 5 (51-56), a position measurement device 6, and a viewpoint tracking video generating device 7. In this embodiment, the viewpoint tracking video display system 1 is set up indoors, such as in a photography studio.

[0017] The screen 2 displays an image input from a viewpoint tracking image generating device 7, which will be described later. The screen 2 also displays a target T on the screen according to a command from the viewpoint tracking image generating device 7. The position of the screen 2 is fixed and does not move. For example, the screen 2 is a typical large 200-inch liquid crystal screen, but the type and size are not particularly limited. It is not necessary to attach a marker M to the screen 2 for measuring its position.

[0018] The glasses 3 are glasses with markers for measuring the viewpoint position of the observer α. As shown in Fig. 2(a), the glasses 3 have multiple spherical markers M (e.g., two markers arranged parallel to each other) attached to the upper part of the frame for measuring the viewpoint position. The markers M are coated with a retroreflective material that reflects infrared rays from the camera 5 described later.

[0019] The range finder 4 measures the distance to a target T displayed on the screen 2, and outputs the measured distance to the viewpoint tracking image generating device 7. The range finder 4 is mounted on a tripod 40, and is positioned facing the screen 2 so as to be aligned with the observer α. For example, the range finder 4 may be a laser distance measuring system or a ToF (Time of Flight) camera. Furthermore, as shown in FIG. 2(b), the range finder 4 has multiple markers M attached thereto, similar to those of the glasses 3. For example, four markers M are attached to a plate 41 fixed to the side of the housing of the range finder 4.

[0020] Camera 5 is a camera for measuring the viewpoint position of observer α and the three-dimensional position of range finder 4. For example, camera 5 may be an infrared camera that emits infrared rays, as in a typical motion capture device. Camera 5 outputs an infrared image of glasses 3 and marker M of range finder 4 to position measurement device 6.

[0021] 1, a first camera 51 is disposed to the right of the observer α, a second camera 52 is disposed to the left rear of the observer α, and a third camera 53 is disposed to the left front of the observer α. These cameras 51 to 53 are mounted on a tripod 50. The fourth to sixth cameras 54 to 56 are disposed in a line below the screen 2 so as to face the observer α. It is sufficient that there are three or more cameras 5. In this embodiment, six cameras 5 are arranged in consideration of the case where the marker M is in the blind spot of the camera 5 and measurement becomes unstable.

[0022] Like a general motion capture device, the position measurement device 6 measures the viewpoint position of the observer α and the three-dimensional position of the range finder 4 using an infrared image input from the camera 5. First, the position measurement device 6 calculates the camera parameters of each camera 5 in advance by camera calibration. Next, the position measurement device 6 detects the image coordinates of the marker M from the infrared image of each camera 5. Next, the position measurement device 6 refers to the camera parameters of each camera 5 and converts the detected image coordinates of the marker M into a three-dimensional position. Then, the position measurement device 6 outputs the position of the marker M attached to the glasses 3 as the viewpoint position and the position of the marker M attached to the range finder 4 as the three-dimensional position of the range finder 4 to the viewpoint tracking image generation device 7.

[0023] The viewpoint tracking image generating device 7 generates an image that tracks the viewpoint position based on the viewpoint position of observer α, the three-dimensional position of a range finder 4 that measures the distance to a target T displayed on the screen 2, and the distance from the range finder 4 to the target T.

[0024] [Configuration of viewpoint tracking image generation device] The configuration of the viewpoint tracking video production device 7 will be described with reference to FIG. As shown in FIG. 3, the viewpoint tracking image generation device 7 includes a target display means (target display means) 70, a display mode switching means 71, a three-dimensional position / distance input means (position / distance input means) 72, a three-dimensional position / distance memory means 73, a measurement / recording repetition means 74, a real space / virtual space correspondence calculation means (display device position estimation means) 75, and a viewpoint tracking image generation means (image generation means) 76.

[0025] The target display means 70 displays a target T at a predetermined position on the screen 2. Specifically, the target display means 70 generates a target image in which the target T is displayed, and outputs the generated target image to the display mode switching means 71. In the example of Fig. 4, the target display means 70 displays a circular target T in the center of the screen 2. Note that the display position, shape, and size of the target T are not particularly limited.

[0026] The display mode switching means 71 switches the display mode of the viewpoint tracking image generation device 7 to a calibration mode or a display mode in accordance with an instruction from the observer α or an operator (not shown) of the viewpoint tracking image display system 1. For example, the display mode of the viewpoint tracking image generation device 7 is switched manually by the observer α or the operator using a “switching button” displayed on the screen of the viewpoint tracking image generation device 7.

[0027] The calibration mode is a mode for estimating a three-dimensional position of the screen 2. In the calibration mode, the display mode switching means 71 displays the target image input from the target display means 70 on the screen 2. The display mode is a mode in which a viewpoint tracking video is displayed on the screen 2. In the display mode, the display mode switching means 71 displays the viewpoint tracking video input from the viewpoint tracking video generating means 76 on the screen 2.

[0028] The three-dimensional position / distance input means 72 receives the distance from the range finder 4 to the target T in real space and the three-dimensional position of the range finder 4 in virtual space for three or more measurement points. Hereinafter, the set of the three-dimensional position of the range finder 4 and the distance from the range finder 4 to the target T may be abbreviated as "three-dimensional position / distance". The three-dimensional position / distance input means 72 then writes the three-dimensional position / distance of each input measurement point into the three-dimensional position / distance storage means 73.

[0029] The three-dimensional position / distance storage means 73 is a storage device such as an HDD (Hard Disk Drive) or SSD (Solid State Drive) that stores the three-dimensional position / distance of each measurement point. The measurement / recording repeating means 74 repeats the three-dimensional position / distance of each measurement point and stores them in the three-dimensional position / distance storage means 73 .

[0030] <Storage of 3D positions and distances> Here, we explain the method for storing the three-dimensional position and distance of each measurement point. Observer α places the range finder 4 in advance at a measurement point set at an arbitrary position in real space. When observer α or an operator commands the calibration mode using the "switch button" displayed on the screen of viewpoint tracking image generating device 7, the target display means 70 displays the target T on the screen 2. As shown in FIG. 4, the range finder 4 irradiates a laser toward the target T and measures the distance between the range finder 4 and the target T. When observer α presses the "measurement button" while irradiating the laser onto the target T, the position measurement device 6 measures the three-dimensional position of the range finder 4 at that time. Then, the three-dimensional position and distance at the measurement point are input to the three-dimensional position and distance input means 72 and stored in the three-dimensional position and distance storage means 73.

[0031] Then, the range finder 4 is moved to a different measurement point to measure the three-dimensional position and distance. The measurement of the three-dimensional position and distance is repeated, and the three-dimensional positions and distances for at least three measurement points are stored in the three-dimensional position and distance storage means 73. After that, the real space / virtual space correspondence calculation means 75 described later performs processing using the three-dimensional positions and distances stored in the three-dimensional position / distance storage means 73.

[0032] After the above process is completed, the observer α or the operator commands the display mode by pressing the "switch button." Then, the viewpoint tracking image generating means 76, which will be described later, generates the viewpoint tracking image.

[0033] The explanation of the configuration of the viewpoint tracking image generating device 7 will be continued below. The real space-virtual space correspondence calculation means 75 refers to the three-dimensional position / distance storage means 73 and calculates the correspondence of the three-dimensional position of the screen 2 between the real space and the virtual space. In other words, the real space-virtual space correspondence calculation means 75 estimates the three-dimensional position of the screen 2 in the virtual space based on the distance from the rangefinder 4 to the target T in the real space and the three-dimensional position of the rangefinder 4 in the virtual space. Then, the real space-virtual space correspondence calculation means 75 outputs the estimated three-dimensional position of the screen 2 to the viewpoint tracking image generation means 76.

[0034] <Screen position estimation method> A method for estimating the position of the screen 2 will be described with reference to FIGS. Fig. 5 illustrates the relationship between the three-dimensional position of the screen 2 arranged in the virtual space and the viewpoint position of the observer α. As shown in Fig. 5, it is necessary to make the viewpoint position and the three-dimensional position of the screen 2 correspond to each other between the real space and the virtual space. In this case, the viewpoint tracking video display system 1 does not measure the three-dimensional position of the screen 2 in the real space, so it estimates the three-dimensional position of the screen 2 in the virtual space with high accuracy, as described below.

[0035] As shown in FIG. 6, the position vector of the target T displayed at the center of the screen 2 is c. In addition, the position vector of the range finder 4 at the i-th and j-th measurement points is q. i ,q j Let the distance from the target T to the range finder 4 be l i ,l j (where i=1, 2, ..., j=i+1). The origin O of the virtual space can be set to any position.

[0036] At this time, the reference point p, which is a unit distance from the target T on the extension line from the target T to the range finder 4, i ,p j The position vector of the distance measuring device 4 is expressed by the following formulas (1) and (2). That is, the real space / virtual space correspondence calculation means 75 calculates the position vector q of the distance measuring device 4 as shown in formulas (1) and (2). i ,q j The difference between the position vector c of the target T and the distance l i ,l j By dividing by and adding the position vector c, the reference point p of measurement point i,j is obtained. i ,p j Calculate the position vector of

[0037]

number

[0038] Here, the reference point p i ,p jThe vector connecting the target T and the reference point p i ,p j The middle m=(p i +p j ) / 2 are orthogonal to each other, so the relationship in equation (3) below holds.

[0039]

number

[0040] Reference point p i ,p j It is assumed that N points have been measured. A loss function L expressed by the formula (4) is set in advance in the real space / virtual space correspondence calculation means 75.

[0041]

number

[0042] By substituting equations (1) and (2) into equation (4), we obtain the following equation (5).

[0043]

number

[0044] In this case, when the loss function L in formula (5) is minimized, the error of the position vector c between the real space and the virtual space is minimized. The position vector c with the minimum error satisfies the following formula (6).

[0045]

number

[0046] Since it is difficult to analytically calculate the position vector c that minimizes equation (5), it is obtained by numerical calculation. Here, the gradient vector L' of the loss function L is defined by the following equation (7). Note that x, y, and z represent the three-dimensional directions in the virtual space. Therefore, c x ,c y,c z are the three-dimensional direction components of the position vector c, and L´ x ,L´ y ,L´ z are the three-dimensional directional components of the gradient vector L'.

[0047]

number

[0048] Moreover, the Jacobian matrix J is defined by the following equation (8).

[0049]

number

[0050] In this case, according to Newton's method, the recurrence formula is defined as the following equation (9).

number

[0051] From the above, the real space / virtual space correspondence calculation means 75 gives the position of the tentative target T as the initial value c0 of the position vector c, and iteratively calculates equation (9). Then, the real space / virtual space correspondence calculation means 75 estimates the position vector c that minimizes the loss function L of equation (5) as the three-dimensional position of the screen 2 in the virtual space.

[0052] Returning to FIG. 3, the description of the configuration of the viewpoint tracking image production device 7 will continue. The viewpoint tracking image generating means 76 places a virtual camera at the viewpoint position input from the position measuring device 6, and generates an image corresponding to the three-dimensional position of the screen 2 in the virtual space. That is, the viewpoint tracking image generating means 76 renders the viewpoint tracking image using camera parameters determined from the relationship between the viewpoint position of the observer α in the virtual space and the three-dimensional position of the screen 2. For example, a method for generating the viewpoint tracking image is described in Japanese Patent No. 6581514, and therefore further description will be omitted. Then, the viewpoint tracking image generating means 76 outputs the generated viewpoint tracking image to the display mode switching means 71.

[0053] [Operation of viewpoint tracking image generation device] The operation of the viewpoint tracking video production device 7 will be described with reference to FIG. 7, in step S1, the target display means 70 displays a target T at a predetermined position on the screen 2. In addition, the measurement / recording repeating means 74 initializes the number n of input measurement points (n=0).

[0054] In step S2, the three-dimensional position / distance input means 72 receives as input the distance from the range finder 4 to the target T in the real space and the three-dimensional position of the range finder 4 in the virtual space. In step S3, the three-dimensional position / distance input means 72 writes the three-dimensional position / distance input in step S2 into the three-dimensional position / distance storage means 73. In addition, the measurement / recording repeating means 74 counts up the number n of input measurement points (n=n+1).

[0055] In step S4, the measurement / recording repeating means 74 judges whether the number n of input measurement points is less than a predetermined limit number (e.g., 4) or not. In other words, the measurement / recording repeating means 74 judges whether the three-dimensional positions / distances have been input for three measurement points or not.

[0056] If the input number n of measurement points is less than the limit number (Yes in step S4), the viewpoint tracking image production device 7 returns to the process of step S2. If the input number n of measurement points is equal to or greater than the limit (No in step S4), the viewpoint tracking image production device 7 proceeds to the process of step S5.

[0057] In step S5, the real space / virtual space correspondence calculation means 75 calculates the correspondence between the real space and the virtual space from the three-dimensional position and distance. That is, the real space / virtual space correspondence calculation means 75 estimates the three-dimensional position of the screen 2 in the virtual space based on the distance from the range finder 4 to the target T in the real space and the three-dimensional position of the range finder 4 in the virtual space.

[0058] In step S6, the real space / virtual space correspondence calculation means 75 determines whether or not to end the calibration. For example, when the observer α or the operator commands the calibration mode, the real space / virtual space correspondence calculation means 75 determines to end the calibration.

[0059] If the calibration is not to be ended (No in step S6), the viewpoint tracking image production device 7 returns to the process of step S2. When the calibration is to be ended (Yes in step S6), the viewpoint tracking image production device 7 proceeds to the process of step S7.

[0060] In step S7, the viewpoint tracking image generating means 76 places a virtual camera at the viewpoint position input from the position measuring device 6, and generates an image corresponding to the three-dimensional position of the screen 2 in the virtual space.

[0061] Note that the viewpoint tracking video production device 7 executes the processes of steps S1 to S6 in the calibration mode, and executes the process of step S7 in the display mode.

[0062] [Actions and Effects] As described above, the viewpoint tracking image generation device 7 according to the embodiment estimates the three-dimensional position of the screen 2 in virtual space from the three-dimensional position and distance, so there is no need to directly measure the three-dimensional position of the screen 2 by the position measurement device 6, and it is possible to narrow the measurement range of the position measurement device 6. As a result, the viewpoint tracking image generation device 7 can reduce the number of cameras 5 by about half, thereby suppressing the device scale and costs.

[0063] (Modification) Although the embodiments have been described in detail above, the present invention is not limited to the above-described embodiments, and includes design modifications and the like within the scope of the present invention. In the present invention, the method for detecting the viewpoint position and the distance meter position is not particularly limited, and the method for generating the viewpoint tracking video is also not particularly limited. In the above embodiment, the position measuring device and the viewpoint tracking image generating device are separate devices, but the present invention is not limited to this. For example, the viewpoint tracking image generating device may have a built-in position measuring device.

[0064] In the above embodiment, the viewpoint tracking image generating device is described as an independent piece of hardware, but the present invention is not limited to this. For example, the present invention can be realized by a program that operates hardware resources such as a CPU, memory, and hard disk of a computer as the viewpoint tracking image generating device. This program may be distributed via a communication line, or may be written on a recording medium such as a CD-ROM or a flash memory and distributed. [Industrial Applicability]

[0065] For example, the viewpoint tracking image generating device according to the present invention can display high quality viewpoint tracking images, and therefore can be used not only in television but also in interactive media such as games. [Explanation of symbols]

[0066] 1. Viewpoint tracking video display system 2 Screen (display device) 3 glasses 4 Rangefinder 5,51~56 Camera 6 Position measuring device 7. Viewpoint tracking image generation device 70 Target display means (goal display means) 71 Display mode switching means 72 Three-dimensional position / distance input means (position / distance input means) 73 3D position / distance storage means 74 Measurement and Recording Repeat Methods 75 Real space / virtual space correspondence calculation means (display device position estimation means) 76 Viewpoint tracking image generating means (image generating means)

Claims

1. A viewpoint tracking image generating device that generates an image that tracks a viewpoint position based on a viewpoint position of a person, a three-dimensional position of a range finder that measures a distance to a target displayed on a display device, and a distance from the range finder to the target, comprising: a position / distance input means for inputting a distance from said range finder to said target in real space and a position of said range finder in virtual space for three or more measurement points; a display device position estimating means for estimating a position of the display device in a virtual space based on a distance from the range finder to the target in real space and a position of the range finder in a virtual space; an image generating means for arranging a virtual camera at the viewpoint position and generating the image corresponding to the position of the display device in a virtual space; A viewpoint tracking image generating device comprising:

2. The display device position estimation means a difference between the position vector of the range finder and the position vector of the target is divided by the distance, and then the position vector of the target is added to the result, thereby calculating a reference point that is located at a unit distance from the target on an extension line from the target to the range finder, for each of the measurement points; A loss function is set in advance based on a relationship in which a vector connecting the reference points and a vector from the target to the midpoint of the reference points are orthogonal to each other; 2. The viewpoint tracking image generating device according to claim 1, wherein the position of the display device in the virtual space is estimated based on the position of the target at which the loss function is minimized.

3. 3. The viewpoint tracking image generating device according to claim 1, wherein the image generating means renders the image using camera parameters determined from the relationship between the viewpoint position and the position of the display device.

4. a target display means for displaying the target at a predetermined position on the display device; The viewpoint tracking image generating device according to claim 1 , further comprising:

5. A program for causing a computer to function as the viewpoint tracking image generating device according to any one of claims 1 to 4.

Images